Note: This is test shard 8 of 8.
[==========] Running 7 tests from 6 test suites.
[----------] Global test environment set-up.
[----------] 1 test from TableRangeRebalancingTest
[ RUN ] TableRangeRebalancingTest.Basic
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240503 01:58:43.535845 398 test_util.cc:268] Using random seed: 255117268
I20240503 01:58:43.542575 398 ts_itest-base.cc:112] Starting cluster with:
I20240503 01:58:43.542675 398 ts_itest-base.cc:113] --------------
I20240503 01:58:43.542728 398 ts_itest-base.cc:114] 3 tablet servers
I20240503 01:58:43.542762 398 ts_itest-base.cc:115] 3 replicas per TS
I20240503 01:58:43.542783 398 ts_itest-base.cc:116] --------------
2024-05-03T01:58:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T01:58:43Z Disabled control of system clock
I20240503 01:58:43.552021 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task42rE7p/build/debug/bin/kudu
/tmp/dist-test-task42rE7p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:32809
--webserver_interface=127.0.99.190
--webserver_port=0
--builtin_ntp_servers=127.0.99.148:46021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.99.190:32809
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240503 01:58:43.655346 410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:58:43.655539 410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:58:43.655589 410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:58:43.658479 410 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:58:43.658535 410 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 01:58:43.658558 410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:58:43.658593 410 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 01:58:43.658625 410 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 01:58:43.662081 410 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:46021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.0.99.190:32809
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:32809
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.99.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.410
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:58:43.662868 410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:58:43.663905 410 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:58:43.672139 418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:58:43.673362 410 server_base.cc:1034] running on GCE node
W20240503 01:58:43.674321 415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:58:43.675364 416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:58:43.675828 410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:58:43.676723 410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:58:43.677874 410 hybrid_clock.cc:648] HybridClock initialized: now 1714701523677844 us; error 50 us; skew 500 ppm
I20240503 01:58:43.679692 410 webserver.cc:460] Webserver started at http://127.0.99.190:34833/ using document root <none> and password file <none>
I20240503 01:58:43.680203 410 fs_manager.cc:359] Metadata directory not provided
I20240503 01:58:43.680261 410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:58:43.680487 410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:58:43.682062 410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "ffd00d7367104ebd8005c14ad7d1e699"
format_stamp: "Formatted at 2024-05-03 01:58:43 on dist-test-slave-dtz3"
I20240503 01:58:43.682535 410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "ffd00d7367104ebd8005c14ad7d1e699"
format_stamp: "Formatted at 2024-05-03 01:58:43 on dist-test-slave-dtz3"
I20240503 01:58:43.685657 410 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 01:58:43.687718 424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:43.688707 410 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 01:58:43.688807 410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/wal
uuid: "ffd00d7367104ebd8005c14ad7d1e699"
format_stamp: "Formatted at 2024-05-03 01:58:43 on dist-test-slave-dtz3"
I20240503 01:58:43.688884 410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:58:43.703161 410 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:58:43.703738 410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:58:43.703917 410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:58:43.710086 410 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.190:32809
I20240503 01:58:43.710091 476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.190:32809 every 8 connection(s)
I20240503 01:58:43.710961 410 server_base.cc:1166] Dumped server information to /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 01:58:43.713326 477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:43.717130 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task42rE7p/build/debug/bin/kudu as pid 410
I20240503 01:58:43.717283 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 01:58:43.718504 477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffd00d7367104ebd8005c14ad7d1e699: Bootstrap starting.
I20240503 01:58:43.721000 477 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ffd00d7367104ebd8005c14ad7d1e699: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:43.721894 477 log.cc:824] T 00000000000000000000000000000000 P ffd00d7367104ebd8005c14ad7d1e699: Log is configured to *not* fsync() on all Append() calls
I20240503 01:58:43.723719 477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffd00d7367104ebd8005c14ad7d1e699: No bootstrap required, opened a new log
I20240503 01:58:43.726668 477 raft_consensus.cc:348] T 00000000000000000000000000000000 P ffd00d7367104ebd8005c14ad7d1e699 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffd00d7367104ebd8005c14ad7d1e699" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 32809 } }
I20240503 01:58:43.726835 477 raft_consensus.cc:374] T 00000000000000000000000000000000 P ffd00d7367104ebd8005c14ad7d1e699 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:43.726893 477 raft_consensus.cc:729] T 00000000000000000000000000000000 P ffd00d7367104ebd8005c14ad7d1e699 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffd00d7367104ebd8005c14ad7d1e699, State: Initialized, Role: FOLLOWER
I20240503 01:58:43.727270 477 consensus_queue.cc:260] T 00000000000000000000000000000000 P ffd00d7367104ebd8005c14ad7d1e699 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffd00d7367104ebd8005c14ad7d1e699" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 32809 } }
I20240503 01:58:43.727408 477 raft_consensus.cc:388] T 00000000000000000000000000000000 P ffd00d7367104ebd8005c14ad7d1e699 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 01:58:43.727473 477 raft_consensus.cc:482] T 00000000000000000000000000000000 P ffd00d7367104ebd8005c14ad7d1e699 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 01:58:43.727569 477 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ffd00d7367104ebd8005c14ad7d1e699 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:43.728511 477 raft_consensus.cc:504] T 00000000000000000000000000000000 P ffd00d7367104ebd8005c14ad7d1e699 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffd00d7367104ebd8005c14ad7d1e699" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 32809 } }
I20240503 01:58:43.728956 477 leader_election.cc:302] T 00000000000000000000000000000000 P ffd00d7367104ebd8005c14ad7d1e699 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ffd00d7367104ebd8005c14ad7d1e699; no voters:
I20240503 01:58:43.729254 477 leader_election.cc:288] T 00000000000000000000000000000000 P ffd00d7367104ebd8005c14ad7d1e699 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 01:58:43.729633 482 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ffd00d7367104ebd8005c14ad7d1e699 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:58:43.729874 482 raft_consensus.cc:686] T 00000000000000000000000000000000 P ffd00d7367104ebd8005c14ad7d1e699 [term 1 LEADER]: Becoming Leader. State: Replica: ffd00d7367104ebd8005c14ad7d1e699, State: Running, Role: LEADER
I20240503 01:58:43.730125 482 consensus_queue.cc:237] T 00000000000000000000000000000000 P ffd00d7367104ebd8005c14ad7d1e699 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffd00d7367104ebd8005c14ad7d1e699" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 32809 } }
I20240503 01:58:43.730361 477 sys_catalog.cc:564] T 00000000000000000000000000000000 P ffd00d7367104ebd8005c14ad7d1e699 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 01:58:43.731076 482 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffd00d7367104ebd8005c14ad7d1e699 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ffd00d7367104ebd8005c14ad7d1e699. Latest consensus state: current_term: 1 leader_uuid: "ffd00d7367104ebd8005c14ad7d1e699" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffd00d7367104ebd8005c14ad7d1e699" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 32809 } } }
I20240503 01:58:43.731185 482 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffd00d7367104ebd8005c14ad7d1e699 [sys.catalog]: This master's current role is: LEADER
I20240503 01:58:43.732355 483 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffd00d7367104ebd8005c14ad7d1e699 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ffd00d7367104ebd8005c14ad7d1e699" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffd00d7367104ebd8005c14ad7d1e699" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 32809 } } }
I20240503 01:58:43.732496 483 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffd00d7367104ebd8005c14ad7d1e699 [sys.catalog]: This master's current role is: LEADER
I20240503 01:58:43.733047 488 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 01:58:43.735217 488 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 01:58:43.739369 488 catalog_manager.cc:1347] Generated new cluster ID: 3a056ce511414b049ffcdf9d673d720f
I20240503 01:58:43.739470 488 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 01:58:43.754101 488 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 01:58:43.754850 488 catalog_manager.cc:1504] Loading token signing keys...
I20240503 01:58:43.764065 488 catalog_manager.cc:5857] T 00000000000000000000000000000000 P ffd00d7367104ebd8005c14ad7d1e699: Generated new TSK 0
I20240503 01:58:43.764626 488 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 01:58:43.773914 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task42rE7p/build/debug/bin/kudu
/tmp/dist-test-task42rE7p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.129:0
--local_ip_for_outbound_sockets=127.0.99.129
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:32809
--builtin_ntp_servers=127.0.99.148:46021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 01:58:43.878379 501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:58:43.878559 501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:58:43.878600 501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:58:43.881460 501 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:58:43.881516 501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:58:43.881583 501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.129
I20240503 01:58:43.884932 501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:46021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:32809
--never_fsync=true
--heap_profile_path=/tmp/kudu.501
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.129
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:58:43.885685 501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:58:43.886682 501 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:58:43.894924 509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:58:43.895733 507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:58:43.896920 506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:58:43.897863 501 server_base.cc:1034] running on GCE node
I20240503 01:58:43.898191 501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:58:43.898662 501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:58:43.899822 501 hybrid_clock.cc:648] HybridClock initialized: now 1714701523899811 us; error 25 us; skew 500 ppm
I20240503 01:58:43.901579 501 webserver.cc:460] Webserver started at http://127.0.99.129:36957/ using document root <none> and password file <none>
I20240503 01:58:43.902086 501 fs_manager.cc:359] Metadata directory not provided
I20240503 01:58:43.902144 501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:58:43.902302 501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:58:43.903872 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "cae31ce250aa4d64953e14da451c568c"
format_stamp: "Formatted at 2024-05-03 01:58:43 on dist-test-slave-dtz3"
I20240503 01:58:43.904351 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "cae31ce250aa4d64953e14da451c568c"
format_stamp: "Formatted at 2024-05-03 01:58:43 on dist-test-slave-dtz3"
I20240503 01:58:43.907398 501 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 01:58:43.909379 515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:43.910302 501 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 01:58:43.910405 501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "cae31ce250aa4d64953e14da451c568c"
format_stamp: "Formatted at 2024-05-03 01:58:43 on dist-test-slave-dtz3"
I20240503 01:58:43.910480 501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:58:43.922935 501 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:58:43.923450 501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:58:43.923607 501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:58:43.924046 501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:58:43.924916 501 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:58:43.924962 501 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:43.925016 501 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:58:43.925042 501 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:43.933102 501 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.129:44403
I20240503 01:58:43.933123 628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.129:44403 every 8 connection(s)
I20240503 01:58:43.933920 501 server_base.cc:1166] Dumped server information to /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 01:58:43.939141 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task42rE7p/build/debug/bin/kudu as pid 501
I20240503 01:58:43.939236 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 01:58:43.941290 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task42rE7p/build/debug/bin/kudu
/tmp/dist-test-task42rE7p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.130:0
--local_ip_for_outbound_sockets=127.0.99.130
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:32809
--builtin_ntp_servers=127.0.99.148:46021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 01:58:43.941870 629 heartbeater.cc:347] Connected to a master server at 127.0.99.190:32809
I20240503 01:58:43.942042 629 heartbeater.cc:464] Registering TS with master...
I20240503 01:58:43.942540 629 heartbeater.cc:511] Master 127.0.99.190:32809 requested a full tablet report, sending...
I20240503 01:58:43.944041 441 ts_manager.cc:194] Registered new tserver with Master: cae31ce250aa4d64953e14da451c568c (127.0.99.129:44403)
I20240503 01:58:43.945150 441 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.129:44829
W20240503 01:58:44.049438 632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:58:44.049649 632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:58:44.049737 632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:58:44.052721 632 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:58:44.052784 632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:58:44.052871 632 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.130
I20240503 01:58:44.056341 632 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:46021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:32809
--never_fsync=true
--heap_profile_path=/tmp/kudu.632
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.130
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:58:44.057221 632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:58:44.058295 632 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:58:44.067021 638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:58:44.068027 640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:58:44.069010 637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:58:44.070045 632 server_base.cc:1034] running on GCE node
I20240503 01:58:44.070375 632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:58:44.070863 632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:58:44.072001 632 hybrid_clock.cc:648] HybridClock initialized: now 1714701524071992 us; error 28 us; skew 500 ppm
I20240503 01:58:44.077597 632 webserver.cc:460] Webserver started at http://127.0.99.130:35899/ using document root <none> and password file <none>
I20240503 01:58:44.078126 632 fs_manager.cc:359] Metadata directory not provided
I20240503 01:58:44.078183 632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:58:44.078341 632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:58:44.079878 632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "82c39407898d4033af3c6a1a688a5197"
format_stamp: "Formatted at 2024-05-03 01:58:44 on dist-test-slave-dtz3"
I20240503 01:58:44.080315 632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "82c39407898d4033af3c6a1a688a5197"
format_stamp: "Formatted at 2024-05-03 01:58:44 on dist-test-slave-dtz3"
I20240503 01:58:44.083513 632 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 01:58:44.085629 646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:44.086575 632 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 01:58:44.086674 632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "82c39407898d4033af3c6a1a688a5197"
format_stamp: "Formatted at 2024-05-03 01:58:44 on dist-test-slave-dtz3"
I20240503 01:58:44.086750 632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:58:44.102109 632 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:58:44.102739 632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:58:44.102880 632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:58:44.103408 632 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:58:44.104282 632 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:58:44.104328 632 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:44.104382 632 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:58:44.104405 632 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:44.117367 632 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.130:44599
I20240503 01:58:44.117417 759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.130:44599 every 8 connection(s)
I20240503 01:58:44.118237 632 server_base.cc:1166] Dumped server information to /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 01:58:44.126515 760 heartbeater.cc:347] Connected to a master server at 127.0.99.190:32809
I20240503 01:58:44.126703 760 heartbeater.cc:464] Registering TS with master...
I20240503 01:58:44.127169 760 heartbeater.cc:511] Master 127.0.99.190:32809 requested a full tablet report, sending...
I20240503 01:58:44.128006 441 ts_manager.cc:194] Registered new tserver with Master: 82c39407898d4033af3c6a1a688a5197 (127.0.99.130:44599)
I20240503 01:58:44.128155 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task42rE7p/build/debug/bin/kudu as pid 632
I20240503 01:58:44.128252 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 01:58:44.128553 441 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.130:55909
I20240503 01:58:44.130296 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task42rE7p/build/debug/bin/kudu
/tmp/dist-test-task42rE7p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.131:0
--local_ip_for_outbound_sockets=127.0.99.131
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:32809
--builtin_ntp_servers=127.0.99.148:46021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 01:58:44.240702 763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:58:44.240906 763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:58:44.240947 763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:58:44.243628 763 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:58:44.243677 763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:58:44.243733 763 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.131
I20240503 01:58:44.247129 763 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:46021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:32809
--never_fsync=true
--heap_profile_path=/tmp/kudu.763
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.131
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:58:44.247862 763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:58:44.248881 763 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:58:44.257421 771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:58:44.258201 769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:58:44.259478 768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:58:44.260308 763 server_base.cc:1034] running on GCE node
I20240503 01:58:44.260617 763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:58:44.261130 763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:58:44.262281 763 hybrid_clock.cc:648] HybridClock initialized: now 1714701524262270 us; error 23 us; skew 500 ppm
I20240503 01:58:44.264272 763 webserver.cc:460] Webserver started at http://127.0.99.131:41759/ using document root <none> and password file <none>
I20240503 01:58:44.264811 763 fs_manager.cc:359] Metadata directory not provided
I20240503 01:58:44.264866 763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:58:44.265030 763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:58:44.266594 763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "d125d3941ab94bd5b970110e5791e171"
format_stamp: "Formatted at 2024-05-03 01:58:44 on dist-test-slave-dtz3"
I20240503 01:58:44.267035 763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "d125d3941ab94bd5b970110e5791e171"
format_stamp: "Formatted at 2024-05-03 01:58:44 on dist-test-slave-dtz3"
I20240503 01:58:44.270149 763 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 01:58:44.272166 777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:44.273140 763 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 01:58:44.273238 763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "d125d3941ab94bd5b970110e5791e171"
format_stamp: "Formatted at 2024-05-03 01:58:44 on dist-test-slave-dtz3"
I20240503 01:58:44.273312 763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:58:44.290030 763 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:58:44.290657 763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:58:44.290802 763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:58:44.291343 763 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:58:44.292258 763 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:58:44.292305 763 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:44.292357 763 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:58:44.292380 763 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:44.305253 763 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.131:44747
I20240503 01:58:44.305301 890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.131:44747 every 8 connection(s)
I20240503 01:58:44.306150 763 server_base.cc:1166] Dumped server information to /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 01:58:44.315709 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task42rE7p/build/debug/bin/kudu as pid 763
I20240503 01:58:44.315810 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 01:58:44.316009 891 heartbeater.cc:347] Connected to a master server at 127.0.99.190:32809
I20240503 01:58:44.316252 891 heartbeater.cc:464] Registering TS with master...
I20240503 01:58:44.316866 891 heartbeater.cc:511] Master 127.0.99.190:32809 requested a full tablet report, sending...
I20240503 01:58:44.317761 440 ts_manager.cc:194] Registered new tserver with Master: d125d3941ab94bd5b970110e5791e171 (127.0.99.131:44747)
I20240503 01:58:44.318372 440 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.131:57989
I20240503 01:58:44.329468 398 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240503 01:58:44.340221 398 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240503 01:58:44.947145 629 heartbeater.cc:503] Master 127.0.99.190:32809 was elected leader, sending a full tablet report...
I20240503 01:58:45.130589 760 heartbeater.cc:503] Master 127.0.99.190:32809 was elected leader, sending a full tablet report...
I20240503 01:58:45.320284 891 heartbeater.cc:503] Master 127.0.99.190:32809 was elected leader, sending a full tablet report...
I20240503 01:58:48.093286 440 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60998:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\200\007\001\000\000\000\000\000\006\001\000\000\000\000\000\007\001\000\377\377\377\177"
indirect_data: "<redacted>"""
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 8
seed: 0
}
}
W20240503 01:58:48.097056 440 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 01:58:48.121418 816 tablet_service.cc:1467] Processing CreateTablet for tablet e0970cda8db54a8593f7d482e861da46 (DEFAULT_TABLE table=rebalance_test_table_0 [id=56e46e8e727d4c9bb8ebe539698280f9]), partition=HASH (key) PARTITION 4, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 01:58:48.122781 816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0970cda8db54a8593f7d482e861da46. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:48.121223 820 tablet_service.cc:1467] Processing CreateTablet for tablet c54888ed43eb4ab4bc79a9ab0a2bcf51 (DEFAULT_TABLE table=rebalance_test_table_0 [id=56e46e8e727d4c9bb8ebe539698280f9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 01:58:48.123323 820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c54888ed43eb4ab4bc79a9ab0a2bcf51. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:48.123746 556 tablet_service.cc:1467] Processing CreateTablet for tablet 0488ca07075c4daca7b8f4d5f827ea56 (DEFAULT_TABLE table=rebalance_test_table_0 [id=56e46e8e727d4c9bb8ebe539698280f9]), partition=HASH (key) PARTITION 3, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 01:58:48.123072 557 tablet_service.cc:1467] Processing CreateTablet for tablet 5a547015b4514b8286adb5c2e6356ba1 (DEFAULT_TABLE table=rebalance_test_table_0 [id=56e46e8e727d4c9bb8ebe539698280f9]), partition=HASH (key) PARTITION 3, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 01:58:48.123077 554 tablet_service.cc:1467] Processing CreateTablet for tablet e0970cda8db54a8593f7d482e861da46 (DEFAULT_TABLE table=rebalance_test_table_0 [id=56e46e8e727d4c9bb8ebe539698280f9]), partition=HASH (key) PARTITION 4, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 01:58:48.123044 562 tablet_service.cc:1467] Processing CreateTablet for tablet 42a69d697389414a973d995a763d8219 (DEFAULT_TABLE table=rebalance_test_table_0 [id=56e46e8e727d4c9bb8ebe539698280f9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 01:58:48.124253 817 tablet_service.cc:1467] Processing CreateTablet for tablet 84bfa8f64219448d8d49a30d50068c22 (DEFAULT_TABLE table=rebalance_test_table_0 [id=56e46e8e727d4c9bb8ebe539698280f9]), partition=HASH (key) PARTITION 4, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 01:58:48.123339 555 tablet_service.cc:1467] Processing CreateTablet for tablet 84bfa8f64219448d8d49a30d50068c22 (DEFAULT_TABLE table=rebalance_test_table_0 [id=56e46e8e727d4c9bb8ebe539698280f9]), partition=HASH (key) PARTITION 4, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 01:58:48.124558 817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84bfa8f64219448d8d49a30d50068c22. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:48.124660 556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0488ca07075c4daca7b8f4d5f827ea56. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:48.124909 563 tablet_service.cc:1467] Processing CreateTablet for tablet d2a69cc1ebd8418f85677c18ef9d80ef (DEFAULT_TABLE table=rebalance_test_table_0 [id=56e46e8e727d4c9bb8ebe539698280f9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 01:58:48.125128 563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2a69cc1ebd8418f85677c18ef9d80ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:48.125180 818 tablet_service.cc:1467] Processing CreateTablet for tablet 0488ca07075c4daca7b8f4d5f827ea56 (DEFAULT_TABLE table=rebalance_test_table_0 [id=56e46e8e727d4c9bb8ebe539698280f9]), partition=HASH (key) PARTITION 3, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 01:58:48.125430 818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0488ca07075c4daca7b8f4d5f827ea56. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:48.126108 560 tablet_service.cc:1467] Processing CreateTablet for tablet 2f1cd41c7fc8437598f73938ae3f2a1d (DEFAULT_TABLE table=rebalance_test_table_0 [id=56e46e8e727d4c9bb8ebe539698280f9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 01:58:48.126360 560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f1cd41c7fc8437598f73938ae3f2a1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:48.126885 559 tablet_service.cc:1467] Processing CreateTablet for tablet a7090b6ae5394aecb146a7fa25b89e79 (DEFAULT_TABLE table=rebalance_test_table_0 [id=56e46e8e727d4c9bb8ebe539698280f9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 01:58:48.127115 559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7090b6ae5394aecb146a7fa25b89e79. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:48.127573 819 tablet_service.cc:1467] Processing CreateTablet for tablet 5a547015b4514b8286adb5c2e6356ba1 (DEFAULT_TABLE table=rebalance_test_table_0 [id=56e46e8e727d4c9bb8ebe539698280f9]), partition=HASH (key) PARTITION 3, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 01:58:48.127832 819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a547015b4514b8286adb5c2e6356ba1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:48.130470 821 tablet_service.cc:1467] Processing CreateTablet for tablet a7090b6ae5394aecb146a7fa25b89e79 (DEFAULT_TABLE table=rebalance_test_table_0 [id=56e46e8e727d4c9bb8ebe539698280f9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 01:58:48.130759 821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7090b6ae5394aecb146a7fa25b89e79. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:48.132025 558 tablet_service.cc:1467] Processing CreateTablet for tablet c54888ed43eb4ab4bc79a9ab0a2bcf51 (DEFAULT_TABLE table=rebalance_test_table_0 [id=56e46e8e727d4c9bb8ebe539698280f9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 01:58:48.132297 558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c54888ed43eb4ab4bc79a9ab0a2bcf51. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:48.133273 822 tablet_service.cc:1467] Processing CreateTablet for tablet 2f1cd41c7fc8437598f73938ae3f2a1d (DEFAULT_TABLE table=rebalance_test_table_0 [id=56e46e8e727d4c9bb8ebe539698280f9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 01:58:48.133535 822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f1cd41c7fc8437598f73938ae3f2a1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:48.133847 910 tablet_bootstrap.cc:492] T a7090b6ae5394aecb146a7fa25b89e79 P cae31ce250aa4d64953e14da451c568c: Bootstrap starting.
I20240503 01:58:48.135320 823 tablet_service.cc:1467] Processing CreateTablet for tablet 86f9a4762bf34b52a8d000a487226e5b (DEFAULT_TABLE table=rebalance_test_table_0 [id=56e46e8e727d4c9bb8ebe539698280f9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 01:58:48.135579 823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86f9a4762bf34b52a8d000a487226e5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:48.135715 561 tablet_service.cc:1467] Processing CreateTablet for tablet 86f9a4762bf34b52a8d000a487226e5b (DEFAULT_TABLE table=rebalance_test_table_0 [id=56e46e8e727d4c9bb8ebe539698280f9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 01:58:48.135955 561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86f9a4762bf34b52a8d000a487226e5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:48.136361 910 tablet_bootstrap.cc:654] T a7090b6ae5394aecb146a7fa25b89e79 P cae31ce250aa4d64953e14da451c568c: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:48.137485 910 log.cc:824] T a7090b6ae5394aecb146a7fa25b89e79 P cae31ce250aa4d64953e14da451c568c: Log is configured to *not* fsync() on all Append() calls
I20240503 01:58:48.138284 824 tablet_service.cc:1467] Processing CreateTablet for tablet 42a69d697389414a973d995a763d8219 (DEFAULT_TABLE table=rebalance_test_table_0 [id=56e46e8e727d4c9bb8ebe539698280f9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 01:58:48.138527 555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84bfa8f64219448d8d49a30d50068c22. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:48.138537 824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42a69d697389414a973d995a763d8219. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:48.138914 554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0970cda8db54a8593f7d482e861da46. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:48.139376 910 tablet_bootstrap.cc:492] T a7090b6ae5394aecb146a7fa25b89e79 P cae31ce250aa4d64953e14da451c568c: No bootstrap required, opened a new log
I20240503 01:58:48.139544 910 ts_tablet_manager.cc:1367] T a7090b6ae5394aecb146a7fa25b89e79 P cae31ce250aa4d64953e14da451c568c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240503 01:58:48.139628 825 tablet_service.cc:1467] Processing CreateTablet for tablet d2a69cc1ebd8418f85677c18ef9d80ef (DEFAULT_TABLE table=rebalance_test_table_0 [id=56e46e8e727d4c9bb8ebe539698280f9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 01:58:48.143311 910 raft_consensus.cc:348] T a7090b6ae5394aecb146a7fa25b89e79 P cae31ce250aa4d64953e14da451c568c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.143543 910 raft_consensus.cc:374] T a7090b6ae5394aecb146a7fa25b89e79 P cae31ce250aa4d64953e14da451c568c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:48.143600 910 raft_consensus.cc:729] T a7090b6ae5394aecb146a7fa25b89e79 P cae31ce250aa4d64953e14da451c568c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cae31ce250aa4d64953e14da451c568c, State: Initialized, Role: FOLLOWER
I20240503 01:58:48.143622 825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2a69cc1ebd8418f85677c18ef9d80ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:48.144043 910 consensus_queue.cc:260] T a7090b6ae5394aecb146a7fa25b89e79 P cae31ce250aa4d64953e14da451c568c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.145934 824 tablet_service.cc:1467] Processing CreateTablet for tablet 1581a7ae381c4228bbb4420b84ba432f (DEFAULT_TABLE table=rebalance_test_table_0 [id=56e46e8e727d4c9bb8ebe539698280f9]), partition=HASH (key) PARTITION 5, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 01:58:48.146006 910 ts_tablet_manager.cc:1398] T a7090b6ae5394aecb146a7fa25b89e79 P cae31ce250aa4d64953e14da451c568c: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20240503 01:58:48.146200 824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1581a7ae381c4228bbb4420b84ba432f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:48.147984 558 tablet_service.cc:1467] Processing CreateTablet for tablet e38e1de36d82402c82b72dce9e765505 (DEFAULT_TABLE table=rebalance_test_table_0 [id=56e46e8e727d4c9bb8ebe539698280f9]), partition=HASH (key) PARTITION 6, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 01:58:48.148296 558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e38e1de36d82402c82b72dce9e765505. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:48.149930 557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a547015b4514b8286adb5c2e6356ba1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:48.150185 910 tablet_bootstrap.cc:492] T 2f1cd41c7fc8437598f73938ae3f2a1d P cae31ce250aa4d64953e14da451c568c: Bootstrap starting.
I20240503 01:58:48.151880 818 tablet_service.cc:1467] Processing CreateTablet for tablet 9f1fb01218234ec4a13fd6f9dfb3fdd6 (DEFAULT_TABLE table=rebalance_test_table_0 [id=56e46e8e727d4c9bb8ebe539698280f9]), partition=HASH (key) PARTITION 7, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 01:58:48.152014 824 tablet_service.cc:1467] Processing CreateTablet for tablet e38e1de36d82402c82b72dce9e765505 (DEFAULT_TABLE table=rebalance_test_table_0 [id=56e46e8e727d4c9bb8ebe539698280f9]), partition=HASH (key) PARTITION 6, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 01:58:48.152102 818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f1fb01218234ec4a13fd6f9dfb3fdd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:48.152202 824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e38e1de36d82402c82b72dce9e765505. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:48.153417 910 tablet_bootstrap.cc:654] T 2f1cd41c7fc8437598f73938ae3f2a1d P cae31ce250aa4d64953e14da451c568c: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:48.153404 561 tablet_service.cc:1467] Processing CreateTablet for tablet 8d1ad90f476a4b2b97f59259ef4bff9f (DEFAULT_TABLE table=rebalance_test_table_0 [id=56e46e8e727d4c9bb8ebe539698280f9]), partition=HASH (key) PARTITION 6, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 01:58:48.153543 821 tablet_service.cc:1467] Processing CreateTablet for tablet ccaa4613da8546f88fb844a6711baabc (DEFAULT_TABLE table=rebalance_test_table_0 [id=56e46e8e727d4c9bb8ebe539698280f9]), partition=HASH (key) PARTITION 7, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 01:58:48.153647 561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d1ad90f476a4b2b97f59259ef4bff9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:48.153808 821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccaa4613da8546f88fb844a6711baabc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:48.154886 910 tablet_bootstrap.cc:492] T 2f1cd41c7fc8437598f73938ae3f2a1d P cae31ce250aa4d64953e14da451c568c: No bootstrap required, opened a new log
I20240503 01:58:48.154980 910 ts_tablet_manager.cc:1367] T 2f1cd41c7fc8437598f73938ae3f2a1d P cae31ce250aa4d64953e14da451c568c: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240503 01:58:48.155666 910 raft_consensus.cc:348] T 2f1cd41c7fc8437598f73938ae3f2a1d P cae31ce250aa4d64953e14da451c568c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.155766 910 raft_consensus.cc:374] T 2f1cd41c7fc8437598f73938ae3f2a1d P cae31ce250aa4d64953e14da451c568c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:48.155793 910 raft_consensus.cc:729] T 2f1cd41c7fc8437598f73938ae3f2a1d P cae31ce250aa4d64953e14da451c568c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cae31ce250aa4d64953e14da451c568c, State: Initialized, Role: FOLLOWER
I20240503 01:58:48.155908 910 consensus_queue.cc:260] T 2f1cd41c7fc8437598f73938ae3f2a1d P cae31ce250aa4d64953e14da451c568c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.146837 683 tablet_service.cc:1467] Processing CreateTablet for tablet 2e5dc6a7de704dcdb0aec15124300b61 (DEFAULT_TABLE table=rebalance_test_table_0 [id=56e46e8e727d4c9bb8ebe539698280f9]), partition=HASH (key) PARTITION 5, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 01:58:48.137818 692 tablet_service.cc:1467] Processing CreateTablet for tablet 86f9a4762bf34b52a8d000a487226e5b (DEFAULT_TABLE table=rebalance_test_table_0 [id=56e46e8e727d4c9bb8ebe539698280f9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 01:58:48.156332 910 ts_tablet_manager.cc:1398] T 2f1cd41c7fc8437598f73938ae3f2a1d P cae31ce250aa4d64953e14da451c568c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:58:48.147451 682 tablet_service.cc:1467] Processing CreateTablet for tablet 8d1ad90f476a4b2b97f59259ef4bff9f (DEFAULT_TABLE table=rebalance_test_table_0 [id=56e46e8e727d4c9bb8ebe539698280f9]), partition=HASH (key) PARTITION 6, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 01:58:48.140058 690 tablet_service.cc:1467] Processing CreateTablet for tablet a7090b6ae5394aecb146a7fa25b89e79 (DEFAULT_TABLE table=rebalance_test_table_0 [id=56e46e8e727d4c9bb8ebe539698280f9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 01:58:48.156513 910 tablet_bootstrap.cc:492] T 0488ca07075c4daca7b8f4d5f827ea56 P cae31ce250aa4d64953e14da451c568c: Bootstrap starting.
I20240503 01:58:48.139303 689 tablet_service.cc:1467] Processing CreateTablet for tablet c54888ed43eb4ab4bc79a9ab0a2bcf51 (DEFAULT_TABLE table=rebalance_test_table_0 [id=56e46e8e727d4c9bb8ebe539698280f9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 01:58:48.130108 685 tablet_service.cc:1467] Processing CreateTablet for tablet e0970cda8db54a8593f7d482e861da46 (DEFAULT_TABLE table=rebalance_test_table_0 [id=56e46e8e727d4c9bb8ebe539698280f9]), partition=HASH (key) PARTITION 4, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 01:58:48.137425 693 tablet_service.cc:1467] Processing CreateTablet for tablet 42a69d697389414a973d995a763d8219 (DEFAULT_TABLE table=rebalance_test_table_0 [id=56e46e8e727d4c9bb8ebe539698280f9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 01:58:48.151666 681 tablet_service.cc:1467] Processing CreateTablet for tablet e38e1de36d82402c82b72dce9e765505 (DEFAULT_TABLE table=rebalance_test_table_0 [id=56e46e8e727d4c9bb8ebe539698280f9]), partition=HASH (key) PARTITION 6, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 01:58:48.132964 686 tablet_service.cc:1467] Processing CreateTablet for tablet 84bfa8f64219448d8d49a30d50068c22 (DEFAULT_TABLE table=rebalance_test_table_0 [id=56e46e8e727d4c9bb8ebe539698280f9]), partition=HASH (key) PARTITION 4, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 01:58:48.151993 680 tablet_service.cc:1467] Processing CreateTablet for tablet 9f1fb01218234ec4a13fd6f9dfb3fdd6 (DEFAULT_TABLE table=rebalance_test_table_0 [id=56e46e8e727d4c9bb8ebe539698280f9]), partition=HASH (key) PARTITION 7, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 01:58:48.145846 684 tablet_service.cc:1467] Processing CreateTablet for tablet 1581a7ae381c4228bbb4420b84ba432f (DEFAULT_TABLE table=rebalance_test_table_0 [id=56e46e8e727d4c9bb8ebe539698280f9]), partition=HASH (key) PARTITION 5, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 01:58:48.120133 694 tablet_service.cc:1467] Processing CreateTablet for tablet d2a69cc1ebd8418f85677c18ef9d80ef (DEFAULT_TABLE table=rebalance_test_table_0 [id=56e46e8e727d4c9bb8ebe539698280f9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 01:58:48.139015 688 tablet_service.cc:1467] Processing CreateTablet for tablet 5a547015b4514b8286adb5c2e6356ba1 (DEFAULT_TABLE table=rebalance_test_table_0 [id=56e46e8e727d4c9bb8ebe539698280f9]), partition=HASH (key) PARTITION 3, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 01:58:48.138010 687 tablet_service.cc:1467] Processing CreateTablet for tablet 0488ca07075c4daca7b8f4d5f827ea56 (DEFAULT_TABLE table=rebalance_test_table_0 [id=56e46e8e727d4c9bb8ebe539698280f9]), partition=HASH (key) PARTITION 3, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 01:58:48.156338 558 tablet_service.cc:1467] Processing CreateTablet for tablet ccaa4613da8546f88fb844a6711baabc (DEFAULT_TABLE table=rebalance_test_table_0 [id=56e46e8e727d4c9bb8ebe539698280f9]), partition=HASH (key) PARTITION 7, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 01:58:48.157547 558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccaa4613da8546f88fb844a6711baabc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:48.140489 691 tablet_service.cc:1467] Processing CreateTablet for tablet 2f1cd41c7fc8437598f73938ae3f2a1d (DEFAULT_TABLE table=rebalance_test_table_0 [id=56e46e8e727d4c9bb8ebe539698280f9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 01:58:48.157755 910 tablet_bootstrap.cc:654] T 0488ca07075c4daca7b8f4d5f827ea56 P cae31ce250aa4d64953e14da451c568c: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:48.158484 692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86f9a4762bf34b52a8d000a487226e5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:48.152635 679 tablet_service.cc:1467] Processing CreateTablet for tablet ccaa4613da8546f88fb844a6711baabc (DEFAULT_TABLE table=rebalance_test_table_0 [id=56e46e8e727d4c9bb8ebe539698280f9]), partition=HASH (key) PARTITION 7, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 01:58:48.158330 554 tablet_service.cc:1467] Processing CreateTablet for tablet 2e5dc6a7de704dcdb0aec15124300b61 (DEFAULT_TABLE table=rebalance_test_table_0 [id=56e46e8e727d4c9bb8ebe539698280f9]), partition=HASH (key) PARTITION 5, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 01:58:48.158917 681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e38e1de36d82402c82b72dce9e765505. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:48.159648 679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccaa4613da8546f88fb844a6711baabc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:48.159826 690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7090b6ae5394aecb146a7fa25b89e79. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:48.160006 684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1581a7ae381c4228bbb4420b84ba432f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:48.160560 689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c54888ed43eb4ab4bc79a9ab0a2bcf51. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:48.160764 685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0970cda8db54a8593f7d482e861da46. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:48.160921 691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f1cd41c7fc8437598f73938ae3f2a1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:48.161101 693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42a69d697389414a973d995a763d8219. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:48.161249 687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0488ca07075c4daca7b8f4d5f827ea56. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:48.161993 910 tablet_bootstrap.cc:492] T 0488ca07075c4daca7b8f4d5f827ea56 P cae31ce250aa4d64953e14da451c568c: No bootstrap required, opened a new log
I20240503 01:58:48.162086 910 ts_tablet_manager.cc:1367] T 0488ca07075c4daca7b8f4d5f827ea56 P cae31ce250aa4d64953e14da451c568c: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240503 01:58:48.162248 682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d1ad90f476a4b2b97f59259ef4bff9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:48.162920 554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e5dc6a7de704dcdb0aec15124300b61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:48.162954 910 raft_consensus.cc:348] T 0488ca07075c4daca7b8f4d5f827ea56 P cae31ce250aa4d64953e14da451c568c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.163050 910 raft_consensus.cc:374] T 0488ca07075c4daca7b8f4d5f827ea56 P cae31ce250aa4d64953e14da451c568c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:48.163081 910 raft_consensus.cc:729] T 0488ca07075c4daca7b8f4d5f827ea56 P cae31ce250aa4d64953e14da451c568c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cae31ce250aa4d64953e14da451c568c, State: Initialized, Role: FOLLOWER
I20240503 01:58:48.163190 910 consensus_queue.cc:260] T 0488ca07075c4daca7b8f4d5f827ea56 P cae31ce250aa4d64953e14da451c568c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.163715 910 ts_tablet_manager.cc:1398] T 0488ca07075c4daca7b8f4d5f827ea56 P cae31ce250aa4d64953e14da451c568c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:58:48.164050 686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84bfa8f64219448d8d49a30d50068c22. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:48.165436 680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f1fb01218234ec4a13fd6f9dfb3fdd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:48.165632 910 tablet_bootstrap.cc:492] T c54888ed43eb4ab4bc79a9ab0a2bcf51 P cae31ce250aa4d64953e14da451c568c: Bootstrap starting.
I20240503 01:58:48.166790 910 tablet_bootstrap.cc:654] T c54888ed43eb4ab4bc79a9ab0a2bcf51 P cae31ce250aa4d64953e14da451c568c: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:48.167013 688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a547015b4514b8286adb5c2e6356ba1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:48.167438 555 tablet_service.cc:1467] Processing CreateTablet for tablet 1581a7ae381c4228bbb4420b84ba432f (DEFAULT_TABLE table=rebalance_test_table_0 [id=56e46e8e727d4c9bb8ebe539698280f9]), partition=HASH (key) PARTITION 5, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 01:58:48.167730 555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1581a7ae381c4228bbb4420b84ba432f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:48.168408 683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e5dc6a7de704dcdb0aec15124300b61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:48.168663 910 tablet_bootstrap.cc:492] T c54888ed43eb4ab4bc79a9ab0a2bcf51 P cae31ce250aa4d64953e14da451c568c: No bootstrap required, opened a new log
I20240503 01:58:48.168743 910 ts_tablet_manager.cc:1367] T c54888ed43eb4ab4bc79a9ab0a2bcf51 P cae31ce250aa4d64953e14da451c568c: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20240503 01:58:48.169274 910 raft_consensus.cc:348] T c54888ed43eb4ab4bc79a9ab0a2bcf51 P cae31ce250aa4d64953e14da451c568c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.169415 910 raft_consensus.cc:374] T c54888ed43eb4ab4bc79a9ab0a2bcf51 P cae31ce250aa4d64953e14da451c568c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:48.169450 910 raft_consensus.cc:729] T c54888ed43eb4ab4bc79a9ab0a2bcf51 P cae31ce250aa4d64953e14da451c568c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cae31ce250aa4d64953e14da451c568c, State: Initialized, Role: FOLLOWER
I20240503 01:58:48.169560 910 consensus_queue.cc:260] T c54888ed43eb4ab4bc79a9ab0a2bcf51 P cae31ce250aa4d64953e14da451c568c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.169965 910 ts_tablet_manager.cc:1398] T c54888ed43eb4ab4bc79a9ab0a2bcf51 P cae31ce250aa4d64953e14da451c568c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:58:48.170490 910 tablet_bootstrap.cc:492] T 86f9a4762bf34b52a8d000a487226e5b P cae31ce250aa4d64953e14da451c568c: Bootstrap starting.
I20240503 01:58:48.170560 825 tablet_service.cc:1467] Processing CreateTablet for tablet 2e5dc6a7de704dcdb0aec15124300b61 (DEFAULT_TABLE table=rebalance_test_table_0 [id=56e46e8e727d4c9bb8ebe539698280f9]), partition=HASH (key) PARTITION 5, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 01:58:48.170883 825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e5dc6a7de704dcdb0aec15124300b61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:48.171577 910 tablet_bootstrap.cc:654] T 86f9a4762bf34b52a8d000a487226e5b P cae31ce250aa4d64953e14da451c568c: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:48.173660 694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2a69cc1ebd8418f85677c18ef9d80ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:48.174047 562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42a69d697389414a973d995a763d8219. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:48.176443 816 tablet_service.cc:1467] Processing CreateTablet for tablet 8d1ad90f476a4b2b97f59259ef4bff9f (DEFAULT_TABLE table=rebalance_test_table_0 [id=56e46e8e727d4c9bb8ebe539698280f9]), partition=HASH (key) PARTITION 6, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 01:58:48.176705 816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d1ad90f476a4b2b97f59259ef4bff9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:48.177047 557 tablet_service.cc:1467] Processing CreateTablet for tablet 9f1fb01218234ec4a13fd6f9dfb3fdd6 (DEFAULT_TABLE table=rebalance_test_table_0 [id=56e46e8e727d4c9bb8ebe539698280f9]), partition=HASH (key) PARTITION 7, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 01:58:48.177371 557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f1fb01218234ec4a13fd6f9dfb3fdd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:48.179626 921 raft_consensus.cc:482] T 2f1cd41c7fc8437598f73938ae3f2a1d P cae31ce250aa4d64953e14da451c568c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:58:48.179797 921 raft_consensus.cc:504] T 2f1cd41c7fc8437598f73938ae3f2a1d P cae31ce250aa4d64953e14da451c568c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.181190 910 tablet_bootstrap.cc:492] T 86f9a4762bf34b52a8d000a487226e5b P cae31ce250aa4d64953e14da451c568c: No bootstrap required, opened a new log
I20240503 01:58:48.181269 910 ts_tablet_manager.cc:1367] T 86f9a4762bf34b52a8d000a487226e5b P cae31ce250aa4d64953e14da451c568c: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240503 01:58:48.181768 910 raft_consensus.cc:348] T 86f9a4762bf34b52a8d000a487226e5b P cae31ce250aa4d64953e14da451c568c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.181864 910 raft_consensus.cc:374] T 86f9a4762bf34b52a8d000a487226e5b P cae31ce250aa4d64953e14da451c568c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:48.181895 910 raft_consensus.cc:729] T 86f9a4762bf34b52a8d000a487226e5b P cae31ce250aa4d64953e14da451c568c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cae31ce250aa4d64953e14da451c568c, State: Initialized, Role: FOLLOWER
I20240503 01:58:48.181993 910 consensus_queue.cc:260] T 86f9a4762bf34b52a8d000a487226e5b P cae31ce250aa4d64953e14da451c568c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.182288 910 ts_tablet_manager.cc:1398] T 86f9a4762bf34b52a8d000a487226e5b P cae31ce250aa4d64953e14da451c568c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:58:48.182440 910 tablet_bootstrap.cc:492] T e0970cda8db54a8593f7d482e861da46 P cae31ce250aa4d64953e14da451c568c: Bootstrap starting.
I20240503 01:58:48.183488 920 tablet_bootstrap.cc:492] T c54888ed43eb4ab4bc79a9ab0a2bcf51 P 82c39407898d4033af3c6a1a688a5197: Bootstrap starting.
I20240503 01:58:48.183580 910 tablet_bootstrap.cc:654] T e0970cda8db54a8593f7d482e861da46 P cae31ce250aa4d64953e14da451c568c: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:48.184360 921 leader_election.cc:288] T 2f1cd41c7fc8437598f73938ae3f2a1d P cae31ce250aa4d64953e14da451c568c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82c39407898d4033af3c6a1a688a5197 (127.0.99.130:44599), d125d3941ab94bd5b970110e5791e171 (127.0.99.131:44747)
I20240503 01:58:48.185066 909 tablet_bootstrap.cc:492] T 5a547015b4514b8286adb5c2e6356ba1 P d125d3941ab94bd5b970110e5791e171: Bootstrap starting.
I20240503 01:58:48.187209 920 tablet_bootstrap.cc:654] T c54888ed43eb4ab4bc79a9ab0a2bcf51 P 82c39407898d4033af3c6a1a688a5197: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:48.186761 714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f1cd41c7fc8437598f73938ae3f2a1d" candidate_uuid: "cae31ce250aa4d64953e14da451c568c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82c39407898d4033af3c6a1a688a5197" is_pre_election: true
I20240503 01:58:48.188119 909 tablet_bootstrap.cc:654] T 5a547015b4514b8286adb5c2e6356ba1 P d125d3941ab94bd5b970110e5791e171: Neither blocks nor log segments found. Creating new log.
W20240503 01:58:48.188608 630 tablet.cc:2369] T 0488ca07075c4daca7b8f4d5f827ea56 P cae31ce250aa4d64953e14da451c568c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 01:58:48.188635 920 log.cc:824] T c54888ed43eb4ab4bc79a9ab0a2bcf51 P 82c39407898d4033af3c6a1a688a5197: Log is configured to *not* fsync() on all Append() calls
W20240503 01:58:48.189646 516 leader_election.cc:341] T 2f1cd41c7fc8437598f73938ae3f2a1d P cae31ce250aa4d64953e14da451c568c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 82c39407898d4033af3c6a1a688a5197 (127.0.99.130:44599): Illegal state: must be running to vote when last-logged opid is not known
I20240503 01:58:48.189724 909 log.cc:824] T 5a547015b4514b8286adb5c2e6356ba1 P d125d3941ab94bd5b970110e5791e171: Log is configured to *not* fsync() on all Append() calls
I20240503 01:58:48.190606 910 tablet_bootstrap.cc:492] T e0970cda8db54a8593f7d482e861da46 P cae31ce250aa4d64953e14da451c568c: No bootstrap required, opened a new log
I20240503 01:58:48.190701 910 ts_tablet_manager.cc:1367] T e0970cda8db54a8593f7d482e861da46 P cae31ce250aa4d64953e14da451c568c: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20240503 01:58:48.190789 920 tablet_bootstrap.cc:492] T c54888ed43eb4ab4bc79a9ab0a2bcf51 P 82c39407898d4033af3c6a1a688a5197: No bootstrap required, opened a new log
I20240503 01:58:48.190985 920 ts_tablet_manager.cc:1367] T c54888ed43eb4ab4bc79a9ab0a2bcf51 P 82c39407898d4033af3c6a1a688a5197: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20240503 01:58:48.191378 910 raft_consensus.cc:348] T e0970cda8db54a8593f7d482e861da46 P cae31ce250aa4d64953e14da451c568c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.191478 910 raft_consensus.cc:374] T e0970cda8db54a8593f7d482e861da46 P cae31ce250aa4d64953e14da451c568c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:48.191510 910 raft_consensus.cc:729] T e0970cda8db54a8593f7d482e861da46 P cae31ce250aa4d64953e14da451c568c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cae31ce250aa4d64953e14da451c568c, State: Initialized, Role: FOLLOWER
I20240503 01:58:48.191691 910 consensus_queue.cc:260] T e0970cda8db54a8593f7d482e861da46 P cae31ce250aa4d64953e14da451c568c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.191956 910 ts_tablet_manager.cc:1398] T e0970cda8db54a8593f7d482e861da46 P cae31ce250aa4d64953e14da451c568c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:58:48.190778 845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f1cd41c7fc8437598f73938ae3f2a1d" candidate_uuid: "cae31ce250aa4d64953e14da451c568c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d125d3941ab94bd5b970110e5791e171" is_pre_election: true
I20240503 01:58:48.192129 910 tablet_bootstrap.cc:492] T 84bfa8f64219448d8d49a30d50068c22 P cae31ce250aa4d64953e14da451c568c: Bootstrap starting.
W20240503 01:58:48.192817 517 leader_election.cc:341] T 2f1cd41c7fc8437598f73938ae3f2a1d P cae31ce250aa4d64953e14da451c568c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d125d3941ab94bd5b970110e5791e171 (127.0.99.131:44747): Illegal state: must be running to vote when last-logged opid is not known
I20240503 01:58:48.192904 517 leader_election.cc:302] T 2f1cd41c7fc8437598f73938ae3f2a1d P cae31ce250aa4d64953e14da451c568c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cae31ce250aa4d64953e14da451c568c; no voters: 82c39407898d4033af3c6a1a688a5197, d125d3941ab94bd5b970110e5791e171
I20240503 01:58:48.193404 909 tablet_bootstrap.cc:492] T 5a547015b4514b8286adb5c2e6356ba1 P d125d3941ab94bd5b970110e5791e171: No bootstrap required, opened a new log
I20240503 01:58:48.193432 914 raft_consensus.cc:2726] T 2f1cd41c7fc8437598f73938ae3f2a1d P cae31ce250aa4d64953e14da451c568c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 01:58:48.193594 909 ts_tablet_manager.cc:1367] T 5a547015b4514b8286adb5c2e6356ba1 P d125d3941ab94bd5b970110e5791e171: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20240503 01:58:48.193825 910 tablet_bootstrap.cc:654] T 84bfa8f64219448d8d49a30d50068c22 P cae31ce250aa4d64953e14da451c568c: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:48.194224 920 raft_consensus.cc:348] T c54888ed43eb4ab4bc79a9ab0a2bcf51 P 82c39407898d4033af3c6a1a688a5197 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.194571 920 raft_consensus.cc:374] T c54888ed43eb4ab4bc79a9ab0a2bcf51 P 82c39407898d4033af3c6a1a688a5197 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:48.194687 920 raft_consensus.cc:729] T c54888ed43eb4ab4bc79a9ab0a2bcf51 P 82c39407898d4033af3c6a1a688a5197 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82c39407898d4033af3c6a1a688a5197, State: Initialized, Role: FOLLOWER
I20240503 01:58:48.195194 920 consensus_queue.cc:260] T c54888ed43eb4ab4bc79a9ab0a2bcf51 P 82c39407898d4033af3c6a1a688a5197 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.195854 910 tablet_bootstrap.cc:492] T 84bfa8f64219448d8d49a30d50068c22 P cae31ce250aa4d64953e14da451c568c: No bootstrap required, opened a new log
I20240503 01:58:48.195995 910 ts_tablet_manager.cc:1367] T 84bfa8f64219448d8d49a30d50068c22 P cae31ce250aa4d64953e14da451c568c: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 01:58:48.196190 920 ts_tablet_manager.cc:1398] T c54888ed43eb4ab4bc79a9ab0a2bcf51 P 82c39407898d4033af3c6a1a688a5197: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240503 01:58:48.196573 920 tablet_bootstrap.cc:492] T 5a547015b4514b8286adb5c2e6356ba1 P 82c39407898d4033af3c6a1a688a5197: Bootstrap starting.
I20240503 01:58:48.196542 910 raft_consensus.cc:348] T 84bfa8f64219448d8d49a30d50068c22 P cae31ce250aa4d64953e14da451c568c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.196637 910 raft_consensus.cc:374] T 84bfa8f64219448d8d49a30d50068c22 P cae31ce250aa4d64953e14da451c568c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:48.196764 910 raft_consensus.cc:729] T 84bfa8f64219448d8d49a30d50068c22 P cae31ce250aa4d64953e14da451c568c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cae31ce250aa4d64953e14da451c568c, State: Initialized, Role: FOLLOWER
I20240503 01:58:48.196926 910 consensus_queue.cc:260] T 84bfa8f64219448d8d49a30d50068c22 P cae31ce250aa4d64953e14da451c568c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.196911 909 raft_consensus.cc:348] T 5a547015b4514b8286adb5c2e6356ba1 P d125d3941ab94bd5b970110e5791e171 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.197173 909 raft_consensus.cc:374] T 5a547015b4514b8286adb5c2e6356ba1 P d125d3941ab94bd5b970110e5791e171 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:48.197265 909 raft_consensus.cc:729] T 5a547015b4514b8286adb5c2e6356ba1 P d125d3941ab94bd5b970110e5791e171 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d125d3941ab94bd5b970110e5791e171, State: Initialized, Role: FOLLOWER
I20240503 01:58:48.197371 910 ts_tablet_manager.cc:1398] T 84bfa8f64219448d8d49a30d50068c22 P cae31ce250aa4d64953e14da451c568c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:58:48.197822 910 tablet_bootstrap.cc:492] T d2a69cc1ebd8418f85677c18ef9d80ef P cae31ce250aa4d64953e14da451c568c: Bootstrap starting.
I20240503 01:58:48.197844 909 consensus_queue.cc:260] T 5a547015b4514b8286adb5c2e6356ba1 P d125d3941ab94bd5b970110e5791e171 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.198130 920 tablet_bootstrap.cc:654] T 5a547015b4514b8286adb5c2e6356ba1 P 82c39407898d4033af3c6a1a688a5197: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:48.198945 910 tablet_bootstrap.cc:654] T d2a69cc1ebd8418f85677c18ef9d80ef P cae31ce250aa4d64953e14da451c568c: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:48.199028 909 ts_tablet_manager.cc:1398] T 5a547015b4514b8286adb5c2e6356ba1 P d125d3941ab94bd5b970110e5791e171: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240503 01:58:48.199770 920 tablet_bootstrap.cc:492] T 5a547015b4514b8286adb5c2e6356ba1 P 82c39407898d4033af3c6a1a688a5197: No bootstrap required, opened a new log
I20240503 01:58:48.199853 920 ts_tablet_manager.cc:1367] T 5a547015b4514b8286adb5c2e6356ba1 P 82c39407898d4033af3c6a1a688a5197: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20240503 01:58:48.200331 910 tablet_bootstrap.cc:492] T d2a69cc1ebd8418f85677c18ef9d80ef P cae31ce250aa4d64953e14da451c568c: No bootstrap required, opened a new log
I20240503 01:58:48.200407 910 ts_tablet_manager.cc:1367] T d2a69cc1ebd8418f85677c18ef9d80ef P cae31ce250aa4d64953e14da451c568c: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 01:58:48.200517 909 tablet_bootstrap.cc:492] T c54888ed43eb4ab4bc79a9ab0a2bcf51 P d125d3941ab94bd5b970110e5791e171: Bootstrap starting.
I20240503 01:58:48.200515 920 raft_consensus.cc:348] T 5a547015b4514b8286adb5c2e6356ba1 P 82c39407898d4033af3c6a1a688a5197 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.200608 920 raft_consensus.cc:374] T 5a547015b4514b8286adb5c2e6356ba1 P 82c39407898d4033af3c6a1a688a5197 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:48.200637 920 raft_consensus.cc:729] T 5a547015b4514b8286adb5c2e6356ba1 P 82c39407898d4033af3c6a1a688a5197 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82c39407898d4033af3c6a1a688a5197, State: Initialized, Role: FOLLOWER
I20240503 01:58:48.200937 920 consensus_queue.cc:260] T 5a547015b4514b8286adb5c2e6356ba1 P 82c39407898d4033af3c6a1a688a5197 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.200929 910 raft_consensus.cc:348] T d2a69cc1ebd8418f85677c18ef9d80ef P cae31ce250aa4d64953e14da451c568c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } } peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } }
I20240503 01:58:48.201036 910 raft_consensus.cc:374] T d2a69cc1ebd8418f85677c18ef9d80ef P cae31ce250aa4d64953e14da451c568c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:48.201100 910 raft_consensus.cc:729] T d2a69cc1ebd8418f85677c18ef9d80ef P cae31ce250aa4d64953e14da451c568c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cae31ce250aa4d64953e14da451c568c, State: Initialized, Role: FOLLOWER
I20240503 01:58:48.201236 910 consensus_queue.cc:260] T d2a69cc1ebd8418f85677c18ef9d80ef P cae31ce250aa4d64953e14da451c568c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } } peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } }
I20240503 01:58:48.201483 910 ts_tablet_manager.cc:1398] T d2a69cc1ebd8418f85677c18ef9d80ef P cae31ce250aa4d64953e14da451c568c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:58:48.201581 920 ts_tablet_manager.cc:1398] T 5a547015b4514b8286adb5c2e6356ba1 P 82c39407898d4033af3c6a1a688a5197: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 01:58:48.201656 909 tablet_bootstrap.cc:654] T c54888ed43eb4ab4bc79a9ab0a2bcf51 P d125d3941ab94bd5b970110e5791e171: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:48.201712 910 tablet_bootstrap.cc:492] T e38e1de36d82402c82b72dce9e765505 P cae31ce250aa4d64953e14da451c568c: Bootstrap starting.
I20240503 01:58:48.201795 920 tablet_bootstrap.cc:492] T ccaa4613da8546f88fb844a6711baabc P 82c39407898d4033af3c6a1a688a5197: Bootstrap starting.
I20240503 01:58:48.202908 910 tablet_bootstrap.cc:654] T e38e1de36d82402c82b72dce9e765505 P cae31ce250aa4d64953e14da451c568c: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:48.203217 909 tablet_bootstrap.cc:492] T c54888ed43eb4ab4bc79a9ab0a2bcf51 P d125d3941ab94bd5b970110e5791e171: No bootstrap required, opened a new log
I20240503 01:58:48.203301 909 ts_tablet_manager.cc:1367] T c54888ed43eb4ab4bc79a9ab0a2bcf51 P d125d3941ab94bd5b970110e5791e171: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:58:48.203430 920 tablet_bootstrap.cc:654] T ccaa4613da8546f88fb844a6711baabc P 82c39407898d4033af3c6a1a688a5197: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:48.203989 909 raft_consensus.cc:348] T c54888ed43eb4ab4bc79a9ab0a2bcf51 P d125d3941ab94bd5b970110e5791e171 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.204083 909 raft_consensus.cc:374] T c54888ed43eb4ab4bc79a9ab0a2bcf51 P d125d3941ab94bd5b970110e5791e171 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:48.204114 909 raft_consensus.cc:729] T c54888ed43eb4ab4bc79a9ab0a2bcf51 P d125d3941ab94bd5b970110e5791e171 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d125d3941ab94bd5b970110e5791e171, State: Initialized, Role: FOLLOWER
I20240503 01:58:48.204226 909 consensus_queue.cc:260] T c54888ed43eb4ab4bc79a9ab0a2bcf51 P d125d3941ab94bd5b970110e5791e171 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.204576 910 tablet_bootstrap.cc:492] T e38e1de36d82402c82b72dce9e765505 P cae31ce250aa4d64953e14da451c568c: No bootstrap required, opened a new log
I20240503 01:58:48.204638 910 ts_tablet_manager.cc:1367] T e38e1de36d82402c82b72dce9e765505 P cae31ce250aa4d64953e14da451c568c: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240503 01:58:48.204684 909 ts_tablet_manager.cc:1398] T c54888ed43eb4ab4bc79a9ab0a2bcf51 P d125d3941ab94bd5b970110e5791e171: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:58:48.204856 909 tablet_bootstrap.cc:492] T 84bfa8f64219448d8d49a30d50068c22 P d125d3941ab94bd5b970110e5791e171: Bootstrap starting.
I20240503 01:58:48.205200 910 raft_consensus.cc:348] T e38e1de36d82402c82b72dce9e765505 P cae31ce250aa4d64953e14da451c568c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.205279 910 raft_consensus.cc:374] T e38e1de36d82402c82b72dce9e765505 P cae31ce250aa4d64953e14da451c568c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:48.205298 910 raft_consensus.cc:729] T e38e1de36d82402c82b72dce9e765505 P cae31ce250aa4d64953e14da451c568c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cae31ce250aa4d64953e14da451c568c, State: Initialized, Role: FOLLOWER
I20240503 01:58:48.205441 910 consensus_queue.cc:260] T e38e1de36d82402c82b72dce9e765505 P cae31ce250aa4d64953e14da451c568c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.205672 910 ts_tablet_manager.cc:1398] T e38e1de36d82402c82b72dce9e765505 P cae31ce250aa4d64953e14da451c568c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:58:48.205840 910 tablet_bootstrap.cc:492] T 5a547015b4514b8286adb5c2e6356ba1 P cae31ce250aa4d64953e14da451c568c: Bootstrap starting.
I20240503 01:58:48.206077 909 tablet_bootstrap.cc:654] T 84bfa8f64219448d8d49a30d50068c22 P d125d3941ab94bd5b970110e5791e171: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:48.207356 910 tablet_bootstrap.cc:654] T 5a547015b4514b8286adb5c2e6356ba1 P cae31ce250aa4d64953e14da451c568c: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:48.208289 909 tablet_bootstrap.cc:492] T 84bfa8f64219448d8d49a30d50068c22 P d125d3941ab94bd5b970110e5791e171: No bootstrap required, opened a new log
I20240503 01:58:48.208395 909 ts_tablet_manager.cc:1367] T 84bfa8f64219448d8d49a30d50068c22 P d125d3941ab94bd5b970110e5791e171: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 01:58:48.208953 909 raft_consensus.cc:348] T 84bfa8f64219448d8d49a30d50068c22 P d125d3941ab94bd5b970110e5791e171 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.209059 909 raft_consensus.cc:374] T 84bfa8f64219448d8d49a30d50068c22 P d125d3941ab94bd5b970110e5791e171 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:48.209100 909 raft_consensus.cc:729] T 84bfa8f64219448d8d49a30d50068c22 P d125d3941ab94bd5b970110e5791e171 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d125d3941ab94bd5b970110e5791e171, State: Initialized, Role: FOLLOWER
I20240503 01:58:48.209223 909 consensus_queue.cc:260] T 84bfa8f64219448d8d49a30d50068c22 P d125d3941ab94bd5b970110e5791e171 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.209304 920 tablet_bootstrap.cc:492] T ccaa4613da8546f88fb844a6711baabc P 82c39407898d4033af3c6a1a688a5197: No bootstrap required, opened a new log
I20240503 01:58:48.209373 910 tablet_bootstrap.cc:492] T 5a547015b4514b8286adb5c2e6356ba1 P cae31ce250aa4d64953e14da451c568c: No bootstrap required, opened a new log
I20240503 01:58:48.209400 920 ts_tablet_manager.cc:1367] T ccaa4613da8546f88fb844a6711baabc P 82c39407898d4033af3c6a1a688a5197: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.002s
I20240503 01:58:48.209450 910 ts_tablet_manager.cc:1367] T 5a547015b4514b8286adb5c2e6356ba1 P cae31ce250aa4d64953e14da451c568c: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240503 01:58:48.209479 909 ts_tablet_manager.cc:1398] T 84bfa8f64219448d8d49a30d50068c22 P d125d3941ab94bd5b970110e5791e171: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:58:48.209642 909 tablet_bootstrap.cc:492] T 2f1cd41c7fc8437598f73938ae3f2a1d P d125d3941ab94bd5b970110e5791e171: Bootstrap starting.
I20240503 01:58:48.209932 920 raft_consensus.cc:348] T ccaa4613da8546f88fb844a6711baabc P 82c39407898d4033af3c6a1a688a5197 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.210049 920 raft_consensus.cc:374] T ccaa4613da8546f88fb844a6711baabc P 82c39407898d4033af3c6a1a688a5197 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:48.210001 910 raft_consensus.cc:348] T 5a547015b4514b8286adb5c2e6356ba1 P cae31ce250aa4d64953e14da451c568c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.210091 920 raft_consensus.cc:729] T ccaa4613da8546f88fb844a6711baabc P 82c39407898d4033af3c6a1a688a5197 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82c39407898d4033af3c6a1a688a5197, State: Initialized, Role: FOLLOWER
I20240503 01:58:48.210116 910 raft_consensus.cc:374] T 5a547015b4514b8286adb5c2e6356ba1 P cae31ce250aa4d64953e14da451c568c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:48.210155 910 raft_consensus.cc:729] T 5a547015b4514b8286adb5c2e6356ba1 P cae31ce250aa4d64953e14da451c568c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cae31ce250aa4d64953e14da451c568c, State: Initialized, Role: FOLLOWER
I20240503 01:58:48.210240 920 consensus_queue.cc:260] T ccaa4613da8546f88fb844a6711baabc P 82c39407898d4033af3c6a1a688a5197 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.210312 910 consensus_queue.cc:260] T 5a547015b4514b8286adb5c2e6356ba1 P cae31ce250aa4d64953e14da451c568c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.210532 920 ts_tablet_manager.cc:1398] T ccaa4613da8546f88fb844a6711baabc P 82c39407898d4033af3c6a1a688a5197: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:58:48.210564 910 ts_tablet_manager.cc:1398] T 5a547015b4514b8286adb5c2e6356ba1 P cae31ce250aa4d64953e14da451c568c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:58:48.210778 910 tablet_bootstrap.cc:492] T 8d1ad90f476a4b2b97f59259ef4bff9f P cae31ce250aa4d64953e14da451c568c: Bootstrap starting.
I20240503 01:58:48.210880 909 tablet_bootstrap.cc:654] T 2f1cd41c7fc8437598f73938ae3f2a1d P d125d3941ab94bd5b970110e5791e171: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:48.211659 920 tablet_bootstrap.cc:492] T 9f1fb01218234ec4a13fd6f9dfb3fdd6 P 82c39407898d4033af3c6a1a688a5197: Bootstrap starting.
I20240503 01:58:48.211936 910 tablet_bootstrap.cc:654] T 8d1ad90f476a4b2b97f59259ef4bff9f P cae31ce250aa4d64953e14da451c568c: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:48.212810 909 tablet_bootstrap.cc:492] T 2f1cd41c7fc8437598f73938ae3f2a1d P d125d3941ab94bd5b970110e5791e171: No bootstrap required, opened a new log
I20240503 01:58:48.212900 909 ts_tablet_manager.cc:1367] T 2f1cd41c7fc8437598f73938ae3f2a1d P d125d3941ab94bd5b970110e5791e171: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:58:48.213052 920 tablet_bootstrap.cc:654] T 9f1fb01218234ec4a13fd6f9dfb3fdd6 P 82c39407898d4033af3c6a1a688a5197: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:48.213397 910 tablet_bootstrap.cc:492] T 8d1ad90f476a4b2b97f59259ef4bff9f P cae31ce250aa4d64953e14da451c568c: No bootstrap required, opened a new log
I20240503 01:58:48.213476 910 ts_tablet_manager.cc:1367] T 8d1ad90f476a4b2b97f59259ef4bff9f P cae31ce250aa4d64953e14da451c568c: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.002s
I20240503 01:58:48.213438 909 raft_consensus.cc:348] T 2f1cd41c7fc8437598f73938ae3f2a1d P d125d3941ab94bd5b970110e5791e171 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.213523 909 raft_consensus.cc:374] T 2f1cd41c7fc8437598f73938ae3f2a1d P d125d3941ab94bd5b970110e5791e171 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:48.213555 909 raft_consensus.cc:729] T 2f1cd41c7fc8437598f73938ae3f2a1d P d125d3941ab94bd5b970110e5791e171 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d125d3941ab94bd5b970110e5791e171, State: Initialized, Role: FOLLOWER
I20240503 01:58:48.213678 909 consensus_queue.cc:260] T 2f1cd41c7fc8437598f73938ae3f2a1d P d125d3941ab94bd5b970110e5791e171 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.213920 909 ts_tablet_manager.cc:1398] T 2f1cd41c7fc8437598f73938ae3f2a1d P d125d3941ab94bd5b970110e5791e171: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:58:48.213989 910 raft_consensus.cc:348] T 8d1ad90f476a4b2b97f59259ef4bff9f P cae31ce250aa4d64953e14da451c568c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.214071 909 tablet_bootstrap.cc:492] T 86f9a4762bf34b52a8d000a487226e5b P d125d3941ab94bd5b970110e5791e171: Bootstrap starting.
I20240503 01:58:48.214071 910 raft_consensus.cc:374] T 8d1ad90f476a4b2b97f59259ef4bff9f P cae31ce250aa4d64953e14da451c568c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:48.214115 910 raft_consensus.cc:729] T 8d1ad90f476a4b2b97f59259ef4bff9f P cae31ce250aa4d64953e14da451c568c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cae31ce250aa4d64953e14da451c568c, State: Initialized, Role: FOLLOWER
I20240503 01:58:48.214236 910 consensus_queue.cc:260] T 8d1ad90f476a4b2b97f59259ef4bff9f P cae31ce250aa4d64953e14da451c568c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.214598 910 ts_tablet_manager.cc:1398] T 8d1ad90f476a4b2b97f59259ef4bff9f P cae31ce250aa4d64953e14da451c568c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:58:48.214784 910 tablet_bootstrap.cc:492] T ccaa4613da8546f88fb844a6711baabc P cae31ce250aa4d64953e14da451c568c: Bootstrap starting.
I20240503 01:58:48.215081 909 tablet_bootstrap.cc:654] T 86f9a4762bf34b52a8d000a487226e5b P d125d3941ab94bd5b970110e5791e171: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:48.215920 910 tablet_bootstrap.cc:654] T ccaa4613da8546f88fb844a6711baabc P cae31ce250aa4d64953e14da451c568c: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:48.216914 909 tablet_bootstrap.cc:492] T 86f9a4762bf34b52a8d000a487226e5b P d125d3941ab94bd5b970110e5791e171: No bootstrap required, opened a new log
I20240503 01:58:48.216995 909 ts_tablet_manager.cc:1367] T 86f9a4762bf34b52a8d000a487226e5b P d125d3941ab94bd5b970110e5791e171: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 01:58:48.217519 909 raft_consensus.cc:348] T 86f9a4762bf34b52a8d000a487226e5b P d125d3941ab94bd5b970110e5791e171 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.217629 909 raft_consensus.cc:374] T 86f9a4762bf34b52a8d000a487226e5b P d125d3941ab94bd5b970110e5791e171 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:48.217665 909 raft_consensus.cc:729] T 86f9a4762bf34b52a8d000a487226e5b P d125d3941ab94bd5b970110e5791e171 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d125d3941ab94bd5b970110e5791e171, State: Initialized, Role: FOLLOWER
I20240503 01:58:48.217810 909 consensus_queue.cc:260] T 86f9a4762bf34b52a8d000a487226e5b P d125d3941ab94bd5b970110e5791e171 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.217862 910 tablet_bootstrap.cc:492] T ccaa4613da8546f88fb844a6711baabc P cae31ce250aa4d64953e14da451c568c: No bootstrap required, opened a new log
I20240503 01:58:48.217955 910 ts_tablet_manager.cc:1367] T ccaa4613da8546f88fb844a6711baabc P cae31ce250aa4d64953e14da451c568c: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240503 01:58:48.218111 909 ts_tablet_manager.cc:1398] T 86f9a4762bf34b52a8d000a487226e5b P d125d3941ab94bd5b970110e5791e171: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:58:48.218282 909 tablet_bootstrap.cc:492] T a7090b6ae5394aecb146a7fa25b89e79 P d125d3941ab94bd5b970110e5791e171: Bootstrap starting.
I20240503 01:58:48.218537 920 tablet_bootstrap.cc:492] T 9f1fb01218234ec4a13fd6f9dfb3fdd6 P 82c39407898d4033af3c6a1a688a5197: No bootstrap required, opened a new log
I20240503 01:58:48.218636 920 ts_tablet_manager.cc:1367] T 9f1fb01218234ec4a13fd6f9dfb3fdd6 P 82c39407898d4033af3c6a1a688a5197: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 01:58:48.218727 910 raft_consensus.cc:348] T ccaa4613da8546f88fb844a6711baabc P cae31ce250aa4d64953e14da451c568c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.218861 910 raft_consensus.cc:374] T ccaa4613da8546f88fb844a6711baabc P cae31ce250aa4d64953e14da451c568c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:48.218884 910 raft_consensus.cc:729] T ccaa4613da8546f88fb844a6711baabc P cae31ce250aa4d64953e14da451c568c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cae31ce250aa4d64953e14da451c568c, State: Initialized, Role: FOLLOWER
I20240503 01:58:48.218992 910 consensus_queue.cc:260] T ccaa4613da8546f88fb844a6711baabc P cae31ce250aa4d64953e14da451c568c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.219199 910 ts_tablet_manager.cc:1398] T ccaa4613da8546f88fb844a6711baabc P cae31ce250aa4d64953e14da451c568c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:58:48.219342 910 tablet_bootstrap.cc:492] T 2e5dc6a7de704dcdb0aec15124300b61 P cae31ce250aa4d64953e14da451c568c: Bootstrap starting.
I20240503 01:58:48.219375 909 tablet_bootstrap.cc:654] T a7090b6ae5394aecb146a7fa25b89e79 P d125d3941ab94bd5b970110e5791e171: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:48.219493 920 raft_consensus.cc:348] T 9f1fb01218234ec4a13fd6f9dfb3fdd6 P 82c39407898d4033af3c6a1a688a5197 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.219601 920 raft_consensus.cc:374] T 9f1fb01218234ec4a13fd6f9dfb3fdd6 P 82c39407898d4033af3c6a1a688a5197 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:48.219633 920 raft_consensus.cc:729] T 9f1fb01218234ec4a13fd6f9dfb3fdd6 P 82c39407898d4033af3c6a1a688a5197 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82c39407898d4033af3c6a1a688a5197, State: Initialized, Role: FOLLOWER
I20240503 01:58:48.219794 920 consensus_queue.cc:260] T 9f1fb01218234ec4a13fd6f9dfb3fdd6 P 82c39407898d4033af3c6a1a688a5197 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.220671 910 tablet_bootstrap.cc:654] T 2e5dc6a7de704dcdb0aec15124300b61 P cae31ce250aa4d64953e14da451c568c: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:48.221916 920 ts_tablet_manager.cc:1398] T 9f1fb01218234ec4a13fd6f9dfb3fdd6 P 82c39407898d4033af3c6a1a688a5197: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 01:58:48.222201 920 tablet_bootstrap.cc:492] T 2e5dc6a7de704dcdb0aec15124300b61 P 82c39407898d4033af3c6a1a688a5197: Bootstrap starting.
I20240503 01:58:48.222689 910 tablet_bootstrap.cc:492] T 2e5dc6a7de704dcdb0aec15124300b61 P cae31ce250aa4d64953e14da451c568c: No bootstrap required, opened a new log
I20240503 01:58:48.222822 910 ts_tablet_manager.cc:1367] T 2e5dc6a7de704dcdb0aec15124300b61 P cae31ce250aa4d64953e14da451c568c: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 01:58:48.222944 909 tablet_bootstrap.cc:492] T a7090b6ae5394aecb146a7fa25b89e79 P d125d3941ab94bd5b970110e5791e171: No bootstrap required, opened a new log
I20240503 01:58:48.223065 909 ts_tablet_manager.cc:1367] T a7090b6ae5394aecb146a7fa25b89e79 P d125d3941ab94bd5b970110e5791e171: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 01:58:48.223451 910 raft_consensus.cc:348] T 2e5dc6a7de704dcdb0aec15124300b61 P cae31ce250aa4d64953e14da451c568c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.223557 920 tablet_bootstrap.cc:654] T 2e5dc6a7de704dcdb0aec15124300b61 P 82c39407898d4033af3c6a1a688a5197: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:48.223618 910 raft_consensus.cc:374] T 2e5dc6a7de704dcdb0aec15124300b61 P cae31ce250aa4d64953e14da451c568c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:48.223687 910 raft_consensus.cc:729] T 2e5dc6a7de704dcdb0aec15124300b61 P cae31ce250aa4d64953e14da451c568c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cae31ce250aa4d64953e14da451c568c, State: Initialized, Role: FOLLOWER
I20240503 01:58:48.223875 910 consensus_queue.cc:260] T 2e5dc6a7de704dcdb0aec15124300b61 P cae31ce250aa4d64953e14da451c568c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.224074 909 raft_consensus.cc:348] T a7090b6ae5394aecb146a7fa25b89e79 P d125d3941ab94bd5b970110e5791e171 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.224180 909 raft_consensus.cc:374] T a7090b6ae5394aecb146a7fa25b89e79 P d125d3941ab94bd5b970110e5791e171 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:48.224201 909 raft_consensus.cc:729] T a7090b6ae5394aecb146a7fa25b89e79 P d125d3941ab94bd5b970110e5791e171 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d125d3941ab94bd5b970110e5791e171, State: Initialized, Role: FOLLOWER
I20240503 01:58:48.224284 909 consensus_queue.cc:260] T a7090b6ae5394aecb146a7fa25b89e79 P d125d3941ab94bd5b970110e5791e171 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.224517 909 ts_tablet_manager.cc:1398] T a7090b6ae5394aecb146a7fa25b89e79 P d125d3941ab94bd5b970110e5791e171: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 01:58:48.224710 909 tablet_bootstrap.cc:492] T 0488ca07075c4daca7b8f4d5f827ea56 P d125d3941ab94bd5b970110e5791e171: Bootstrap starting.
I20240503 01:58:48.224778 910 ts_tablet_manager.cc:1398] T 2e5dc6a7de704dcdb0aec15124300b61 P cae31ce250aa4d64953e14da451c568c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:58:48.224959 910 tablet_bootstrap.cc:492] T 1581a7ae381c4228bbb4420b84ba432f P cae31ce250aa4d64953e14da451c568c: Bootstrap starting.
I20240503 01:58:48.225944 920 tablet_bootstrap.cc:492] T 2e5dc6a7de704dcdb0aec15124300b61 P 82c39407898d4033af3c6a1a688a5197: No bootstrap required, opened a new log
I20240503 01:58:48.226138 909 tablet_bootstrap.cc:654] T 0488ca07075c4daca7b8f4d5f827ea56 P d125d3941ab94bd5b970110e5791e171: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:48.226110 920 ts_tablet_manager.cc:1367] T 2e5dc6a7de704dcdb0aec15124300b61 P 82c39407898d4033af3c6a1a688a5197: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20240503 01:58:48.226495 910 tablet_bootstrap.cc:654] T 1581a7ae381c4228bbb4420b84ba432f P cae31ce250aa4d64953e14da451c568c: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:48.226845 920 raft_consensus.cc:348] T 2e5dc6a7de704dcdb0aec15124300b61 P 82c39407898d4033af3c6a1a688a5197 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.227001 920 raft_consensus.cc:374] T 2e5dc6a7de704dcdb0aec15124300b61 P 82c39407898d4033af3c6a1a688a5197 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:48.227069 920 raft_consensus.cc:729] T 2e5dc6a7de704dcdb0aec15124300b61 P 82c39407898d4033af3c6a1a688a5197 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82c39407898d4033af3c6a1a688a5197, State: Initialized, Role: FOLLOWER
I20240503 01:58:48.227659 920 consensus_queue.cc:260] T 2e5dc6a7de704dcdb0aec15124300b61 P 82c39407898d4033af3c6a1a688a5197 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.228022 920 ts_tablet_manager.cc:1398] T 2e5dc6a7de704dcdb0aec15124300b61 P 82c39407898d4033af3c6a1a688a5197: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:58:48.228075 910 tablet_bootstrap.cc:492] T 1581a7ae381c4228bbb4420b84ba432f P cae31ce250aa4d64953e14da451c568c: No bootstrap required, opened a new log
I20240503 01:58:48.228188 910 ts_tablet_manager.cc:1367] T 1581a7ae381c4228bbb4420b84ba432f P cae31ce250aa4d64953e14da451c568c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:58:48.228725 920 tablet_bootstrap.cc:492] T a7090b6ae5394aecb146a7fa25b89e79 P 82c39407898d4033af3c6a1a688a5197: Bootstrap starting.
I20240503 01:58:48.228796 910 raft_consensus.cc:348] T 1581a7ae381c4228bbb4420b84ba432f P cae31ce250aa4d64953e14da451c568c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.228905 910 raft_consensus.cc:374] T 1581a7ae381c4228bbb4420b84ba432f P cae31ce250aa4d64953e14da451c568c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:48.228950 910 raft_consensus.cc:729] T 1581a7ae381c4228bbb4420b84ba432f P cae31ce250aa4d64953e14da451c568c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cae31ce250aa4d64953e14da451c568c, State: Initialized, Role: FOLLOWER
I20240503 01:58:48.229130 909 tablet_bootstrap.cc:492] T 0488ca07075c4daca7b8f4d5f827ea56 P d125d3941ab94bd5b970110e5791e171: No bootstrap required, opened a new log
I20240503 01:58:48.229097 910 consensus_queue.cc:260] T 1581a7ae381c4228bbb4420b84ba432f P cae31ce250aa4d64953e14da451c568c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.229226 909 ts_tablet_manager.cc:1367] T 0488ca07075c4daca7b8f4d5f827ea56 P d125d3941ab94bd5b970110e5791e171: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 01:58:48.229380 910 ts_tablet_manager.cc:1398] T 1581a7ae381c4228bbb4420b84ba432f P cae31ce250aa4d64953e14da451c568c: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 01:58:48.229538 910 tablet_bootstrap.cc:492] T 42a69d697389414a973d995a763d8219 P cae31ce250aa4d64953e14da451c568c: Bootstrap starting.
I20240503 01:58:48.229846 909 raft_consensus.cc:348] T 0488ca07075c4daca7b8f4d5f827ea56 P d125d3941ab94bd5b970110e5791e171 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.229955 909 raft_consensus.cc:374] T 0488ca07075c4daca7b8f4d5f827ea56 P d125d3941ab94bd5b970110e5791e171 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:48.229988 909 raft_consensus.cc:729] T 0488ca07075c4daca7b8f4d5f827ea56 P d125d3941ab94bd5b970110e5791e171 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d125d3941ab94bd5b970110e5791e171, State: Initialized, Role: FOLLOWER
I20240503 01:58:48.230166 909 consensus_queue.cc:260] T 0488ca07075c4daca7b8f4d5f827ea56 P d125d3941ab94bd5b970110e5791e171 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.230566 909 ts_tablet_manager.cc:1398] T 0488ca07075c4daca7b8f4d5f827ea56 P d125d3941ab94bd5b970110e5791e171: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 01:58:48.230798 910 tablet_bootstrap.cc:654] T 42a69d697389414a973d995a763d8219 P cae31ce250aa4d64953e14da451c568c: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:48.231137 909 tablet_bootstrap.cc:492] T 42a69d697389414a973d995a763d8219 P d125d3941ab94bd5b970110e5791e171: Bootstrap starting.
I20240503 01:58:48.231176 920 tablet_bootstrap.cc:654] T a7090b6ae5394aecb146a7fa25b89e79 P 82c39407898d4033af3c6a1a688a5197: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:48.232681 909 tablet_bootstrap.cc:654] T 42a69d697389414a973d995a763d8219 P d125d3941ab94bd5b970110e5791e171: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:48.233505 920 tablet_bootstrap.cc:492] T a7090b6ae5394aecb146a7fa25b89e79 P 82c39407898d4033af3c6a1a688a5197: No bootstrap required, opened a new log
I20240503 01:58:48.233666 920 ts_tablet_manager.cc:1367] T a7090b6ae5394aecb146a7fa25b89e79 P 82c39407898d4033af3c6a1a688a5197: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 01:58:48.234071 910 tablet_bootstrap.cc:492] T 42a69d697389414a973d995a763d8219 P cae31ce250aa4d64953e14da451c568c: No bootstrap required, opened a new log
I20240503 01:58:48.234171 909 tablet_bootstrap.cc:492] T 42a69d697389414a973d995a763d8219 P d125d3941ab94bd5b970110e5791e171: No bootstrap required, opened a new log
I20240503 01:58:48.234176 910 ts_tablet_manager.cc:1367] T 42a69d697389414a973d995a763d8219 P cae31ce250aa4d64953e14da451c568c: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 01:58:48.234256 909 ts_tablet_manager.cc:1367] T 42a69d697389414a973d995a763d8219 P d125d3941ab94bd5b970110e5791e171: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.002s
I20240503 01:58:48.234257 920 raft_consensus.cc:348] T a7090b6ae5394aecb146a7fa25b89e79 P 82c39407898d4033af3c6a1a688a5197 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.234416 920 raft_consensus.cc:374] T a7090b6ae5394aecb146a7fa25b89e79 P 82c39407898d4033af3c6a1a688a5197 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:48.234464 920 raft_consensus.cc:729] T a7090b6ae5394aecb146a7fa25b89e79 P 82c39407898d4033af3c6a1a688a5197 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82c39407898d4033af3c6a1a688a5197, State: Initialized, Role: FOLLOWER
I20240503 01:58:48.234601 920 consensus_queue.cc:260] T a7090b6ae5394aecb146a7fa25b89e79 P 82c39407898d4033af3c6a1a688a5197 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.234745 909 raft_consensus.cc:348] T 42a69d697389414a973d995a763d8219 P d125d3941ab94bd5b970110e5791e171 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.234745 910 raft_consensus.cc:348] T 42a69d697389414a973d995a763d8219 P cae31ce250aa4d64953e14da451c568c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.234856 910 raft_consensus.cc:374] T 42a69d697389414a973d995a763d8219 P cae31ce250aa4d64953e14da451c568c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:48.234856 909 raft_consensus.cc:374] T 42a69d697389414a973d995a763d8219 P d125d3941ab94bd5b970110e5791e171 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:48.234850 920 ts_tablet_manager.cc:1398] T a7090b6ae5394aecb146a7fa25b89e79 P 82c39407898d4033af3c6a1a688a5197: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:58:48.234894 909 raft_consensus.cc:729] T 42a69d697389414a973d995a763d8219 P d125d3941ab94bd5b970110e5791e171 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d125d3941ab94bd5b970110e5791e171, State: Initialized, Role: FOLLOWER
I20240503 01:58:48.234894 910 raft_consensus.cc:729] T 42a69d697389414a973d995a763d8219 P cae31ce250aa4d64953e14da451c568c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cae31ce250aa4d64953e14da451c568c, State: Initialized, Role: FOLLOWER
I20240503 01:58:48.235011 910 consensus_queue.cc:260] T 42a69d697389414a973d995a763d8219 P cae31ce250aa4d64953e14da451c568c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.235047 909 consensus_queue.cc:260] T 42a69d697389414a973d995a763d8219 P d125d3941ab94bd5b970110e5791e171 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.235292 910 ts_tablet_manager.cc:1398] T 42a69d697389414a973d995a763d8219 P cae31ce250aa4d64953e14da451c568c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:58:48.235291 909 ts_tablet_manager.cc:1398] T 42a69d697389414a973d995a763d8219 P d125d3941ab94bd5b970110e5791e171: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:58:48.235459 910 tablet_bootstrap.cc:492] T 9f1fb01218234ec4a13fd6f9dfb3fdd6 P cae31ce250aa4d64953e14da451c568c: Bootstrap starting.
I20240503 01:58:48.235459 909 tablet_bootstrap.cc:492] T e0970cda8db54a8593f7d482e861da46 P d125d3941ab94bd5b970110e5791e171: Bootstrap starting.
I20240503 01:58:48.236485 920 tablet_bootstrap.cc:492] T e0970cda8db54a8593f7d482e861da46 P 82c39407898d4033af3c6a1a688a5197: Bootstrap starting.
I20240503 01:58:48.236693 909 tablet_bootstrap.cc:654] T e0970cda8db54a8593f7d482e861da46 P d125d3941ab94bd5b970110e5791e171: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:48.236732 910 tablet_bootstrap.cc:654] T 9f1fb01218234ec4a13fd6f9dfb3fdd6 P cae31ce250aa4d64953e14da451c568c: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:48.238430 920 tablet_bootstrap.cc:654] T e0970cda8db54a8593f7d482e861da46 P 82c39407898d4033af3c6a1a688a5197: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:48.238991 909 tablet_bootstrap.cc:492] T e0970cda8db54a8593f7d482e861da46 P d125d3941ab94bd5b970110e5791e171: No bootstrap required, opened a new log
I20240503 01:58:48.238994 910 tablet_bootstrap.cc:492] T 9f1fb01218234ec4a13fd6f9dfb3fdd6 P cae31ce250aa4d64953e14da451c568c: No bootstrap required, opened a new log
I20240503 01:58:48.239073 909 ts_tablet_manager.cc:1367] T e0970cda8db54a8593f7d482e861da46 P d125d3941ab94bd5b970110e5791e171: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240503 01:58:48.239089 910 ts_tablet_manager.cc:1367] T 9f1fb01218234ec4a13fd6f9dfb3fdd6 P cae31ce250aa4d64953e14da451c568c: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240503 01:58:48.239593 910 raft_consensus.cc:348] T 9f1fb01218234ec4a13fd6f9dfb3fdd6 P cae31ce250aa4d64953e14da451c568c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.239737 910 raft_consensus.cc:374] T 9f1fb01218234ec4a13fd6f9dfb3fdd6 P cae31ce250aa4d64953e14da451c568c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:48.239774 910 raft_consensus.cc:729] T 9f1fb01218234ec4a13fd6f9dfb3fdd6 P cae31ce250aa4d64953e14da451c568c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cae31ce250aa4d64953e14da451c568c, State: Initialized, Role: FOLLOWER
I20240503 01:58:48.239706 909 raft_consensus.cc:348] T e0970cda8db54a8593f7d482e861da46 P d125d3941ab94bd5b970110e5791e171 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.239859 909 raft_consensus.cc:374] T e0970cda8db54a8593f7d482e861da46 P d125d3941ab94bd5b970110e5791e171 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:48.239895 909 raft_consensus.cc:729] T e0970cda8db54a8593f7d482e861da46 P d125d3941ab94bd5b970110e5791e171 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d125d3941ab94bd5b970110e5791e171, State: Initialized, Role: FOLLOWER
I20240503 01:58:48.239902 910 consensus_queue.cc:260] T 9f1fb01218234ec4a13fd6f9dfb3fdd6 P cae31ce250aa4d64953e14da451c568c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.239992 909 consensus_queue.cc:260] T e0970cda8db54a8593f7d482e861da46 P d125d3941ab94bd5b970110e5791e171 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.240173 910 ts_tablet_manager.cc:1398] T 9f1fb01218234ec4a13fd6f9dfb3fdd6 P cae31ce250aa4d64953e14da451c568c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:58:48.240187 935 raft_consensus.cc:482] T 86f9a4762bf34b52a8d000a487226e5b P d125d3941ab94bd5b970110e5791e171 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:58:48.240286 909 ts_tablet_manager.cc:1398] T e0970cda8db54a8593f7d482e861da46 P d125d3941ab94bd5b970110e5791e171: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:58:48.240401 920 tablet_bootstrap.cc:492] T e0970cda8db54a8593f7d482e861da46 P 82c39407898d4033af3c6a1a688a5197: No bootstrap required, opened a new log
I20240503 01:58:48.240406 935 raft_consensus.cc:504] T 86f9a4762bf34b52a8d000a487226e5b P d125d3941ab94bd5b970110e5791e171 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.240486 920 ts_tablet_manager.cc:1367] T e0970cda8db54a8593f7d482e861da46 P 82c39407898d4033af3c6a1a688a5197: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 01:58:48.241029 920 raft_consensus.cc:348] T e0970cda8db54a8593f7d482e861da46 P 82c39407898d4033af3c6a1a688a5197 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.241101 920 raft_consensus.cc:374] T e0970cda8db54a8593f7d482e861da46 P 82c39407898d4033af3c6a1a688a5197 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:48.241120 920 raft_consensus.cc:729] T e0970cda8db54a8593f7d482e861da46 P 82c39407898d4033af3c6a1a688a5197 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82c39407898d4033af3c6a1a688a5197, State: Initialized, Role: FOLLOWER
I20240503 01:58:48.241215 920 consensus_queue.cc:260] T e0970cda8db54a8593f7d482e861da46 P 82c39407898d4033af3c6a1a688a5197 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.241528 920 ts_tablet_manager.cc:1398] T e0970cda8db54a8593f7d482e861da46 P 82c39407898d4033af3c6a1a688a5197: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:58:48.241676 920 tablet_bootstrap.cc:492] T 84bfa8f64219448d8d49a30d50068c22 P 82c39407898d4033af3c6a1a688a5197: Bootstrap starting.
I20240503 01:58:48.241765 909 tablet_bootstrap.cc:492] T d2a69cc1ebd8418f85677c18ef9d80ef P d125d3941ab94bd5b970110e5791e171: Bootstrap starting.
I20240503 01:58:48.242064 935 leader_election.cc:288] T 86f9a4762bf34b52a8d000a487226e5b P d125d3941ab94bd5b970110e5791e171 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82c39407898d4033af3c6a1a688a5197 (127.0.99.130:44599), cae31ce250aa4d64953e14da451c568c (127.0.99.129:44403)
I20240503 01:58:48.242944 920 tablet_bootstrap.cc:654] T 84bfa8f64219448d8d49a30d50068c22 P 82c39407898d4033af3c6a1a688a5197: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:48.246001 909 tablet_bootstrap.cc:654] T d2a69cc1ebd8418f85677c18ef9d80ef P d125d3941ab94bd5b970110e5791e171: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:48.246095 920 tablet_bootstrap.cc:492] T 84bfa8f64219448d8d49a30d50068c22 P 82c39407898d4033af3c6a1a688a5197: No bootstrap required, opened a new log
I20240503 01:58:48.246265 920 ts_tablet_manager.cc:1367] T 84bfa8f64219448d8d49a30d50068c22 P 82c39407898d4033af3c6a1a688a5197: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20240503 01:58:48.246807 920 raft_consensus.cc:348] T 84bfa8f64219448d8d49a30d50068c22 P 82c39407898d4033af3c6a1a688a5197 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.246943 920 raft_consensus.cc:374] T 84bfa8f64219448d8d49a30d50068c22 P 82c39407898d4033af3c6a1a688a5197 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:48.246994 920 raft_consensus.cc:729] T 84bfa8f64219448d8d49a30d50068c22 P 82c39407898d4033af3c6a1a688a5197 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82c39407898d4033af3c6a1a688a5197, State: Initialized, Role: FOLLOWER
I20240503 01:58:48.247167 920 consensus_queue.cc:260] T 84bfa8f64219448d8d49a30d50068c22 P 82c39407898d4033af3c6a1a688a5197 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.247412 912 raft_consensus.cc:482] T c54888ed43eb4ab4bc79a9ab0a2bcf51 P cae31ce250aa4d64953e14da451c568c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:58:48.247512 912 raft_consensus.cc:504] T c54888ed43eb4ab4bc79a9ab0a2bcf51 P cae31ce250aa4d64953e14da451c568c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.247992 912 leader_election.cc:288] T c54888ed43eb4ab4bc79a9ab0a2bcf51 P cae31ce250aa4d64953e14da451c568c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82c39407898d4033af3c6a1a688a5197 (127.0.99.130:44599), d125d3941ab94bd5b970110e5791e171 (127.0.99.131:44747)
I20240503 01:58:48.248569 714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c54888ed43eb4ab4bc79a9ab0a2bcf51" candidate_uuid: "cae31ce250aa4d64953e14da451c568c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82c39407898d4033af3c6a1a688a5197" is_pre_election: true
I20240503 01:58:48.248870 920 ts_tablet_manager.cc:1398] T 84bfa8f64219448d8d49a30d50068c22 P 82c39407898d4033af3c6a1a688a5197: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240503 01:58:48.248932 714 raft_consensus.cc:2454] T c54888ed43eb4ab4bc79a9ab0a2bcf51 P 82c39407898d4033af3c6a1a688a5197 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cae31ce250aa4d64953e14da451c568c in term 0.
I20240503 01:58:48.248857 713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86f9a4762bf34b52a8d000a487226e5b" candidate_uuid: "d125d3941ab94bd5b970110e5791e171" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82c39407898d4033af3c6a1a688a5197" is_pre_election: true
I20240503 01:58:48.249082 920 tablet_bootstrap.cc:492] T d2a69cc1ebd8418f85677c18ef9d80ef P 82c39407898d4033af3c6a1a688a5197: Bootstrap starting.
I20240503 01:58:48.249164 845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c54888ed43eb4ab4bc79a9ab0a2bcf51" candidate_uuid: "cae31ce250aa4d64953e14da451c568c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d125d3941ab94bd5b970110e5791e171" is_pre_election: true
I20240503 01:58:48.249300 845 raft_consensus.cc:2454] T c54888ed43eb4ab4bc79a9ab0a2bcf51 P d125d3941ab94bd5b970110e5791e171 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cae31ce250aa4d64953e14da451c568c in term 0.
I20240503 01:58:48.249626 912 raft_consensus.cc:482] T a7090b6ae5394aecb146a7fa25b89e79 P cae31ce250aa4d64953e14da451c568c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:58:48.249642 517 leader_election.cc:302] T c54888ed43eb4ab4bc79a9ab0a2bcf51 P cae31ce250aa4d64953e14da451c568c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cae31ce250aa4d64953e14da451c568c, d125d3941ab94bd5b970110e5791e171; no voters:
I20240503 01:58:48.249727 912 raft_consensus.cc:504] T a7090b6ae5394aecb146a7fa25b89e79 P cae31ce250aa4d64953e14da451c568c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
W20240503 01:58:48.249738 778 leader_election.cc:341] T 86f9a4762bf34b52a8d000a487226e5b P d125d3941ab94bd5b970110e5791e171 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 82c39407898d4033af3c6a1a688a5197 (127.0.99.130:44599): Illegal state: must be running to vote when last-logged opid is not known
I20240503 01:58:48.249876 914 raft_consensus.cc:2781] T c54888ed43eb4ab4bc79a9ab0a2bcf51 P cae31ce250aa4d64953e14da451c568c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:58:48.249922 914 raft_consensus.cc:482] T c54888ed43eb4ab4bc79a9ab0a2bcf51 P cae31ce250aa4d64953e14da451c568c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:58:48.250020 914 raft_consensus.cc:3037] T c54888ed43eb4ab4bc79a9ab0a2bcf51 P cae31ce250aa4d64953e14da451c568c [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:48.250097 912 leader_election.cc:288] T a7090b6ae5394aecb146a7fa25b89e79 P cae31ce250aa4d64953e14da451c568c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82c39407898d4033af3c6a1a688a5197 (127.0.99.130:44599), d125d3941ab94bd5b970110e5791e171 (127.0.99.131:44747)
I20240503 01:58:48.250347 713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7090b6ae5394aecb146a7fa25b89e79" candidate_uuid: "cae31ce250aa4d64953e14da451c568c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82c39407898d4033af3c6a1a688a5197" is_pre_election: true
I20240503 01:58:48.250471 845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7090b6ae5394aecb146a7fa25b89e79" candidate_uuid: "cae31ce250aa4d64953e14da451c568c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d125d3941ab94bd5b970110e5791e171" is_pre_election: true
I20240503 01:58:48.250475 713 raft_consensus.cc:2454] T a7090b6ae5394aecb146a7fa25b89e79 P 82c39407898d4033af3c6a1a688a5197 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cae31ce250aa4d64953e14da451c568c in term 0.
I20240503 01:58:48.250571 845 raft_consensus.cc:2454] T a7090b6ae5394aecb146a7fa25b89e79 P d125d3941ab94bd5b970110e5791e171 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cae31ce250aa4d64953e14da451c568c in term 0.
I20240503 01:58:48.250828 517 leader_election.cc:302] T a7090b6ae5394aecb146a7fa25b89e79 P cae31ce250aa4d64953e14da451c568c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cae31ce250aa4d64953e14da451c568c, d125d3941ab94bd5b970110e5791e171; no voters:
I20240503 01:58:48.250993 912 raft_consensus.cc:2781] T a7090b6ae5394aecb146a7fa25b89e79 P cae31ce250aa4d64953e14da451c568c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:58:48.251056 912 raft_consensus.cc:482] T a7090b6ae5394aecb146a7fa25b89e79 P cae31ce250aa4d64953e14da451c568c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:58:48.251093 912 raft_consensus.cc:3037] T a7090b6ae5394aecb146a7fa25b89e79 P cae31ce250aa4d64953e14da451c568c [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:48.251406 914 raft_consensus.cc:504] T c54888ed43eb4ab4bc79a9ab0a2bcf51 P cae31ce250aa4d64953e14da451c568c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.251751 914 leader_election.cc:288] T c54888ed43eb4ab4bc79a9ab0a2bcf51 P cae31ce250aa4d64953e14da451c568c [CANDIDATE]: Term 1 election: Requested vote from peers 82c39407898d4033af3c6a1a688a5197 (127.0.99.130:44599), d125d3941ab94bd5b970110e5791e171 (127.0.99.131:44747)
I20240503 01:58:48.251951 713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c54888ed43eb4ab4bc79a9ab0a2bcf51" candidate_uuid: "cae31ce250aa4d64953e14da451c568c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82c39407898d4033af3c6a1a688a5197"
I20240503 01:58:48.252009 912 raft_consensus.cc:504] T a7090b6ae5394aecb146a7fa25b89e79 P cae31ce250aa4d64953e14da451c568c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.252072 713 raft_consensus.cc:3037] T c54888ed43eb4ab4bc79a9ab0a2bcf51 P 82c39407898d4033af3c6a1a688a5197 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:48.252316 912 leader_election.cc:288] T a7090b6ae5394aecb146a7fa25b89e79 P cae31ce250aa4d64953e14da451c568c [CANDIDATE]: Term 1 election: Requested vote from peers 82c39407898d4033af3c6a1a688a5197 (127.0.99.130:44599), d125d3941ab94bd5b970110e5791e171 (127.0.99.131:44747)
I20240503 01:58:48.252338 845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c54888ed43eb4ab4bc79a9ab0a2bcf51" candidate_uuid: "cae31ce250aa4d64953e14da451c568c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d125d3941ab94bd5b970110e5791e171"
I20240503 01:58:48.252460 845 raft_consensus.cc:3037] T c54888ed43eb4ab4bc79a9ab0a2bcf51 P d125d3941ab94bd5b970110e5791e171 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:48.252750 844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7090b6ae5394aecb146a7fa25b89e79" candidate_uuid: "cae31ce250aa4d64953e14da451c568c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d125d3941ab94bd5b970110e5791e171"
I20240503 01:58:48.252849 844 raft_consensus.cc:3037] T a7090b6ae5394aecb146a7fa25b89e79 P d125d3941ab94bd5b970110e5791e171 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:48.253818 845 raft_consensus.cc:2454] T c54888ed43eb4ab4bc79a9ab0a2bcf51 P d125d3941ab94bd5b970110e5791e171 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cae31ce250aa4d64953e14da451c568c in term 1.
I20240503 01:58:48.254012 844 raft_consensus.cc:2454] T a7090b6ae5394aecb146a7fa25b89e79 P d125d3941ab94bd5b970110e5791e171 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cae31ce250aa4d64953e14da451c568c in term 1.
I20240503 01:58:48.254139 713 raft_consensus.cc:2454] T c54888ed43eb4ab4bc79a9ab0a2bcf51 P 82c39407898d4033af3c6a1a688a5197 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cae31ce250aa4d64953e14da451c568c in term 1.
I20240503 01:58:48.254267 920 tablet_bootstrap.cc:654] T d2a69cc1ebd8418f85677c18ef9d80ef P 82c39407898d4033af3c6a1a688a5197: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:48.254257 517 leader_election.cc:302] T c54888ed43eb4ab4bc79a9ab0a2bcf51 P cae31ce250aa4d64953e14da451c568c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cae31ce250aa4d64953e14da451c568c, d125d3941ab94bd5b970110e5791e171; no voters:
I20240503 01:58:48.254436 517 leader_election.cc:302] T a7090b6ae5394aecb146a7fa25b89e79 P cae31ce250aa4d64953e14da451c568c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cae31ce250aa4d64953e14da451c568c, d125d3941ab94bd5b970110e5791e171; no voters:
I20240503 01:58:48.254618 914 raft_consensus.cc:2781] T c54888ed43eb4ab4bc79a9ab0a2bcf51 P cae31ce250aa4d64953e14da451c568c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:58:48.254722 914 raft_consensus.cc:686] T c54888ed43eb4ab4bc79a9ab0a2bcf51 P cae31ce250aa4d64953e14da451c568c [term 1 LEADER]: Becoming Leader. State: Replica: cae31ce250aa4d64953e14da451c568c, State: Running, Role: LEADER
I20240503 01:58:48.254717 713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7090b6ae5394aecb146a7fa25b89e79" candidate_uuid: "cae31ce250aa4d64953e14da451c568c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82c39407898d4033af3c6a1a688a5197"
I20240503 01:58:48.254827 713 raft_consensus.cc:3037] T a7090b6ae5394aecb146a7fa25b89e79 P 82c39407898d4033af3c6a1a688a5197 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:48.254859 912 raft_consensus.cc:2781] T a7090b6ae5394aecb146a7fa25b89e79 P cae31ce250aa4d64953e14da451c568c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:58:48.254927 912 raft_consensus.cc:686] T a7090b6ae5394aecb146a7fa25b89e79 P cae31ce250aa4d64953e14da451c568c [term 1 LEADER]: Becoming Leader. State: Replica: cae31ce250aa4d64953e14da451c568c, State: Running, Role: LEADER
I20240503 01:58:48.255060 912 consensus_queue.cc:237] T a7090b6ae5394aecb146a7fa25b89e79 P cae31ce250aa4d64953e14da451c568c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.255637 713 raft_consensus.cc:2454] T a7090b6ae5394aecb146a7fa25b89e79 P 82c39407898d4033af3c6a1a688a5197 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cae31ce250aa4d64953e14da451c568c in term 1.
I20240503 01:58:48.255060 914 consensus_queue.cc:237] T c54888ed43eb4ab4bc79a9ab0a2bcf51 P cae31ce250aa4d64953e14da451c568c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.258966 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86f9a4762bf34b52a8d000a487226e5b" candidate_uuid: "d125d3941ab94bd5b970110e5791e171" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cae31ce250aa4d64953e14da451c568c" is_pre_election: true
I20240503 01:58:48.258764 438 catalog_manager.cc:5484] T c54888ed43eb4ab4bc79a9ab0a2bcf51 P cae31ce250aa4d64953e14da451c568c reported cstate change: term changed from 0 to 1, leader changed from <none> to cae31ce250aa4d64953e14da451c568c (127.0.99.129). New cstate: current_term: 1 leader_uuid: "cae31ce250aa4d64953e14da451c568c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } health_report { overall_health: UNKNOWN } } }
I20240503 01:58:48.259104 909 tablet_bootstrap.cc:492] T d2a69cc1ebd8418f85677c18ef9d80ef P d125d3941ab94bd5b970110e5791e171: No bootstrap required, opened a new log
I20240503 01:58:48.259156 583 raft_consensus.cc:2454] T 86f9a4762bf34b52a8d000a487226e5b P cae31ce250aa4d64953e14da451c568c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d125d3941ab94bd5b970110e5791e171 in term 0.
I20240503 01:58:48.259234 909 ts_tablet_manager.cc:1367] T d2a69cc1ebd8418f85677c18ef9d80ef P d125d3941ab94bd5b970110e5791e171: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.001s
I20240503 01:58:48.259420 438 catalog_manager.cc:5484] T a7090b6ae5394aecb146a7fa25b89e79 P cae31ce250aa4d64953e14da451c568c reported cstate change: term changed from 0 to 1, leader changed from <none> to cae31ce250aa4d64953e14da451c568c (127.0.99.129). New cstate: current_term: 1 leader_uuid: "cae31ce250aa4d64953e14da451c568c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } health_report { overall_health: UNKNOWN } } }
I20240503 01:58:48.259723 780 leader_election.cc:302] T 86f9a4762bf34b52a8d000a487226e5b P d125d3941ab94bd5b970110e5791e171 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cae31ce250aa4d64953e14da451c568c, d125d3941ab94bd5b970110e5791e171; no voters: 82c39407898d4033af3c6a1a688a5197
I20240503 01:58:48.259827 909 raft_consensus.cc:348] T d2a69cc1ebd8418f85677c18ef9d80ef P d125d3941ab94bd5b970110e5791e171 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } } peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } }
I20240503 01:58:48.259917 909 raft_consensus.cc:374] T d2a69cc1ebd8418f85677c18ef9d80ef P d125d3941ab94bd5b970110e5791e171 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:48.259949 909 raft_consensus.cc:729] T d2a69cc1ebd8418f85677c18ef9d80ef P d125d3941ab94bd5b970110e5791e171 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d125d3941ab94bd5b970110e5791e171, State: Initialized, Role: FOLLOWER
I20240503 01:58:48.260090 920 tablet_bootstrap.cc:492] T d2a69cc1ebd8418f85677c18ef9d80ef P 82c39407898d4033af3c6a1a688a5197: No bootstrap required, opened a new log
I20240503 01:58:48.260108 909 consensus_queue.cc:260] T d2a69cc1ebd8418f85677c18ef9d80ef P d125d3941ab94bd5b970110e5791e171 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } } peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } }
I20240503 01:58:48.260352 929 raft_consensus.cc:2781] T 86f9a4762bf34b52a8d000a487226e5b P d125d3941ab94bd5b970110e5791e171 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:58:48.260407 909 ts_tablet_manager.cc:1398] T d2a69cc1ebd8418f85677c18ef9d80ef P d125d3941ab94bd5b970110e5791e171: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:58:48.260429 929 raft_consensus.cc:482] T 86f9a4762bf34b52a8d000a487226e5b P d125d3941ab94bd5b970110e5791e171 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:58:48.260493 929 raft_consensus.cc:3037] T 86f9a4762bf34b52a8d000a487226e5b P d125d3941ab94bd5b970110e5791e171 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:48.260569 909 tablet_bootstrap.cc:492] T 1581a7ae381c4228bbb4420b84ba432f P d125d3941ab94bd5b970110e5791e171: Bootstrap starting.
I20240503 01:58:48.261430 920 ts_tablet_manager.cc:1367] T d2a69cc1ebd8418f85677c18ef9d80ef P 82c39407898d4033af3c6a1a688a5197: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.001s
I20240503 01:58:48.261951 909 tablet_bootstrap.cc:654] T 1581a7ae381c4228bbb4420b84ba432f P d125d3941ab94bd5b970110e5791e171: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:48.262018 920 raft_consensus.cc:348] T d2a69cc1ebd8418f85677c18ef9d80ef P 82c39407898d4033af3c6a1a688a5197 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } } peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } }
I20240503 01:58:48.262140 920 raft_consensus.cc:374] T d2a69cc1ebd8418f85677c18ef9d80ef P 82c39407898d4033af3c6a1a688a5197 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:48.262144 929 raft_consensus.cc:504] T 86f9a4762bf34b52a8d000a487226e5b P d125d3941ab94bd5b970110e5791e171 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.262197 920 raft_consensus.cc:729] T d2a69cc1ebd8418f85677c18ef9d80ef P 82c39407898d4033af3c6a1a688a5197 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82c39407898d4033af3c6a1a688a5197, State: Initialized, Role: FOLLOWER
I20240503 01:58:48.262370 920 consensus_queue.cc:260] T d2a69cc1ebd8418f85677c18ef9d80ef P 82c39407898d4033af3c6a1a688a5197 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } } peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } }
I20240503 01:58:48.262868 920 ts_tablet_manager.cc:1398] T d2a69cc1ebd8418f85677c18ef9d80ef P 82c39407898d4033af3c6a1a688a5197: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:58:48.263078 929 leader_election.cc:288] T 86f9a4762bf34b52a8d000a487226e5b P d125d3941ab94bd5b970110e5791e171 [CANDIDATE]: Term 1 election: Requested vote from peers 82c39407898d4033af3c6a1a688a5197 (127.0.99.130:44599), cae31ce250aa4d64953e14da451c568c (127.0.99.129:44403)
I20240503 01:58:48.263103 920 tablet_bootstrap.cc:492] T 42a69d697389414a973d995a763d8219 P 82c39407898d4033af3c6a1a688a5197: Bootstrap starting.
I20240503 01:58:48.263562 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86f9a4762bf34b52a8d000a487226e5b" candidate_uuid: "d125d3941ab94bd5b970110e5791e171" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cae31ce250aa4d64953e14da451c568c"
I20240503 01:58:48.263679 583 raft_consensus.cc:3037] T 86f9a4762bf34b52a8d000a487226e5b P cae31ce250aa4d64953e14da451c568c [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:48.263650 713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86f9a4762bf34b52a8d000a487226e5b" candidate_uuid: "d125d3941ab94bd5b970110e5791e171" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82c39407898d4033af3c6a1a688a5197"
W20240503 01:58:48.264010 778 leader_election.cc:341] T 86f9a4762bf34b52a8d000a487226e5b P d125d3941ab94bd5b970110e5791e171 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 82c39407898d4033af3c6a1a688a5197 (127.0.99.130:44599): Illegal state: must be running to vote when last-logged opid is not known
I20240503 01:58:48.264802 583 raft_consensus.cc:2454] T 86f9a4762bf34b52a8d000a487226e5b P cae31ce250aa4d64953e14da451c568c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d125d3941ab94bd5b970110e5791e171 in term 1.
I20240503 01:58:48.264843 909 tablet_bootstrap.cc:492] T 1581a7ae381c4228bbb4420b84ba432f P d125d3941ab94bd5b970110e5791e171: No bootstrap required, opened a new log
I20240503 01:58:48.264912 909 ts_tablet_manager.cc:1367] T 1581a7ae381c4228bbb4420b84ba432f P d125d3941ab94bd5b970110e5791e171: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 01:58:48.265245 780 leader_election.cc:302] T 86f9a4762bf34b52a8d000a487226e5b P d125d3941ab94bd5b970110e5791e171 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cae31ce250aa4d64953e14da451c568c, d125d3941ab94bd5b970110e5791e171; no voters: 82c39407898d4033af3c6a1a688a5197
I20240503 01:58:48.265517 929 raft_consensus.cc:2781] T 86f9a4762bf34b52a8d000a487226e5b P d125d3941ab94bd5b970110e5791e171 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:58:48.265616 929 raft_consensus.cc:686] T 86f9a4762bf34b52a8d000a487226e5b P d125d3941ab94bd5b970110e5791e171 [term 1 LEADER]: Becoming Leader. State: Replica: d125d3941ab94bd5b970110e5791e171, State: Running, Role: LEADER
I20240503 01:58:48.265518 909 raft_consensus.cc:348] T 1581a7ae381c4228bbb4420b84ba432f P d125d3941ab94bd5b970110e5791e171 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.265743 909 raft_consensus.cc:374] T 1581a7ae381c4228bbb4420b84ba432f P d125d3941ab94bd5b970110e5791e171 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:48.265836 909 raft_consensus.cc:729] T 1581a7ae381c4228bbb4420b84ba432f P d125d3941ab94bd5b970110e5791e171 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d125d3941ab94bd5b970110e5791e171, State: Initialized, Role: FOLLOWER
I20240503 01:58:48.265936 920 tablet_bootstrap.cc:654] T 42a69d697389414a973d995a763d8219 P 82c39407898d4033af3c6a1a688a5197: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:48.265993 929 consensus_queue.cc:237] T 86f9a4762bf34b52a8d000a487226e5b P d125d3941ab94bd5b970110e5791e171 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.265997 909 consensus_queue.cc:260] T 1581a7ae381c4228bbb4420b84ba432f P d125d3941ab94bd5b970110e5791e171 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.266556 909 ts_tablet_manager.cc:1398] T 1581a7ae381c4228bbb4420b84ba432f P d125d3941ab94bd5b970110e5791e171: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:58:48.266772 909 tablet_bootstrap.cc:492] T 9f1fb01218234ec4a13fd6f9dfb3fdd6 P d125d3941ab94bd5b970110e5791e171: Bootstrap starting.
I20240503 01:58:48.267642 920 tablet_bootstrap.cc:492] T 42a69d697389414a973d995a763d8219 P 82c39407898d4033af3c6a1a688a5197: No bootstrap required, opened a new log
I20240503 01:58:48.267767 920 ts_tablet_manager.cc:1367] T 42a69d697389414a973d995a763d8219 P 82c39407898d4033af3c6a1a688a5197: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s
I20240503 01:58:48.267776 909 tablet_bootstrap.cc:654] T 9f1fb01218234ec4a13fd6f9dfb3fdd6 P d125d3941ab94bd5b970110e5791e171: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:48.268397 920 raft_consensus.cc:348] T 42a69d697389414a973d995a763d8219 P 82c39407898d4033af3c6a1a688a5197 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.268555 920 raft_consensus.cc:374] T 42a69d697389414a973d995a763d8219 P 82c39407898d4033af3c6a1a688a5197 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:48.268623 920 raft_consensus.cc:729] T 42a69d697389414a973d995a763d8219 P 82c39407898d4033af3c6a1a688a5197 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82c39407898d4033af3c6a1a688a5197, State: Initialized, Role: FOLLOWER
I20240503 01:58:48.268791 920 consensus_queue.cc:260] T 42a69d697389414a973d995a763d8219 P 82c39407898d4033af3c6a1a688a5197 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.269129 920 ts_tablet_manager.cc:1398] T 42a69d697389414a973d995a763d8219 P 82c39407898d4033af3c6a1a688a5197: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:58:48.269462 920 tablet_bootstrap.cc:492] T 2f1cd41c7fc8437598f73938ae3f2a1d P 82c39407898d4033af3c6a1a688a5197: Bootstrap starting.
I20240503 01:58:48.269817 909 tablet_bootstrap.cc:492] T 9f1fb01218234ec4a13fd6f9dfb3fdd6 P d125d3941ab94bd5b970110e5791e171: No bootstrap required, opened a new log
I20240503 01:58:48.269976 909 ts_tablet_manager.cc:1367] T 9f1fb01218234ec4a13fd6f9dfb3fdd6 P d125d3941ab94bd5b970110e5791e171: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:58:48.269933 441 catalog_manager.cc:5484] T 86f9a4762bf34b52a8d000a487226e5b P d125d3941ab94bd5b970110e5791e171 reported cstate change: term changed from 0 to 1, leader changed from <none> to d125d3941ab94bd5b970110e5791e171 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "d125d3941ab94bd5b970110e5791e171" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } health_report { overall_health: HEALTHY } } }
I20240503 01:58:48.270620 909 raft_consensus.cc:348] T 9f1fb01218234ec4a13fd6f9dfb3fdd6 P d125d3941ab94bd5b970110e5791e171 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.270743 909 raft_consensus.cc:374] T 9f1fb01218234ec4a13fd6f9dfb3fdd6 P d125d3941ab94bd5b970110e5791e171 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:48.270797 909 raft_consensus.cc:729] T 9f1fb01218234ec4a13fd6f9dfb3fdd6 P d125d3941ab94bd5b970110e5791e171 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d125d3941ab94bd5b970110e5791e171, State: Initialized, Role: FOLLOWER
I20240503 01:58:48.270951 909 consensus_queue.cc:260] T 9f1fb01218234ec4a13fd6f9dfb3fdd6 P d125d3941ab94bd5b970110e5791e171 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.270982 920 tablet_bootstrap.cc:654] T 2f1cd41c7fc8437598f73938ae3f2a1d P 82c39407898d4033af3c6a1a688a5197: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:48.271310 909 ts_tablet_manager.cc:1398] T 9f1fb01218234ec4a13fd6f9dfb3fdd6 P d125d3941ab94bd5b970110e5791e171: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:58:48.271530 909 tablet_bootstrap.cc:492] T ccaa4613da8546f88fb844a6711baabc P d125d3941ab94bd5b970110e5791e171: Bootstrap starting.
I20240503 01:58:48.272351 920 tablet_bootstrap.cc:492] T 2f1cd41c7fc8437598f73938ae3f2a1d P 82c39407898d4033af3c6a1a688a5197: No bootstrap required, opened a new log
I20240503 01:58:48.272454 920 ts_tablet_manager.cc:1367] T 2f1cd41c7fc8437598f73938ae3f2a1d P 82c39407898d4033af3c6a1a688a5197: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:58:48.272950 920 raft_consensus.cc:348] T 2f1cd41c7fc8437598f73938ae3f2a1d P 82c39407898d4033af3c6a1a688a5197 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.273031 909 tablet_bootstrap.cc:654] T ccaa4613da8546f88fb844a6711baabc P d125d3941ab94bd5b970110e5791e171: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:48.273046 920 raft_consensus.cc:374] T 2f1cd41c7fc8437598f73938ae3f2a1d P 82c39407898d4033af3c6a1a688a5197 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:48.273090 920 raft_consensus.cc:729] T 2f1cd41c7fc8437598f73938ae3f2a1d P 82c39407898d4033af3c6a1a688a5197 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82c39407898d4033af3c6a1a688a5197, State: Initialized, Role: FOLLOWER
I20240503 01:58:48.273206 920 consensus_queue.cc:260] T 2f1cd41c7fc8437598f73938ae3f2a1d P 82c39407898d4033af3c6a1a688a5197 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.273469 920 ts_tablet_manager.cc:1398] T 2f1cd41c7fc8437598f73938ae3f2a1d P 82c39407898d4033af3c6a1a688a5197: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:58:48.273627 920 tablet_bootstrap.cc:492] T e38e1de36d82402c82b72dce9e765505 P 82c39407898d4033af3c6a1a688a5197: Bootstrap starting.
I20240503 01:58:48.274935 920 tablet_bootstrap.cc:654] T e38e1de36d82402c82b72dce9e765505 P 82c39407898d4033af3c6a1a688a5197: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:48.275475 909 tablet_bootstrap.cc:492] T ccaa4613da8546f88fb844a6711baabc P d125d3941ab94bd5b970110e5791e171: No bootstrap required, opened a new log
I20240503 01:58:48.275537 909 ts_tablet_manager.cc:1367] T ccaa4613da8546f88fb844a6711baabc P d125d3941ab94bd5b970110e5791e171: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240503 01:58:48.276018 909 raft_consensus.cc:348] T ccaa4613da8546f88fb844a6711baabc P d125d3941ab94bd5b970110e5791e171 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.276173 920 tablet_bootstrap.cc:492] T e38e1de36d82402c82b72dce9e765505 P 82c39407898d4033af3c6a1a688a5197: No bootstrap required, opened a new log
I20240503 01:58:48.276155 909 raft_consensus.cc:374] T ccaa4613da8546f88fb844a6711baabc P d125d3941ab94bd5b970110e5791e171 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:48.276232 909 raft_consensus.cc:729] T ccaa4613da8546f88fb844a6711baabc P d125d3941ab94bd5b970110e5791e171 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d125d3941ab94bd5b970110e5791e171, State: Initialized, Role: FOLLOWER
I20240503 01:58:48.276264 920 ts_tablet_manager.cc:1367] T e38e1de36d82402c82b72dce9e765505 P 82c39407898d4033af3c6a1a688a5197: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 01:58:48.276402 909 consensus_queue.cc:260] T ccaa4613da8546f88fb844a6711baabc P d125d3941ab94bd5b970110e5791e171 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.276819 909 ts_tablet_manager.cc:1398] T ccaa4613da8546f88fb844a6711baabc P d125d3941ab94bd5b970110e5791e171: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:58:48.276798 920 raft_consensus.cc:348] T e38e1de36d82402c82b72dce9e765505 P 82c39407898d4033af3c6a1a688a5197 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.276892 920 raft_consensus.cc:374] T e38e1de36d82402c82b72dce9e765505 P 82c39407898d4033af3c6a1a688a5197 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:48.276980 920 raft_consensus.cc:729] T e38e1de36d82402c82b72dce9e765505 P 82c39407898d4033af3c6a1a688a5197 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82c39407898d4033af3c6a1a688a5197, State: Initialized, Role: FOLLOWER
I20240503 01:58:48.277071 909 tablet_bootstrap.cc:492] T e38e1de36d82402c82b72dce9e765505 P d125d3941ab94bd5b970110e5791e171: Bootstrap starting.
I20240503 01:58:48.277108 920 consensus_queue.cc:260] T e38e1de36d82402c82b72dce9e765505 P 82c39407898d4033af3c6a1a688a5197 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.277428 920 ts_tablet_manager.cc:1398] T e38e1de36d82402c82b72dce9e765505 P 82c39407898d4033af3c6a1a688a5197: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:58:48.277606 920 tablet_bootstrap.cc:492] T 1581a7ae381c4228bbb4420b84ba432f P 82c39407898d4033af3c6a1a688a5197: Bootstrap starting.
I20240503 01:58:48.278364 909 tablet_bootstrap.cc:654] T e38e1de36d82402c82b72dce9e765505 P d125d3941ab94bd5b970110e5791e171: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:48.278569 920 tablet_bootstrap.cc:654] T 1581a7ae381c4228bbb4420b84ba432f P 82c39407898d4033af3c6a1a688a5197: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:48.279470 914 raft_consensus.cc:482] T 0488ca07075c4daca7b8f4d5f827ea56 P cae31ce250aa4d64953e14da451c568c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:58:48.279569 914 raft_consensus.cc:504] T 0488ca07075c4daca7b8f4d5f827ea56 P cae31ce250aa4d64953e14da451c568c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.279997 914 leader_election.cc:288] T 0488ca07075c4daca7b8f4d5f827ea56 P cae31ce250aa4d64953e14da451c568c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82c39407898d4033af3c6a1a688a5197 (127.0.99.130:44599), d125d3941ab94bd5b970110e5791e171 (127.0.99.131:44747)
I20240503 01:58:48.280180 920 tablet_bootstrap.cc:492] T 1581a7ae381c4228bbb4420b84ba432f P 82c39407898d4033af3c6a1a688a5197: No bootstrap required, opened a new log
I20240503 01:58:48.280253 920 ts_tablet_manager.cc:1367] T 1581a7ae381c4228bbb4420b84ba432f P 82c39407898d4033af3c6a1a688a5197: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240503 01:58:48.280268 713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0488ca07075c4daca7b8f4d5f827ea56" candidate_uuid: "cae31ce250aa4d64953e14da451c568c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82c39407898d4033af3c6a1a688a5197" is_pre_election: true
I20240503 01:58:48.280412 844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0488ca07075c4daca7b8f4d5f827ea56" candidate_uuid: "cae31ce250aa4d64953e14da451c568c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d125d3941ab94bd5b970110e5791e171" is_pre_election: true
I20240503 01:58:48.280546 844 raft_consensus.cc:2454] T 0488ca07075c4daca7b8f4d5f827ea56 P d125d3941ab94bd5b970110e5791e171 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cae31ce250aa4d64953e14da451c568c in term 0.
W20240503 01:58:48.280694 516 leader_election.cc:341] T 0488ca07075c4daca7b8f4d5f827ea56 P cae31ce250aa4d64953e14da451c568c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 82c39407898d4033af3c6a1a688a5197 (127.0.99.130:44599): Illegal state: must be running to vote when last-logged opid is not known
I20240503 01:58:48.280845 517 leader_election.cc:302] T 0488ca07075c4daca7b8f4d5f827ea56 P cae31ce250aa4d64953e14da451c568c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cae31ce250aa4d64953e14da451c568c, d125d3941ab94bd5b970110e5791e171; no voters: 82c39407898d4033af3c6a1a688a5197
I20240503 01:58:48.280858 920 raft_consensus.cc:348] T 1581a7ae381c4228bbb4420b84ba432f P 82c39407898d4033af3c6a1a688a5197 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.280922 909 tablet_bootstrap.cc:492] T e38e1de36d82402c82b72dce9e765505 P d125d3941ab94bd5b970110e5791e171: No bootstrap required, opened a new log
I20240503 01:58:48.280937 920 raft_consensus.cc:374] T 1581a7ae381c4228bbb4420b84ba432f P 82c39407898d4033af3c6a1a688a5197 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:48.280979 920 raft_consensus.cc:729] T 1581a7ae381c4228bbb4420b84ba432f P 82c39407898d4033af3c6a1a688a5197 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82c39407898d4033af3c6a1a688a5197, State: Initialized, Role: FOLLOWER
I20240503 01:58:48.281008 909 ts_tablet_manager.cc:1367] T e38e1de36d82402c82b72dce9e765505 P d125d3941ab94bd5b970110e5791e171: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20240503 01:58:48.281031 914 raft_consensus.cc:2781] T 0488ca07075c4daca7b8f4d5f827ea56 P cae31ce250aa4d64953e14da451c568c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:58:48.281077 914 raft_consensus.cc:482] T 0488ca07075c4daca7b8f4d5f827ea56 P cae31ce250aa4d64953e14da451c568c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:58:48.281126 914 raft_consensus.cc:3037] T 0488ca07075c4daca7b8f4d5f827ea56 P cae31ce250aa4d64953e14da451c568c [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:48.281081 920 consensus_queue.cc:260] T 1581a7ae381c4228bbb4420b84ba432f P 82c39407898d4033af3c6a1a688a5197 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.281354 920 ts_tablet_manager.cc:1398] T 1581a7ae381c4228bbb4420b84ba432f P 82c39407898d4033af3c6a1a688a5197: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:58:48.281551 920 tablet_bootstrap.cc:492] T 8d1ad90f476a4b2b97f59259ef4bff9f P 82c39407898d4033af3c6a1a688a5197: Bootstrap starting.
I20240503 01:58:48.281648 909 raft_consensus.cc:348] T e38e1de36d82402c82b72dce9e765505 P d125d3941ab94bd5b970110e5791e171 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.281738 909 raft_consensus.cc:374] T e38e1de36d82402c82b72dce9e765505 P d125d3941ab94bd5b970110e5791e171 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:48.281766 909 raft_consensus.cc:729] T e38e1de36d82402c82b72dce9e765505 P d125d3941ab94bd5b970110e5791e171 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d125d3941ab94bd5b970110e5791e171, State: Initialized, Role: FOLLOWER
I20240503 01:58:48.281878 909 consensus_queue.cc:260] T e38e1de36d82402c82b72dce9e765505 P d125d3941ab94bd5b970110e5791e171 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.282187 909 ts_tablet_manager.cc:1398] T e38e1de36d82402c82b72dce9e765505 P d125d3941ab94bd5b970110e5791e171: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:58:48.282196 914 raft_consensus.cc:504] T 0488ca07075c4daca7b8f4d5f827ea56 P cae31ce250aa4d64953e14da451c568c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.282346 909 tablet_bootstrap.cc:492] T 2e5dc6a7de704dcdb0aec15124300b61 P d125d3941ab94bd5b970110e5791e171: Bootstrap starting.
I20240503 01:58:48.282519 914 leader_election.cc:288] T 0488ca07075c4daca7b8f4d5f827ea56 P cae31ce250aa4d64953e14da451c568c [CANDIDATE]: Term 1 election: Requested vote from peers 82c39407898d4033af3c6a1a688a5197 (127.0.99.130:44599), d125d3941ab94bd5b970110e5791e171 (127.0.99.131:44747)
I20240503 01:58:48.283138 920 tablet_bootstrap.cc:654] T 8d1ad90f476a4b2b97f59259ef4bff9f P 82c39407898d4033af3c6a1a688a5197: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:48.283406 909 tablet_bootstrap.cc:654] T 2e5dc6a7de704dcdb0aec15124300b61 P d125d3941ab94bd5b970110e5791e171: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:48.283727 713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0488ca07075c4daca7b8f4d5f827ea56" candidate_uuid: "cae31ce250aa4d64953e14da451c568c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82c39407898d4033af3c6a1a688a5197"
I20240503 01:58:48.284507 844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0488ca07075c4daca7b8f4d5f827ea56" candidate_uuid: "cae31ce250aa4d64953e14da451c568c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d125d3941ab94bd5b970110e5791e171"
W20240503 01:58:48.284579 516 leader_election.cc:341] T 0488ca07075c4daca7b8f4d5f827ea56 P cae31ce250aa4d64953e14da451c568c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 82c39407898d4033af3c6a1a688a5197 (127.0.99.130:44599): Illegal state: must be running to vote when last-logged opid is not known
I20240503 01:58:48.284605 844 raft_consensus.cc:3037] T 0488ca07075c4daca7b8f4d5f827ea56 P d125d3941ab94bd5b970110e5791e171 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:48.285106 909 tablet_bootstrap.cc:492] T 2e5dc6a7de704dcdb0aec15124300b61 P d125d3941ab94bd5b970110e5791e171: No bootstrap required, opened a new log
I20240503 01:58:48.285229 909 ts_tablet_manager.cc:1367] T 2e5dc6a7de704dcdb0aec15124300b61 P d125d3941ab94bd5b970110e5791e171: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:58:48.285449 920 tablet_bootstrap.cc:492] T 8d1ad90f476a4b2b97f59259ef4bff9f P 82c39407898d4033af3c6a1a688a5197: No bootstrap required, opened a new log
I20240503 01:58:48.285524 920 ts_tablet_manager.cc:1367] T 8d1ad90f476a4b2b97f59259ef4bff9f P 82c39407898d4033af3c6a1a688a5197: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240503 01:58:48.285759 909 raft_consensus.cc:348] T 2e5dc6a7de704dcdb0aec15124300b61 P d125d3941ab94bd5b970110e5791e171 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.285864 909 raft_consensus.cc:374] T 2e5dc6a7de704dcdb0aec15124300b61 P d125d3941ab94bd5b970110e5791e171 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:48.285903 909 raft_consensus.cc:729] T 2e5dc6a7de704dcdb0aec15124300b61 P d125d3941ab94bd5b970110e5791e171 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d125d3941ab94bd5b970110e5791e171, State: Initialized, Role: FOLLOWER
I20240503 01:58:48.285799 844 raft_consensus.cc:2454] T 0488ca07075c4daca7b8f4d5f827ea56 P d125d3941ab94bd5b970110e5791e171 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cae31ce250aa4d64953e14da451c568c in term 1.
I20240503 01:58:48.286028 909 consensus_queue.cc:260] T 2e5dc6a7de704dcdb0aec15124300b61 P d125d3941ab94bd5b970110e5791e171 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.286051 920 raft_consensus.cc:348] T 8d1ad90f476a4b2b97f59259ef4bff9f P 82c39407898d4033af3c6a1a688a5197 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.286285 920 raft_consensus.cc:374] T 8d1ad90f476a4b2b97f59259ef4bff9f P 82c39407898d4033af3c6a1a688a5197 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:48.286260 517 leader_election.cc:302] T 0488ca07075c4daca7b8f4d5f827ea56 P cae31ce250aa4d64953e14da451c568c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cae31ce250aa4d64953e14da451c568c, d125d3941ab94bd5b970110e5791e171; no voters: 82c39407898d4033af3c6a1a688a5197
I20240503 01:58:48.286320 920 raft_consensus.cc:729] T 8d1ad90f476a4b2b97f59259ef4bff9f P 82c39407898d4033af3c6a1a688a5197 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82c39407898d4033af3c6a1a688a5197, State: Initialized, Role: FOLLOWER
I20240503 01:58:48.286423 920 consensus_queue.cc:260] T 8d1ad90f476a4b2b97f59259ef4bff9f P 82c39407898d4033af3c6a1a688a5197 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.286444 928 raft_consensus.cc:482] T 5a547015b4514b8286adb5c2e6356ba1 P 82c39407898d4033af3c6a1a688a5197 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:58:48.286473 914 raft_consensus.cc:2781] T 0488ca07075c4daca7b8f4d5f827ea56 P cae31ce250aa4d64953e14da451c568c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:58:48.286540 914 raft_consensus.cc:686] T 0488ca07075c4daca7b8f4d5f827ea56 P cae31ce250aa4d64953e14da451c568c [term 1 LEADER]: Becoming Leader. State: Replica: cae31ce250aa4d64953e14da451c568c, State: Running, Role: LEADER
I20240503 01:58:48.286530 928 raft_consensus.cc:504] T 5a547015b4514b8286adb5c2e6356ba1 P 82c39407898d4033af3c6a1a688a5197 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.286675 914 consensus_queue.cc:237] T 0488ca07075c4daca7b8f4d5f827ea56 P cae31ce250aa4d64953e14da451c568c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.286825 920 ts_tablet_manager.cc:1398] T 8d1ad90f476a4b2b97f59259ef4bff9f P 82c39407898d4033af3c6a1a688a5197: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:58:48.286973 920 tablet_bootstrap.cc:492] T 86f9a4762bf34b52a8d000a487226e5b P 82c39407898d4033af3c6a1a688a5197: Bootstrap starting.
I20240503 01:58:48.287952 928 leader_election.cc:288] T 5a547015b4514b8286adb5c2e6356ba1 P 82c39407898d4033af3c6a1a688a5197 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cae31ce250aa4d64953e14da451c568c (127.0.99.129:44403), d125d3941ab94bd5b970110e5791e171 (127.0.99.131:44747)
I20240503 01:58:48.288143 920 tablet_bootstrap.cc:654] T 86f9a4762bf34b52a8d000a487226e5b P 82c39407898d4033af3c6a1a688a5197: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:48.288472 440 catalog_manager.cc:5484] T 0488ca07075c4daca7b8f4d5f827ea56 P cae31ce250aa4d64953e14da451c568c reported cstate change: term changed from 0 to 1, leader changed from <none> to cae31ce250aa4d64953e14da451c568c (127.0.99.129). New cstate: current_term: 1 leader_uuid: "cae31ce250aa4d64953e14da451c568c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } health_report { overall_health: UNKNOWN } } }
I20240503 01:58:48.288909 909 ts_tablet_manager.cc:1398] T 2e5dc6a7de704dcdb0aec15124300b61 P d125d3941ab94bd5b970110e5791e171: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 01:58:48.289083 909 tablet_bootstrap.cc:492] T 8d1ad90f476a4b2b97f59259ef4bff9f P d125d3941ab94bd5b970110e5791e171: Bootstrap starting.
I20240503 01:58:48.291361 909 tablet_bootstrap.cc:654] T 8d1ad90f476a4b2b97f59259ef4bff9f P d125d3941ab94bd5b970110e5791e171: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:48.292063 920 tablet_bootstrap.cc:492] T 86f9a4762bf34b52a8d000a487226e5b P 82c39407898d4033af3c6a1a688a5197: No bootstrap required, opened a new log
I20240503 01:58:48.292096 973 raft_consensus.cc:482] T 8d1ad90f476a4b2b97f59259ef4bff9f P 82c39407898d4033af3c6a1a688a5197 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:58:48.292305 920 ts_tablet_manager.cc:1367] T 86f9a4762bf34b52a8d000a487226e5b P 82c39407898d4033af3c6a1a688a5197: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 01:58:48.292330 973 raft_consensus.cc:504] T 8d1ad90f476a4b2b97f59259ef4bff9f P 82c39407898d4033af3c6a1a688a5197 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.292752 973 leader_election.cc:288] T 8d1ad90f476a4b2b97f59259ef4bff9f P 82c39407898d4033af3c6a1a688a5197 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cae31ce250aa4d64953e14da451c568c (127.0.99.129:44403), d125d3941ab94bd5b970110e5791e171 (127.0.99.131:44747)
I20240503 01:58:48.292863 920 raft_consensus.cc:348] T 86f9a4762bf34b52a8d000a487226e5b P 82c39407898d4033af3c6a1a688a5197 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.292991 920 raft_consensus.cc:374] T 86f9a4762bf34b52a8d000a487226e5b P 82c39407898d4033af3c6a1a688a5197 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:48.293061 920 raft_consensus.cc:729] T 86f9a4762bf34b52a8d000a487226e5b P 82c39407898d4033af3c6a1a688a5197 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82c39407898d4033af3c6a1a688a5197, State: Initialized, Role: FOLLOWER
I20240503 01:58:48.293277 920 consensus_queue.cc:260] T 86f9a4762bf34b52a8d000a487226e5b P 82c39407898d4033af3c6a1a688a5197 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.293741 909 tablet_bootstrap.cc:492] T 8d1ad90f476a4b2b97f59259ef4bff9f P d125d3941ab94bd5b970110e5791e171: No bootstrap required, opened a new log
I20240503 01:58:48.293823 909 ts_tablet_manager.cc:1367] T 8d1ad90f476a4b2b97f59259ef4bff9f P d125d3941ab94bd5b970110e5791e171: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s
I20240503 01:58:48.294142 920 ts_tablet_manager.cc:1398] T 86f9a4762bf34b52a8d000a487226e5b P 82c39407898d4033af3c6a1a688a5197: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:58:48.294337 920 tablet_bootstrap.cc:492] T 0488ca07075c4daca7b8f4d5f827ea56 P 82c39407898d4033af3c6a1a688a5197: Bootstrap starting.
I20240503 01:58:48.294377 909 raft_consensus.cc:348] T 8d1ad90f476a4b2b97f59259ef4bff9f P d125d3941ab94bd5b970110e5791e171 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.294485 909 raft_consensus.cc:374] T 8d1ad90f476a4b2b97f59259ef4bff9f P d125d3941ab94bd5b970110e5791e171 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:48.294526 909 raft_consensus.cc:729] T 8d1ad90f476a4b2b97f59259ef4bff9f P d125d3941ab94bd5b970110e5791e171 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d125d3941ab94bd5b970110e5791e171, State: Initialized, Role: FOLLOWER
I20240503 01:58:48.294606 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a547015b4514b8286adb5c2e6356ba1" candidate_uuid: "82c39407898d4033af3c6a1a688a5197" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cae31ce250aa4d64953e14da451c568c" is_pre_election: true
I20240503 01:58:48.294646 909 consensus_queue.cc:260] T 8d1ad90f476a4b2b97f59259ef4bff9f P d125d3941ab94bd5b970110e5791e171 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.294711 583 raft_consensus.cc:2454] T 5a547015b4514b8286adb5c2e6356ba1 P cae31ce250aa4d64953e14da451c568c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82c39407898d4033af3c6a1a688a5197 in term 0.
I20240503 01:58:48.294904 909 ts_tablet_manager.cc:1398] T 8d1ad90f476a4b2b97f59259ef4bff9f P d125d3941ab94bd5b970110e5791e171: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:58:48.294978 582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d1ad90f476a4b2b97f59259ef4bff9f" candidate_uuid: "82c39407898d4033af3c6a1a688a5197" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cae31ce250aa4d64953e14da451c568c" is_pre_election: true
I20240503 01:58:48.295076 582 raft_consensus.cc:2454] T 8d1ad90f476a4b2b97f59259ef4bff9f P cae31ce250aa4d64953e14da451c568c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82c39407898d4033af3c6a1a688a5197 in term 0.
I20240503 01:58:48.295652 649 leader_election.cc:302] T 5a547015b4514b8286adb5c2e6356ba1 P 82c39407898d4033af3c6a1a688a5197 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82c39407898d4033af3c6a1a688a5197, cae31ce250aa4d64953e14da451c568c; no voters:
I20240503 01:58:48.295922 649 leader_election.cc:302] T 8d1ad90f476a4b2b97f59259ef4bff9f P 82c39407898d4033af3c6a1a688a5197 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82c39407898d4033af3c6a1a688a5197, cae31ce250aa4d64953e14da451c568c; no voters:
I20240503 01:58:48.296063 973 raft_consensus.cc:2781] T 5a547015b4514b8286adb5c2e6356ba1 P 82c39407898d4033af3c6a1a688a5197 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:58:48.296063 928 raft_consensus.cc:2781] T 8d1ad90f476a4b2b97f59259ef4bff9f P 82c39407898d4033af3c6a1a688a5197 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:58:48.296130 973 raft_consensus.cc:482] T 5a547015b4514b8286adb5c2e6356ba1 P 82c39407898d4033af3c6a1a688a5197 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:58:48.296141 928 raft_consensus.cc:482] T 8d1ad90f476a4b2b97f59259ef4bff9f P 82c39407898d4033af3c6a1a688a5197 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:58:48.296188 973 raft_consensus.cc:3037] T 5a547015b4514b8286adb5c2e6356ba1 P 82c39407898d4033af3c6a1a688a5197 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:48.296201 928 raft_consensus.cc:3037] T 8d1ad90f476a4b2b97f59259ef4bff9f P 82c39407898d4033af3c6a1a688a5197 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:48.296298 920 tablet_bootstrap.cc:654] T 0488ca07075c4daca7b8f4d5f827ea56 P 82c39407898d4033af3c6a1a688a5197: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:48.297340 928 raft_consensus.cc:504] T 8d1ad90f476a4b2b97f59259ef4bff9f P 82c39407898d4033af3c6a1a688a5197 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.297405 973 raft_consensus.cc:504] T 5a547015b4514b8286adb5c2e6356ba1 P 82c39407898d4033af3c6a1a688a5197 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.297652 928 leader_election.cc:288] T 8d1ad90f476a4b2b97f59259ef4bff9f P 82c39407898d4033af3c6a1a688a5197 [CANDIDATE]: Term 1 election: Requested vote from peers cae31ce250aa4d64953e14da451c568c (127.0.99.129:44403), d125d3941ab94bd5b970110e5791e171 (127.0.99.131:44747)
I20240503 01:58:48.297664 973 leader_election.cc:288] T 5a547015b4514b8286adb5c2e6356ba1 P 82c39407898d4033af3c6a1a688a5197 [CANDIDATE]: Term 1 election: Requested vote from peers cae31ce250aa4d64953e14da451c568c (127.0.99.129:44403), d125d3941ab94bd5b970110e5791e171 (127.0.99.131:44747)
I20240503 01:58:48.298162 582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d1ad90f476a4b2b97f59259ef4bff9f" candidate_uuid: "82c39407898d4033af3c6a1a688a5197" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cae31ce250aa4d64953e14da451c568c"
I20240503 01:58:48.298264 582 raft_consensus.cc:3037] T 8d1ad90f476a4b2b97f59259ef4bff9f P cae31ce250aa4d64953e14da451c568c [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:48.299288 582 raft_consensus.cc:2454] T 8d1ad90f476a4b2b97f59259ef4bff9f P cae31ce250aa4d64953e14da451c568c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82c39407898d4033af3c6a1a688a5197 in term 1.
I20240503 01:58:48.299525 920 tablet_bootstrap.cc:492] T 0488ca07075c4daca7b8f4d5f827ea56 P 82c39407898d4033af3c6a1a688a5197: No bootstrap required, opened a new log
I20240503 01:58:48.299492 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a547015b4514b8286adb5c2e6356ba1" candidate_uuid: "82c39407898d4033af3c6a1a688a5197" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cae31ce250aa4d64953e14da451c568c"
I20240503 01:58:48.299602 583 raft_consensus.cc:3037] T 5a547015b4514b8286adb5c2e6356ba1 P cae31ce250aa4d64953e14da451c568c [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:48.299600 649 leader_election.cc:302] T 8d1ad90f476a4b2b97f59259ef4bff9f P 82c39407898d4033af3c6a1a688a5197 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82c39407898d4033af3c6a1a688a5197, cae31ce250aa4d64953e14da451c568c; no voters:
I20240503 01:58:48.299731 845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d1ad90f476a4b2b97f59259ef4bff9f" candidate_uuid: "82c39407898d4033af3c6a1a688a5197" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d125d3941ab94bd5b970110e5791e171" is_pre_election: true
I20240503 01:58:48.299837 845 raft_consensus.cc:2454] T 8d1ad90f476a4b2b97f59259ef4bff9f P d125d3941ab94bd5b970110e5791e171 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82c39407898d4033af3c6a1a688a5197 in term 0.
I20240503 01:58:48.299904 973 raft_consensus.cc:2781] T 8d1ad90f476a4b2b97f59259ef4bff9f P 82c39407898d4033af3c6a1a688a5197 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:58:48.299898 844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a547015b4514b8286adb5c2e6356ba1" candidate_uuid: "82c39407898d4033af3c6a1a688a5197" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d125d3941ab94bd5b970110e5791e171" is_pre_election: true
I20240503 01:58:48.299613 920 ts_tablet_manager.cc:1367] T 0488ca07075c4daca7b8f4d5f827ea56 P 82c39407898d4033af3c6a1a688a5197: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 01:58:48.299990 973 raft_consensus.cc:686] T 8d1ad90f476a4b2b97f59259ef4bff9f P 82c39407898d4033af3c6a1a688a5197 [term 1 LEADER]: Becoming Leader. State: Replica: 82c39407898d4033af3c6a1a688a5197, State: Running, Role: LEADER
I20240503 01:58:48.300004 844 raft_consensus.cc:2454] T 5a547015b4514b8286adb5c2e6356ba1 P d125d3941ab94bd5b970110e5791e171 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82c39407898d4033af3c6a1a688a5197 in term 0.
I20240503 01:58:48.300209 843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a547015b4514b8286adb5c2e6356ba1" candidate_uuid: "82c39407898d4033af3c6a1a688a5197" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d125d3941ab94bd5b970110e5791e171"
I20240503 01:58:48.300311 843 raft_consensus.cc:3037] T 5a547015b4514b8286adb5c2e6356ba1 P d125d3941ab94bd5b970110e5791e171 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:48.300349 973 consensus_queue.cc:237] T 8d1ad90f476a4b2b97f59259ef4bff9f P 82c39407898d4033af3c6a1a688a5197 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.300426 583 raft_consensus.cc:2454] T 5a547015b4514b8286adb5c2e6356ba1 P cae31ce250aa4d64953e14da451c568c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82c39407898d4033af3c6a1a688a5197 in term 1.
I20240503 01:58:48.300634 920 raft_consensus.cc:348] T 0488ca07075c4daca7b8f4d5f827ea56 P 82c39407898d4033af3c6a1a688a5197 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.300761 920 raft_consensus.cc:374] T 0488ca07075c4daca7b8f4d5f827ea56 P 82c39407898d4033af3c6a1a688a5197 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:48.300797 920 raft_consensus.cc:729] T 0488ca07075c4daca7b8f4d5f827ea56 P 82c39407898d4033af3c6a1a688a5197 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82c39407898d4033af3c6a1a688a5197, State: Initialized, Role: FOLLOWER
I20240503 01:58:48.300860 842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d1ad90f476a4b2b97f59259ef4bff9f" candidate_uuid: "82c39407898d4033af3c6a1a688a5197" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d125d3941ab94bd5b970110e5791e171"
I20240503 01:58:48.300907 920 consensus_queue.cc:260] T 0488ca07075c4daca7b8f4d5f827ea56 P 82c39407898d4033af3c6a1a688a5197 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.300941 842 raft_consensus.cc:3037] T 8d1ad90f476a4b2b97f59259ef4bff9f P d125d3941ab94bd5b970110e5791e171 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:48.301358 920 ts_tablet_manager.cc:1398] T 0488ca07075c4daca7b8f4d5f827ea56 P 82c39407898d4033af3c6a1a688a5197: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:58:48.301597 843 raft_consensus.cc:2454] T 5a547015b4514b8286adb5c2e6356ba1 P d125d3941ab94bd5b970110e5791e171 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82c39407898d4033af3c6a1a688a5197 in term 1.
I20240503 01:58:48.301860 842 raft_consensus.cc:2454] T 8d1ad90f476a4b2b97f59259ef4bff9f P d125d3941ab94bd5b970110e5791e171 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82c39407898d4033af3c6a1a688a5197 in term 1.
I20240503 01:58:48.302456 648 leader_election.cc:302] T 5a547015b4514b8286adb5c2e6356ba1 P 82c39407898d4033af3c6a1a688a5197 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82c39407898d4033af3c6a1a688a5197, d125d3941ab94bd5b970110e5791e171; no voters:
I20240503 01:58:48.302704 928 raft_consensus.cc:2781] T 5a547015b4514b8286adb5c2e6356ba1 P 82c39407898d4033af3c6a1a688a5197 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:58:48.302779 928 raft_consensus.cc:686] T 5a547015b4514b8286adb5c2e6356ba1 P 82c39407898d4033af3c6a1a688a5197 [term 1 LEADER]: Becoming Leader. State: Replica: 82c39407898d4033af3c6a1a688a5197, State: Running, Role: LEADER
I20240503 01:58:48.302906 928 consensus_queue.cc:237] T 5a547015b4514b8286adb5c2e6356ba1 P 82c39407898d4033af3c6a1a688a5197 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.304334 441 catalog_manager.cc:5484] T 8d1ad90f476a4b2b97f59259ef4bff9f P 82c39407898d4033af3c6a1a688a5197 reported cstate change: term changed from 0 to 1, leader changed from <none> to 82c39407898d4033af3c6a1a688a5197 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "82c39407898d4033af3c6a1a688a5197" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } health_report { overall_health: UNKNOWN } } }
I20240503 01:58:48.306427 935 raft_consensus.cc:482] T 2f1cd41c7fc8437598f73938ae3f2a1d P d125d3941ab94bd5b970110e5791e171 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:58:48.306511 935 raft_consensus.cc:504] T 2f1cd41c7fc8437598f73938ae3f2a1d P d125d3941ab94bd5b970110e5791e171 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.307147 713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f1cd41c7fc8437598f73938ae3f2a1d" candidate_uuid: "d125d3941ab94bd5b970110e5791e171" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82c39407898d4033af3c6a1a688a5197" is_pre_election: true
I20240503 01:58:48.307298 713 raft_consensus.cc:2454] T 2f1cd41c7fc8437598f73938ae3f2a1d P 82c39407898d4033af3c6a1a688a5197 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d125d3941ab94bd5b970110e5791e171 in term 0.
I20240503 01:58:48.307334 935 leader_election.cc:288] T 2f1cd41c7fc8437598f73938ae3f2a1d P d125d3941ab94bd5b970110e5791e171 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82c39407898d4033af3c6a1a688a5197 (127.0.99.130:44599), cae31ce250aa4d64953e14da451c568c (127.0.99.129:44403)
I20240503 01:58:48.307535 778 leader_election.cc:302] T 2f1cd41c7fc8437598f73938ae3f2a1d P d125d3941ab94bd5b970110e5791e171 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82c39407898d4033af3c6a1a688a5197, d125d3941ab94bd5b970110e5791e171; no voters:
I20240503 01:58:48.307767 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f1cd41c7fc8437598f73938ae3f2a1d" candidate_uuid: "d125d3941ab94bd5b970110e5791e171" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cae31ce250aa4d64953e14da451c568c" is_pre_election: true
I20240503 01:58:48.307885 583 raft_consensus.cc:2454] T 2f1cd41c7fc8437598f73938ae3f2a1d P cae31ce250aa4d64953e14da451c568c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d125d3941ab94bd5b970110e5791e171 in term 0.
I20240503 01:58:48.308229 935 raft_consensus.cc:2781] T 2f1cd41c7fc8437598f73938ae3f2a1d P d125d3941ab94bd5b970110e5791e171 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:58:48.308284 935 raft_consensus.cc:482] T 2f1cd41c7fc8437598f73938ae3f2a1d P d125d3941ab94bd5b970110e5791e171 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:58:48.308327 935 raft_consensus.cc:3037] T 2f1cd41c7fc8437598f73938ae3f2a1d P d125d3941ab94bd5b970110e5791e171 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:48.308629 441 catalog_manager.cc:5484] T 5a547015b4514b8286adb5c2e6356ba1 P 82c39407898d4033af3c6a1a688a5197 reported cstate change: term changed from 0 to 1, leader changed from <none> to 82c39407898d4033af3c6a1a688a5197 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "82c39407898d4033af3c6a1a688a5197" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } health_report { overall_health: UNKNOWN } } }
I20240503 01:58:48.309120 935 raft_consensus.cc:504] T 2f1cd41c7fc8437598f73938ae3f2a1d P d125d3941ab94bd5b970110e5791e171 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.309391 935 leader_election.cc:288] T 2f1cd41c7fc8437598f73938ae3f2a1d P d125d3941ab94bd5b970110e5791e171 [CANDIDATE]: Term 1 election: Requested vote from peers 82c39407898d4033af3c6a1a688a5197 (127.0.99.130:44599), cae31ce250aa4d64953e14da451c568c (127.0.99.129:44403)
I20240503 01:58:48.309751 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f1cd41c7fc8437598f73938ae3f2a1d" candidate_uuid: "d125d3941ab94bd5b970110e5791e171" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cae31ce250aa4d64953e14da451c568c"
I20240503 01:58:48.309844 583 raft_consensus.cc:3037] T 2f1cd41c7fc8437598f73938ae3f2a1d P cae31ce250aa4d64953e14da451c568c [term 0 FOLLOWER]: Advancing to term 1
W20240503 01:58:48.310330 892 tablet.cc:2369] T 0488ca07075c4daca7b8f4d5f827ea56 P d125d3941ab94bd5b970110e5791e171: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 01:58:48.310335 713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f1cd41c7fc8437598f73938ae3f2a1d" candidate_uuid: "d125d3941ab94bd5b970110e5791e171" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82c39407898d4033af3c6a1a688a5197"
I20240503 01:58:48.310443 713 raft_consensus.cc:3037] T 2f1cd41c7fc8437598f73938ae3f2a1d P 82c39407898d4033af3c6a1a688a5197 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:48.310755 583 raft_consensus.cc:2454] T 2f1cd41c7fc8437598f73938ae3f2a1d P cae31ce250aa4d64953e14da451c568c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d125d3941ab94bd5b970110e5791e171 in term 1.
I20240503 01:58:48.311033 780 leader_election.cc:302] T 2f1cd41c7fc8437598f73938ae3f2a1d P d125d3941ab94bd5b970110e5791e171 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cae31ce250aa4d64953e14da451c568c, d125d3941ab94bd5b970110e5791e171; no voters:
I20240503 01:58:48.311169 935 raft_consensus.cc:2781] T 2f1cd41c7fc8437598f73938ae3f2a1d P d125d3941ab94bd5b970110e5791e171 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:58:48.311229 935 raft_consensus.cc:686] T 2f1cd41c7fc8437598f73938ae3f2a1d P d125d3941ab94bd5b970110e5791e171 [term 1 LEADER]: Becoming Leader. State: Replica: d125d3941ab94bd5b970110e5791e171, State: Running, Role: LEADER
I20240503 01:58:48.311317 713 raft_consensus.cc:2454] T 2f1cd41c7fc8437598f73938ae3f2a1d P 82c39407898d4033af3c6a1a688a5197 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d125d3941ab94bd5b970110e5791e171 in term 1.
I20240503 01:58:48.311348 935 consensus_queue.cc:237] T 2f1cd41c7fc8437598f73938ae3f2a1d P d125d3941ab94bd5b970110e5791e171 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.312927 441 catalog_manager.cc:5484] T 2f1cd41c7fc8437598f73938ae3f2a1d P d125d3941ab94bd5b970110e5791e171 reported cstate change: term changed from 0 to 1, leader changed from <none> to d125d3941ab94bd5b970110e5791e171 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "d125d3941ab94bd5b970110e5791e171" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } health_report { overall_health: HEALTHY } } }
I20240503 01:58:48.318018 921 raft_consensus.cc:482] T 2e5dc6a7de704dcdb0aec15124300b61 P cae31ce250aa4d64953e14da451c568c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:58:48.318202 921 raft_consensus.cc:504] T 2e5dc6a7de704dcdb0aec15124300b61 P cae31ce250aa4d64953e14da451c568c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.318564 921 leader_election.cc:288] T 2e5dc6a7de704dcdb0aec15124300b61 P cae31ce250aa4d64953e14da451c568c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82c39407898d4033af3c6a1a688a5197 (127.0.99.130:44599), d125d3941ab94bd5b970110e5791e171 (127.0.99.131:44747)
I20240503 01:58:48.318856 713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e5dc6a7de704dcdb0aec15124300b61" candidate_uuid: "cae31ce250aa4d64953e14da451c568c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82c39407898d4033af3c6a1a688a5197" is_pre_election: true
I20240503 01:58:48.318908 842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e5dc6a7de704dcdb0aec15124300b61" candidate_uuid: "cae31ce250aa4d64953e14da451c568c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d125d3941ab94bd5b970110e5791e171" is_pre_election: true
I20240503 01:58:48.318969 713 raft_consensus.cc:2454] T 2e5dc6a7de704dcdb0aec15124300b61 P 82c39407898d4033af3c6a1a688a5197 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cae31ce250aa4d64953e14da451c568c in term 0.
I20240503 01:58:48.319119 842 raft_consensus.cc:2454] T 2e5dc6a7de704dcdb0aec15124300b61 P d125d3941ab94bd5b970110e5791e171 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cae31ce250aa4d64953e14da451c568c in term 0.
I20240503 01:58:48.319231 516 leader_election.cc:302] T 2e5dc6a7de704dcdb0aec15124300b61 P cae31ce250aa4d64953e14da451c568c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82c39407898d4033af3c6a1a688a5197, cae31ce250aa4d64953e14da451c568c; no voters:
I20240503 01:58:48.319381 921 raft_consensus.cc:2781] T 2e5dc6a7de704dcdb0aec15124300b61 P cae31ce250aa4d64953e14da451c568c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:58:48.319428 921 raft_consensus.cc:482] T 2e5dc6a7de704dcdb0aec15124300b61 P cae31ce250aa4d64953e14da451c568c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:58:48.319460 921 raft_consensus.cc:3037] T 2e5dc6a7de704dcdb0aec15124300b61 P cae31ce250aa4d64953e14da451c568c [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:48.320529 921 raft_consensus.cc:504] T 2e5dc6a7de704dcdb0aec15124300b61 P cae31ce250aa4d64953e14da451c568c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.320863 921 leader_election.cc:288] T 2e5dc6a7de704dcdb0aec15124300b61 P cae31ce250aa4d64953e14da451c568c [CANDIDATE]: Term 1 election: Requested vote from peers 82c39407898d4033af3c6a1a688a5197 (127.0.99.130:44599), d125d3941ab94bd5b970110e5791e171 (127.0.99.131:44747)
I20240503 01:58:48.321141 842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e5dc6a7de704dcdb0aec15124300b61" candidate_uuid: "cae31ce250aa4d64953e14da451c568c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d125d3941ab94bd5b970110e5791e171"
I20240503 01:58:48.321251 842 raft_consensus.cc:3037] T 2e5dc6a7de704dcdb0aec15124300b61 P d125d3941ab94bd5b970110e5791e171 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:48.321242 713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e5dc6a7de704dcdb0aec15124300b61" candidate_uuid: "cae31ce250aa4d64953e14da451c568c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82c39407898d4033af3c6a1a688a5197"
I20240503 01:58:48.321331 713 raft_consensus.cc:3037] T 2e5dc6a7de704dcdb0aec15124300b61 P 82c39407898d4033af3c6a1a688a5197 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:48.322042 842 raft_consensus.cc:2454] T 2e5dc6a7de704dcdb0aec15124300b61 P d125d3941ab94bd5b970110e5791e171 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cae31ce250aa4d64953e14da451c568c in term 1.
I20240503 01:58:48.322171 713 raft_consensus.cc:2454] T 2e5dc6a7de704dcdb0aec15124300b61 P 82c39407898d4033af3c6a1a688a5197 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cae31ce250aa4d64953e14da451c568c in term 1.
I20240503 01:58:48.322340 517 leader_election.cc:302] T 2e5dc6a7de704dcdb0aec15124300b61 P cae31ce250aa4d64953e14da451c568c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cae31ce250aa4d64953e14da451c568c, d125d3941ab94bd5b970110e5791e171; no voters:
I20240503 01:58:48.322496 921 raft_consensus.cc:2781] T 2e5dc6a7de704dcdb0aec15124300b61 P cae31ce250aa4d64953e14da451c568c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:58:48.322571 921 raft_consensus.cc:686] T 2e5dc6a7de704dcdb0aec15124300b61 P cae31ce250aa4d64953e14da451c568c [term 1 LEADER]: Becoming Leader. State: Replica: cae31ce250aa4d64953e14da451c568c, State: Running, Role: LEADER
I20240503 01:58:48.322693 921 consensus_queue.cc:237] T 2e5dc6a7de704dcdb0aec15124300b61 P cae31ce250aa4d64953e14da451c568c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.324260 441 catalog_manager.cc:5484] T 2e5dc6a7de704dcdb0aec15124300b61 P cae31ce250aa4d64953e14da451c568c reported cstate change: term changed from 0 to 1, leader changed from <none> to cae31ce250aa4d64953e14da451c568c (127.0.99.129). New cstate: current_term: 1 leader_uuid: "cae31ce250aa4d64953e14da451c568c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } health_report { overall_health: UNKNOWN } } }
I20240503 01:58:48.383598 914 raft_consensus.cc:482] T 42a69d697389414a973d995a763d8219 P cae31ce250aa4d64953e14da451c568c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:58:48.383740 914 raft_consensus.cc:504] T 42a69d697389414a973d995a763d8219 P cae31ce250aa4d64953e14da451c568c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.384151 914 leader_election.cc:288] T 42a69d697389414a973d995a763d8219 P cae31ce250aa4d64953e14da451c568c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82c39407898d4033af3c6a1a688a5197 (127.0.99.130:44599), d125d3941ab94bd5b970110e5791e171 (127.0.99.131:44747)
I20240503 01:58:48.384573 713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42a69d697389414a973d995a763d8219" candidate_uuid: "cae31ce250aa4d64953e14da451c568c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82c39407898d4033af3c6a1a688a5197" is_pre_election: true
I20240503 01:58:48.384573 842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42a69d697389414a973d995a763d8219" candidate_uuid: "cae31ce250aa4d64953e14da451c568c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d125d3941ab94bd5b970110e5791e171" is_pre_election: true
I20240503 01:58:48.384745 713 raft_consensus.cc:2454] T 42a69d697389414a973d995a763d8219 P 82c39407898d4033af3c6a1a688a5197 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cae31ce250aa4d64953e14da451c568c in term 0.
I20240503 01:58:48.384750 842 raft_consensus.cc:2454] T 42a69d697389414a973d995a763d8219 P d125d3941ab94bd5b970110e5791e171 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cae31ce250aa4d64953e14da451c568c in term 0.
I20240503 01:58:48.385027 516 leader_election.cc:302] T 42a69d697389414a973d995a763d8219 P cae31ce250aa4d64953e14da451c568c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82c39407898d4033af3c6a1a688a5197, cae31ce250aa4d64953e14da451c568c; no voters:
I20240503 01:58:48.385188 914 raft_consensus.cc:2781] T 42a69d697389414a973d995a763d8219 P cae31ce250aa4d64953e14da451c568c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:58:48.385243 914 raft_consensus.cc:482] T 42a69d697389414a973d995a763d8219 P cae31ce250aa4d64953e14da451c568c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:58:48.385282 914 raft_consensus.cc:3037] T 42a69d697389414a973d995a763d8219 P cae31ce250aa4d64953e14da451c568c [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:48.386094 914 raft_consensus.cc:504] T 42a69d697389414a973d995a763d8219 P cae31ce250aa4d64953e14da451c568c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.386428 914 leader_election.cc:288] T 42a69d697389414a973d995a763d8219 P cae31ce250aa4d64953e14da451c568c [CANDIDATE]: Term 1 election: Requested vote from peers 82c39407898d4033af3c6a1a688a5197 (127.0.99.130:44599), d125d3941ab94bd5b970110e5791e171 (127.0.99.131:44747)
I20240503 01:58:48.386718 842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42a69d697389414a973d995a763d8219" candidate_uuid: "cae31ce250aa4d64953e14da451c568c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d125d3941ab94bd5b970110e5791e171"
I20240503 01:58:48.386742 713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42a69d697389414a973d995a763d8219" candidate_uuid: "cae31ce250aa4d64953e14da451c568c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82c39407898d4033af3c6a1a688a5197"
I20240503 01:58:48.386813 842 raft_consensus.cc:3037] T 42a69d697389414a973d995a763d8219 P d125d3941ab94bd5b970110e5791e171 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:48.386821 713 raft_consensus.cc:3037] T 42a69d697389414a973d995a763d8219 P 82c39407898d4033af3c6a1a688a5197 [term 0 FOLLOWER]: Advancing to term 1
W20240503 01:58:48.387423 761 tablet.cc:2369] T 0488ca07075c4daca7b8f4d5f827ea56 P 82c39407898d4033af3c6a1a688a5197: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 01:58:48.387563 713 raft_consensus.cc:2454] T 42a69d697389414a973d995a763d8219 P 82c39407898d4033af3c6a1a688a5197 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cae31ce250aa4d64953e14da451c568c in term 1.
I20240503 01:58:48.387770 842 raft_consensus.cc:2454] T 42a69d697389414a973d995a763d8219 P d125d3941ab94bd5b970110e5791e171 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cae31ce250aa4d64953e14da451c568c in term 1.
I20240503 01:58:48.387849 516 leader_election.cc:302] T 42a69d697389414a973d995a763d8219 P cae31ce250aa4d64953e14da451c568c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82c39407898d4033af3c6a1a688a5197, cae31ce250aa4d64953e14da451c568c; no voters:
I20240503 01:58:48.388022 914 raft_consensus.cc:2781] T 42a69d697389414a973d995a763d8219 P cae31ce250aa4d64953e14da451c568c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:58:48.388104 914 raft_consensus.cc:686] T 42a69d697389414a973d995a763d8219 P cae31ce250aa4d64953e14da451c568c [term 1 LEADER]: Becoming Leader. State: Replica: cae31ce250aa4d64953e14da451c568c, State: Running, Role: LEADER
I20240503 01:58:48.388234 914 consensus_queue.cc:237] T 42a69d697389414a973d995a763d8219 P cae31ce250aa4d64953e14da451c568c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.389832 441 catalog_manager.cc:5484] T 42a69d697389414a973d995a763d8219 P cae31ce250aa4d64953e14da451c568c reported cstate change: term changed from 0 to 1, leader changed from <none> to cae31ce250aa4d64953e14da451c568c (127.0.99.129). New cstate: current_term: 1 leader_uuid: "cae31ce250aa4d64953e14da451c568c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } health_report { overall_health: UNKNOWN } } }
I20240503 01:58:48.458312 980 raft_consensus.cc:482] T e38e1de36d82402c82b72dce9e765505 P 82c39407898d4033af3c6a1a688a5197 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:58:48.458477 980 raft_consensus.cc:504] T e38e1de36d82402c82b72dce9e765505 P 82c39407898d4033af3c6a1a688a5197 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.458747 973 raft_consensus.cc:482] T ccaa4613da8546f88fb844a6711baabc P 82c39407898d4033af3c6a1a688a5197 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:58:48.458848 973 raft_consensus.cc:504] T ccaa4613da8546f88fb844a6711baabc P 82c39407898d4033af3c6a1a688a5197 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.459060 980 leader_election.cc:288] T e38e1de36d82402c82b72dce9e765505 P 82c39407898d4033af3c6a1a688a5197 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cae31ce250aa4d64953e14da451c568c (127.0.99.129:44403), d125d3941ab94bd5b970110e5791e171 (127.0.99.131:44747)
I20240503 01:58:48.459203 973 leader_election.cc:288] T ccaa4613da8546f88fb844a6711baabc P 82c39407898d4033af3c6a1a688a5197 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cae31ce250aa4d64953e14da451c568c (127.0.99.129:44403), d125d3941ab94bd5b970110e5791e171 (127.0.99.131:44747)
I20240503 01:58:48.459604 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e38e1de36d82402c82b72dce9e765505" candidate_uuid: "82c39407898d4033af3c6a1a688a5197" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cae31ce250aa4d64953e14da451c568c" is_pre_election: true
I20240503 01:58:48.459775 583 raft_consensus.cc:2454] T e38e1de36d82402c82b72dce9e765505 P cae31ce250aa4d64953e14da451c568c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82c39407898d4033af3c6a1a688a5197 in term 0.
I20240503 01:58:48.460300 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccaa4613da8546f88fb844a6711baabc" candidate_uuid: "82c39407898d4033af3c6a1a688a5197" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cae31ce250aa4d64953e14da451c568c" is_pre_election: true
I20240503 01:58:48.460314 649 leader_election.cc:302] T e38e1de36d82402c82b72dce9e765505 P 82c39407898d4033af3c6a1a688a5197 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82c39407898d4033af3c6a1a688a5197, cae31ce250aa4d64953e14da451c568c; no voters:
I20240503 01:58:48.460357 843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccaa4613da8546f88fb844a6711baabc" candidate_uuid: "82c39407898d4033af3c6a1a688a5197" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d125d3941ab94bd5b970110e5791e171" is_pre_election: true
I20240503 01:58:48.460413 583 raft_consensus.cc:2454] T ccaa4613da8546f88fb844a6711baabc P cae31ce250aa4d64953e14da451c568c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82c39407898d4033af3c6a1a688a5197 in term 0.
I20240503 01:58:48.460481 973 raft_consensus.cc:2781] T e38e1de36d82402c82b72dce9e765505 P 82c39407898d4033af3c6a1a688a5197 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:58:48.460477 843 raft_consensus.cc:2454] T ccaa4613da8546f88fb844a6711baabc P d125d3941ab94bd5b970110e5791e171 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82c39407898d4033af3c6a1a688a5197 in term 0.
I20240503 01:58:48.460520 973 raft_consensus.cc:482] T e38e1de36d82402c82b72dce9e765505 P 82c39407898d4033af3c6a1a688a5197 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:58:48.460554 973 raft_consensus.cc:3037] T e38e1de36d82402c82b72dce9e765505 P 82c39407898d4033af3c6a1a688a5197 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:48.460741 649 leader_election.cc:302] T ccaa4613da8546f88fb844a6711baabc P 82c39407898d4033af3c6a1a688a5197 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82c39407898d4033af3c6a1a688a5197, cae31ce250aa4d64953e14da451c568c; no voters:
I20240503 01:58:48.460880 842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e38e1de36d82402c82b72dce9e765505" candidate_uuid: "82c39407898d4033af3c6a1a688a5197" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d125d3941ab94bd5b970110e5791e171" is_pre_election: true
I20240503 01:58:48.460966 842 raft_consensus.cc:2454] T e38e1de36d82402c82b72dce9e765505 P d125d3941ab94bd5b970110e5791e171 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82c39407898d4033af3c6a1a688a5197 in term 0.
I20240503 01:58:48.461077 980 raft_consensus.cc:2781] T ccaa4613da8546f88fb844a6711baabc P 82c39407898d4033af3c6a1a688a5197 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:58:48.461161 980 raft_consensus.cc:482] T ccaa4613da8546f88fb844a6711baabc P 82c39407898d4033af3c6a1a688a5197 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:58:48.461210 980 raft_consensus.cc:3037] T ccaa4613da8546f88fb844a6711baabc P 82c39407898d4033af3c6a1a688a5197 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:48.461684 973 raft_consensus.cc:504] T e38e1de36d82402c82b72dce9e765505 P 82c39407898d4033af3c6a1a688a5197 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.461972 980 raft_consensus.cc:504] T ccaa4613da8546f88fb844a6711baabc P 82c39407898d4033af3c6a1a688a5197 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.462250 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e38e1de36d82402c82b72dce9e765505" candidate_uuid: "82c39407898d4033af3c6a1a688a5197" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cae31ce250aa4d64953e14da451c568c"
I20240503 01:58:48.462312 980 leader_election.cc:288] T ccaa4613da8546f88fb844a6711baabc P 82c39407898d4033af3c6a1a688a5197 [CANDIDATE]: Term 1 election: Requested vote from peers cae31ce250aa4d64953e14da451c568c (127.0.99.129:44403), d125d3941ab94bd5b970110e5791e171 (127.0.99.131:44747)
I20240503 01:58:48.462342 583 raft_consensus.cc:3037] T e38e1de36d82402c82b72dce9e765505 P cae31ce250aa4d64953e14da451c568c [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:48.461972 973 leader_election.cc:288] T e38e1de36d82402c82b72dce9e765505 P 82c39407898d4033af3c6a1a688a5197 [CANDIDATE]: Term 1 election: Requested vote from peers cae31ce250aa4d64953e14da451c568c (127.0.99.129:44403), d125d3941ab94bd5b970110e5791e171 (127.0.99.131:44747)
I20240503 01:58:48.462924 914 raft_consensus.cc:482] T d2a69cc1ebd8418f85677c18ef9d80ef P cae31ce250aa4d64953e14da451c568c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:58:48.463016 914 raft_consensus.cc:504] T d2a69cc1ebd8418f85677c18ef9d80ef P cae31ce250aa4d64953e14da451c568c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } } peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } }
I20240503 01:58:48.463325 583 raft_consensus.cc:2454] T e38e1de36d82402c82b72dce9e765505 P cae31ce250aa4d64953e14da451c568c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82c39407898d4033af3c6a1a688a5197 in term 1.
I20240503 01:58:48.463346 914 leader_election.cc:288] T d2a69cc1ebd8418f85677c18ef9d80ef P cae31ce250aa4d64953e14da451c568c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d125d3941ab94bd5b970110e5791e171 (127.0.99.131:44747), 82c39407898d4033af3c6a1a688a5197 (127.0.99.130:44599)
I20240503 01:58:48.463436 973 raft_consensus.cc:482] T 86f9a4762bf34b52a8d000a487226e5b P 82c39407898d4033af3c6a1a688a5197 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:58:48.463502 973 raft_consensus.cc:504] T 86f9a4762bf34b52a8d000a487226e5b P 82c39407898d4033af3c6a1a688a5197 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.463713 844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2a69cc1ebd8418f85677c18ef9d80ef" candidate_uuid: "cae31ce250aa4d64953e14da451c568c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d125d3941ab94bd5b970110e5791e171" is_pre_election: true
I20240503 01:58:48.463801 973 leader_election.cc:288] T 86f9a4762bf34b52a8d000a487226e5b P 82c39407898d4033af3c6a1a688a5197 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cae31ce250aa4d64953e14da451c568c (127.0.99.129:44403), d125d3941ab94bd5b970110e5791e171 (127.0.99.131:44747)
I20240503 01:58:48.463804 844 raft_consensus.cc:2454] T d2a69cc1ebd8418f85677c18ef9d80ef P d125d3941ab94bd5b970110e5791e171 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cae31ce250aa4d64953e14da451c568c in term 0.
I20240503 01:58:48.463884 649 leader_election.cc:302] T e38e1de36d82402c82b72dce9e765505 P 82c39407898d4033af3c6a1a688a5197 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82c39407898d4033af3c6a1a688a5197, cae31ce250aa4d64953e14da451c568c; no voters:
I20240503 01:58:48.464061 713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2a69cc1ebd8418f85677c18ef9d80ef" candidate_uuid: "cae31ce250aa4d64953e14da451c568c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82c39407898d4033af3c6a1a688a5197" is_pre_election: true
I20240503 01:58:48.464147 973 raft_consensus.cc:2781] T e38e1de36d82402c82b72dce9e765505 P 82c39407898d4033af3c6a1a688a5197 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:58:48.464174 713 raft_consensus.cc:2454] T d2a69cc1ebd8418f85677c18ef9d80ef P 82c39407898d4033af3c6a1a688a5197 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cae31ce250aa4d64953e14da451c568c in term 0.
I20240503 01:58:48.464226 973 raft_consensus.cc:686] T e38e1de36d82402c82b72dce9e765505 P 82c39407898d4033af3c6a1a688a5197 [term 1 LEADER]: Becoming Leader. State: Replica: 82c39407898d4033af3c6a1a688a5197, State: Running, Role: LEADER
I20240503 01:58:48.464301 843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccaa4613da8546f88fb844a6711baabc" candidate_uuid: "82c39407898d4033af3c6a1a688a5197" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d125d3941ab94bd5b970110e5791e171"
I20240503 01:58:48.464381 843 raft_consensus.cc:3037] T ccaa4613da8546f88fb844a6711baabc P d125d3941ab94bd5b970110e5791e171 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:48.464365 973 consensus_queue.cc:237] T e38e1de36d82402c82b72dce9e765505 P 82c39407898d4033af3c6a1a688a5197 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.464393 582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86f9a4762bf34b52a8d000a487226e5b" candidate_uuid: "82c39407898d4033af3c6a1a688a5197" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cae31ce250aa4d64953e14da451c568c" is_pre_election: true
I20240503 01:58:48.464457 516 leader_election.cc:302] T d2a69cc1ebd8418f85677c18ef9d80ef P cae31ce250aa4d64953e14da451c568c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82c39407898d4033af3c6a1a688a5197, cae31ce250aa4d64953e14da451c568c; no voters:
I20240503 01:58:48.464556 582 raft_consensus.cc:2379] T 86f9a4762bf34b52a8d000a487226e5b P cae31ce250aa4d64953e14da451c568c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82c39407898d4033af3c6a1a688a5197 in current term 1: Already voted for candidate d125d3941ab94bd5b970110e5791e171 in this term.
I20240503 01:58:48.465021 844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86f9a4762bf34b52a8d000a487226e5b" candidate_uuid: "82c39407898d4033af3c6a1a688a5197" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d125d3941ab94bd5b970110e5791e171" is_pre_election: true
I20240503 01:58:48.465219 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccaa4613da8546f88fb844a6711baabc" candidate_uuid: "82c39407898d4033af3c6a1a688a5197" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cae31ce250aa4d64953e14da451c568c"
I20240503 01:58:48.465317 583 raft_consensus.cc:3037] T ccaa4613da8546f88fb844a6711baabc P cae31ce250aa4d64953e14da451c568c [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:48.465328 843 raft_consensus.cc:2454] T ccaa4613da8546f88fb844a6711baabc P d125d3941ab94bd5b970110e5791e171 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82c39407898d4033af3c6a1a688a5197 in term 1.
I20240503 01:58:48.465401 648 leader_election.cc:302] T 86f9a4762bf34b52a8d000a487226e5b P 82c39407898d4033af3c6a1a688a5197 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82c39407898d4033af3c6a1a688a5197; no voters: cae31ce250aa4d64953e14da451c568c, d125d3941ab94bd5b970110e5791e171
I20240503 01:58:48.466143 583 raft_consensus.cc:2454] T ccaa4613da8546f88fb844a6711baabc P cae31ce250aa4d64953e14da451c568c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82c39407898d4033af3c6a1a688a5197 in term 1.
I20240503 01:58:48.466182 648 leader_election.cc:302] T ccaa4613da8546f88fb844a6711baabc P 82c39407898d4033af3c6a1a688a5197 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82c39407898d4033af3c6a1a688a5197, d125d3941ab94bd5b970110e5791e171; no voters:
I20240503 01:58:48.466272 980 raft_consensus.cc:3037] T 86f9a4762bf34b52a8d000a487226e5b P 82c39407898d4033af3c6a1a688a5197 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:48.466302 973 raft_consensus.cc:2781] T ccaa4613da8546f88fb844a6711baabc P 82c39407898d4033af3c6a1a688a5197 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:58:48.466430 973 raft_consensus.cc:686] T ccaa4613da8546f88fb844a6711baabc P 82c39407898d4033af3c6a1a688a5197 [term 1 LEADER]: Becoming Leader. State: Replica: 82c39407898d4033af3c6a1a688a5197, State: Running, Role: LEADER
I20240503 01:58:48.466534 973 consensus_queue.cc:237] T ccaa4613da8546f88fb844a6711baabc P 82c39407898d4033af3c6a1a688a5197 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.466732 914 raft_consensus.cc:2781] T d2a69cc1ebd8418f85677c18ef9d80ef P cae31ce250aa4d64953e14da451c568c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:58:48.466779 914 raft_consensus.cc:482] T d2a69cc1ebd8418f85677c18ef9d80ef P cae31ce250aa4d64953e14da451c568c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:58:48.466809 914 raft_consensus.cc:3037] T d2a69cc1ebd8418f85677c18ef9d80ef P cae31ce250aa4d64953e14da451c568c [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:48.466961 842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e38e1de36d82402c82b72dce9e765505" candidate_uuid: "82c39407898d4033af3c6a1a688a5197" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d125d3941ab94bd5b970110e5791e171"
I20240503 01:58:48.467036 842 raft_consensus.cc:3037] T e38e1de36d82402c82b72dce9e765505 P d125d3941ab94bd5b970110e5791e171 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:48.467113 980 raft_consensus.cc:2726] T 86f9a4762bf34b52a8d000a487226e5b P 82c39407898d4033af3c6a1a688a5197 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 01:58:48.467761 441 catalog_manager.cc:5484] T e38e1de36d82402c82b72dce9e765505 P 82c39407898d4033af3c6a1a688a5197 reported cstate change: term changed from 0 to 1, leader changed from <none> to 82c39407898d4033af3c6a1a688a5197 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "82c39407898d4033af3c6a1a688a5197" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } health_report { overall_health: UNKNOWN } } }
I20240503 01:58:48.467875 842 raft_consensus.cc:2454] T e38e1de36d82402c82b72dce9e765505 P d125d3941ab94bd5b970110e5791e171 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82c39407898d4033af3c6a1a688a5197 in term 1.
I20240503 01:58:48.470263 914 raft_consensus.cc:504] T d2a69cc1ebd8418f85677c18ef9d80ef P cae31ce250aa4d64953e14da451c568c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } } peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } }
I20240503 01:58:48.470568 914 leader_election.cc:288] T d2a69cc1ebd8418f85677c18ef9d80ef P cae31ce250aa4d64953e14da451c568c [CANDIDATE]: Term 1 election: Requested vote from peers d125d3941ab94bd5b970110e5791e171 (127.0.99.131:44747), 82c39407898d4033af3c6a1a688a5197 (127.0.99.130:44599)
I20240503 01:58:48.471099 713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2a69cc1ebd8418f85677c18ef9d80ef" candidate_uuid: "cae31ce250aa4d64953e14da451c568c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82c39407898d4033af3c6a1a688a5197"
I20240503 01:58:48.471189 713 raft_consensus.cc:3037] T d2a69cc1ebd8418f85677c18ef9d80ef P 82c39407898d4033af3c6a1a688a5197 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:48.471601 441 catalog_manager.cc:5484] T ccaa4613da8546f88fb844a6711baabc P 82c39407898d4033af3c6a1a688a5197 reported cstate change: term changed from 0 to 1, leader changed from <none> to 82c39407898d4033af3c6a1a688a5197 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "82c39407898d4033af3c6a1a688a5197" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } health_report { overall_health: UNKNOWN } } }
I20240503 01:58:48.471930 713 raft_consensus.cc:2454] T d2a69cc1ebd8418f85677c18ef9d80ef P 82c39407898d4033af3c6a1a688a5197 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cae31ce250aa4d64953e14da451c568c in term 1.
I20240503 01:58:48.472227 516 leader_election.cc:302] T d2a69cc1ebd8418f85677c18ef9d80ef P cae31ce250aa4d64953e14da451c568c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82c39407898d4033af3c6a1a688a5197, cae31ce250aa4d64953e14da451c568c; no voters:
I20240503 01:58:48.472357 843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2a69cc1ebd8418f85677c18ef9d80ef" candidate_uuid: "cae31ce250aa4d64953e14da451c568c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d125d3941ab94bd5b970110e5791e171"
I20240503 01:58:48.472389 914 raft_consensus.cc:2781] T d2a69cc1ebd8418f85677c18ef9d80ef P cae31ce250aa4d64953e14da451c568c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:58:48.472467 843 raft_consensus.cc:3037] T d2a69cc1ebd8418f85677c18ef9d80ef P d125d3941ab94bd5b970110e5791e171 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:48.472507 914 raft_consensus.cc:686] T d2a69cc1ebd8418f85677c18ef9d80ef P cae31ce250aa4d64953e14da451c568c [term 1 LEADER]: Becoming Leader. State: Replica: cae31ce250aa4d64953e14da451c568c, State: Running, Role: LEADER
I20240503 01:58:48.472661 914 consensus_queue.cc:237] T d2a69cc1ebd8418f85677c18ef9d80ef P cae31ce250aa4d64953e14da451c568c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } } peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } }
I20240503 01:58:48.473340 843 raft_consensus.cc:2454] T d2a69cc1ebd8418f85677c18ef9d80ef P d125d3941ab94bd5b970110e5791e171 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cae31ce250aa4d64953e14da451c568c in term 1.
I20240503 01:58:48.474226 438 catalog_manager.cc:5484] T d2a69cc1ebd8418f85677c18ef9d80ef P cae31ce250aa4d64953e14da451c568c reported cstate change: term changed from 0 to 1, leader changed from <none> to cae31ce250aa4d64953e14da451c568c (127.0.99.129). New cstate: current_term: 1 leader_uuid: "cae31ce250aa4d64953e14da451c568c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } health_report { overall_health: HEALTHY } } }
I20240503 01:58:48.484328 935 raft_consensus.cc:482] T e0970cda8db54a8593f7d482e861da46 P d125d3941ab94bd5b970110e5791e171 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:58:48.484417 935 raft_consensus.cc:504] T e0970cda8db54a8593f7d482e861da46 P d125d3941ab94bd5b970110e5791e171 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.484791 935 leader_election.cc:288] T e0970cda8db54a8593f7d482e861da46 P d125d3941ab94bd5b970110e5791e171 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82c39407898d4033af3c6a1a688a5197 (127.0.99.130:44599), cae31ce250aa4d64953e14da451c568c (127.0.99.129:44403)
I20240503 01:58:48.485116 713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0970cda8db54a8593f7d482e861da46" candidate_uuid: "d125d3941ab94bd5b970110e5791e171" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82c39407898d4033af3c6a1a688a5197" is_pre_election: true
I20240503 01:58:48.485127 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0970cda8db54a8593f7d482e861da46" candidate_uuid: "d125d3941ab94bd5b970110e5791e171" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cae31ce250aa4d64953e14da451c568c" is_pre_election: true
I20240503 01:58:48.485236 583 raft_consensus.cc:2454] T e0970cda8db54a8593f7d482e861da46 P cae31ce250aa4d64953e14da451c568c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d125d3941ab94bd5b970110e5791e171 in term 0.
I20240503 01:58:48.485236 713 raft_consensus.cc:2454] T e0970cda8db54a8593f7d482e861da46 P 82c39407898d4033af3c6a1a688a5197 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d125d3941ab94bd5b970110e5791e171 in term 0.
I20240503 01:58:48.485482 778 leader_election.cc:302] T e0970cda8db54a8593f7d482e861da46 P d125d3941ab94bd5b970110e5791e171 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82c39407898d4033af3c6a1a688a5197, d125d3941ab94bd5b970110e5791e171; no voters:
I20240503 01:58:48.485641 935 raft_consensus.cc:2781] T e0970cda8db54a8593f7d482e861da46 P d125d3941ab94bd5b970110e5791e171 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:58:48.485699 935 raft_consensus.cc:482] T e0970cda8db54a8593f7d482e861da46 P d125d3941ab94bd5b970110e5791e171 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:58:48.485733 935 raft_consensus.cc:3037] T e0970cda8db54a8593f7d482e861da46 P d125d3941ab94bd5b970110e5791e171 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:48.486470 935 raft_consensus.cc:504] T e0970cda8db54a8593f7d482e861da46 P d125d3941ab94bd5b970110e5791e171 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.486747 935 leader_election.cc:288] T e0970cda8db54a8593f7d482e861da46 P d125d3941ab94bd5b970110e5791e171 [CANDIDATE]: Term 1 election: Requested vote from peers 82c39407898d4033af3c6a1a688a5197 (127.0.99.130:44599), cae31ce250aa4d64953e14da451c568c (127.0.99.129:44403)
I20240503 01:58:48.487049 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0970cda8db54a8593f7d482e861da46" candidate_uuid: "d125d3941ab94bd5b970110e5791e171" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cae31ce250aa4d64953e14da451c568c"
I20240503 01:58:48.487066 713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0970cda8db54a8593f7d482e861da46" candidate_uuid: "d125d3941ab94bd5b970110e5791e171" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82c39407898d4033af3c6a1a688a5197"
I20240503 01:58:48.487154 583 raft_consensus.cc:3037] T e0970cda8db54a8593f7d482e861da46 P cae31ce250aa4d64953e14da451c568c [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:48.487154 713 raft_consensus.cc:3037] T e0970cda8db54a8593f7d482e861da46 P 82c39407898d4033af3c6a1a688a5197 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:48.488010 713 raft_consensus.cc:2454] T e0970cda8db54a8593f7d482e861da46 P 82c39407898d4033af3c6a1a688a5197 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d125d3941ab94bd5b970110e5791e171 in term 1.
I20240503 01:58:48.488010 583 raft_consensus.cc:2454] T e0970cda8db54a8593f7d482e861da46 P cae31ce250aa4d64953e14da451c568c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d125d3941ab94bd5b970110e5791e171 in term 1.
I20240503 01:58:48.488288 778 leader_election.cc:302] T e0970cda8db54a8593f7d482e861da46 P d125d3941ab94bd5b970110e5791e171 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82c39407898d4033af3c6a1a688a5197, d125d3941ab94bd5b970110e5791e171; no voters:
I20240503 01:58:48.488471 935 raft_consensus.cc:2781] T e0970cda8db54a8593f7d482e861da46 P d125d3941ab94bd5b970110e5791e171 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:58:48.488562 935 raft_consensus.cc:686] T e0970cda8db54a8593f7d482e861da46 P d125d3941ab94bd5b970110e5791e171 [term 1 LEADER]: Becoming Leader. State: Replica: d125d3941ab94bd5b970110e5791e171, State: Running, Role: LEADER
I20240503 01:58:48.488708 935 consensus_queue.cc:237] T e0970cda8db54a8593f7d482e861da46 P d125d3941ab94bd5b970110e5791e171 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.490149 935 raft_consensus.cc:482] T 9f1fb01218234ec4a13fd6f9dfb3fdd6 P d125d3941ab94bd5b970110e5791e171 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:58:48.490240 935 raft_consensus.cc:504] T 9f1fb01218234ec4a13fd6f9dfb3fdd6 P d125d3941ab94bd5b970110e5791e171 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.490314 438 catalog_manager.cc:5484] T e0970cda8db54a8593f7d482e861da46 P d125d3941ab94bd5b970110e5791e171 reported cstate change: term changed from 0 to 1, leader changed from <none> to d125d3941ab94bd5b970110e5791e171 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "d125d3941ab94bd5b970110e5791e171" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } health_report { overall_health: HEALTHY } } }
I20240503 01:58:48.490542 935 leader_election.cc:288] T 9f1fb01218234ec4a13fd6f9dfb3fdd6 P d125d3941ab94bd5b970110e5791e171 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cae31ce250aa4d64953e14da451c568c (127.0.99.129:44403), 82c39407898d4033af3c6a1a688a5197 (127.0.99.130:44599)
I20240503 01:58:48.490752 980 raft_consensus.cc:482] T 0488ca07075c4daca7b8f4d5f827ea56 P 82c39407898d4033af3c6a1a688a5197 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:58:48.490864 980 raft_consensus.cc:504] T 0488ca07075c4daca7b8f4d5f827ea56 P 82c39407898d4033af3c6a1a688a5197 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.490867 713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f1fb01218234ec4a13fd6f9dfb3fdd6" candidate_uuid: "d125d3941ab94bd5b970110e5791e171" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82c39407898d4033af3c6a1a688a5197" is_pre_election: true
I20240503 01:58:48.491036 713 raft_consensus.cc:2454] T 9f1fb01218234ec4a13fd6f9dfb3fdd6 P 82c39407898d4033af3c6a1a688a5197 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d125d3941ab94bd5b970110e5791e171 in term 0.
I20240503 01:58:48.491142 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f1fb01218234ec4a13fd6f9dfb3fdd6" candidate_uuid: "d125d3941ab94bd5b970110e5791e171" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cae31ce250aa4d64953e14da451c568c" is_pre_election: true
I20240503 01:58:48.491269 583 raft_consensus.cc:2454] T 9f1fb01218234ec4a13fd6f9dfb3fdd6 P cae31ce250aa4d64953e14da451c568c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d125d3941ab94bd5b970110e5791e171 in term 0.
I20240503 01:58:48.491354 778 leader_election.cc:302] T 9f1fb01218234ec4a13fd6f9dfb3fdd6 P d125d3941ab94bd5b970110e5791e171 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82c39407898d4033af3c6a1a688a5197, d125d3941ab94bd5b970110e5791e171; no voters:
I20240503 01:58:48.491497 980 leader_election.cc:288] T 0488ca07075c4daca7b8f4d5f827ea56 P 82c39407898d4033af3c6a1a688a5197 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cae31ce250aa4d64953e14da451c568c (127.0.99.129:44403), d125d3941ab94bd5b970110e5791e171 (127.0.99.131:44747)
I20240503 01:58:48.492095 935 raft_consensus.cc:2781] T 9f1fb01218234ec4a13fd6f9dfb3fdd6 P d125d3941ab94bd5b970110e5791e171 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:58:48.492172 935 raft_consensus.cc:482] T 9f1fb01218234ec4a13fd6f9dfb3fdd6 P d125d3941ab94bd5b970110e5791e171 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:58:48.492210 935 raft_consensus.cc:3037] T 9f1fb01218234ec4a13fd6f9dfb3fdd6 P d125d3941ab94bd5b970110e5791e171 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:48.492211 843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0488ca07075c4daca7b8f4d5f827ea56" candidate_uuid: "82c39407898d4033af3c6a1a688a5197" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d125d3941ab94bd5b970110e5791e171" is_pre_election: true
I20240503 01:58:48.491850 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0488ca07075c4daca7b8f4d5f827ea56" candidate_uuid: "82c39407898d4033af3c6a1a688a5197" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cae31ce250aa4d64953e14da451c568c" is_pre_election: true
I20240503 01:58:48.492422 843 raft_consensus.cc:2379] T 0488ca07075c4daca7b8f4d5f827ea56 P d125d3941ab94bd5b970110e5791e171 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82c39407898d4033af3c6a1a688a5197 in current term 1: Already voted for candidate cae31ce250aa4d64953e14da451c568c in this term.
I20240503 01:58:48.492766 648 leader_election.cc:302] T 0488ca07075c4daca7b8f4d5f827ea56 P 82c39407898d4033af3c6a1a688a5197 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82c39407898d4033af3c6a1a688a5197; no voters: cae31ce250aa4d64953e14da451c568c, d125d3941ab94bd5b970110e5791e171
I20240503 01:58:48.493273 935 raft_consensus.cc:504] T 9f1fb01218234ec4a13fd6f9dfb3fdd6 P d125d3941ab94bd5b970110e5791e171 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.493588 935 leader_election.cc:288] T 9f1fb01218234ec4a13fd6f9dfb3fdd6 P d125d3941ab94bd5b970110e5791e171 [CANDIDATE]: Term 1 election: Requested vote from peers cae31ce250aa4d64953e14da451c568c (127.0.99.129:44403), 82c39407898d4033af3c6a1a688a5197 (127.0.99.130:44599)
I20240503 01:58:48.493839 980 raft_consensus.cc:3037] T 0488ca07075c4daca7b8f4d5f827ea56 P 82c39407898d4033af3c6a1a688a5197 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:48.493862 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f1fb01218234ec4a13fd6f9dfb3fdd6" candidate_uuid: "d125d3941ab94bd5b970110e5791e171" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cae31ce250aa4d64953e14da451c568c"
I20240503 01:58:48.493862 713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f1fb01218234ec4a13fd6f9dfb3fdd6" candidate_uuid: "d125d3941ab94bd5b970110e5791e171" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82c39407898d4033af3c6a1a688a5197"
I20240503 01:58:48.493952 583 raft_consensus.cc:3037] T 9f1fb01218234ec4a13fd6f9dfb3fdd6 P cae31ce250aa4d64953e14da451c568c [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:48.493952 713 raft_consensus.cc:3037] T 9f1fb01218234ec4a13fd6f9dfb3fdd6 P 82c39407898d4033af3c6a1a688a5197 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:48.494526 980 raft_consensus.cc:2726] T 0488ca07075c4daca7b8f4d5f827ea56 P 82c39407898d4033af3c6a1a688a5197 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 01:58:48.494782 713 raft_consensus.cc:2454] T 9f1fb01218234ec4a13fd6f9dfb3fdd6 P 82c39407898d4033af3c6a1a688a5197 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d125d3941ab94bd5b970110e5791e171 in term 1.
I20240503 01:58:48.494782 583 raft_consensus.cc:2454] T 9f1fb01218234ec4a13fd6f9dfb3fdd6 P cae31ce250aa4d64953e14da451c568c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d125d3941ab94bd5b970110e5791e171 in term 1.
I20240503 01:58:48.495067 780 leader_election.cc:302] T 9f1fb01218234ec4a13fd6f9dfb3fdd6 P d125d3941ab94bd5b970110e5791e171 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cae31ce250aa4d64953e14da451c568c, d125d3941ab94bd5b970110e5791e171; no voters:
I20240503 01:58:48.495229 935 raft_consensus.cc:2781] T 9f1fb01218234ec4a13fd6f9dfb3fdd6 P d125d3941ab94bd5b970110e5791e171 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:58:48.495321 935 raft_consensus.cc:686] T 9f1fb01218234ec4a13fd6f9dfb3fdd6 P d125d3941ab94bd5b970110e5791e171 [term 1 LEADER]: Becoming Leader. State: Replica: d125d3941ab94bd5b970110e5791e171, State: Running, Role: LEADER
I20240503 01:58:48.495440 935 consensus_queue.cc:237] T 9f1fb01218234ec4a13fd6f9dfb3fdd6 P d125d3941ab94bd5b970110e5791e171 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.496737 438 catalog_manager.cc:5484] T 9f1fb01218234ec4a13fd6f9dfb3fdd6 P d125d3941ab94bd5b970110e5791e171 reported cstate change: term changed from 0 to 1, leader changed from <none> to d125d3941ab94bd5b970110e5791e171 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "d125d3941ab94bd5b970110e5791e171" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } health_report { overall_health: HEALTHY } } }
I20240503 01:58:48.504985 914 raft_consensus.cc:482] T 84bfa8f64219448d8d49a30d50068c22 P cae31ce250aa4d64953e14da451c568c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:58:48.505084 914 raft_consensus.cc:504] T 84bfa8f64219448d8d49a30d50068c22 P cae31ce250aa4d64953e14da451c568c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.505434 914 leader_election.cc:288] T 84bfa8f64219448d8d49a30d50068c22 P cae31ce250aa4d64953e14da451c568c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82c39407898d4033af3c6a1a688a5197 (127.0.99.130:44599), d125d3941ab94bd5b970110e5791e171 (127.0.99.131:44747)
I20240503 01:58:48.505753 843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84bfa8f64219448d8d49a30d50068c22" candidate_uuid: "cae31ce250aa4d64953e14da451c568c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d125d3941ab94bd5b970110e5791e171" is_pre_election: true
I20240503 01:58:48.505753 713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84bfa8f64219448d8d49a30d50068c22" candidate_uuid: "cae31ce250aa4d64953e14da451c568c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82c39407898d4033af3c6a1a688a5197" is_pre_election: true
I20240503 01:58:48.505854 713 raft_consensus.cc:2454] T 84bfa8f64219448d8d49a30d50068c22 P 82c39407898d4033af3c6a1a688a5197 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cae31ce250aa4d64953e14da451c568c in term 0.
I20240503 01:58:48.505863 843 raft_consensus.cc:2454] T 84bfa8f64219448d8d49a30d50068c22 P d125d3941ab94bd5b970110e5791e171 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cae31ce250aa4d64953e14da451c568c in term 0.
I20240503 01:58:48.506122 517 leader_election.cc:302] T 84bfa8f64219448d8d49a30d50068c22 P cae31ce250aa4d64953e14da451c568c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cae31ce250aa4d64953e14da451c568c, d125d3941ab94bd5b970110e5791e171; no voters:
I20240503 01:58:48.506261 914 raft_consensus.cc:2781] T 84bfa8f64219448d8d49a30d50068c22 P cae31ce250aa4d64953e14da451c568c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:58:48.506332 914 raft_consensus.cc:482] T 84bfa8f64219448d8d49a30d50068c22 P cae31ce250aa4d64953e14da451c568c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:58:48.506361 914 raft_consensus.cc:3037] T 84bfa8f64219448d8d49a30d50068c22 P cae31ce250aa4d64953e14da451c568c [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:48.507009 914 raft_consensus.cc:504] T 84bfa8f64219448d8d49a30d50068c22 P cae31ce250aa4d64953e14da451c568c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.507256 914 leader_election.cc:288] T 84bfa8f64219448d8d49a30d50068c22 P cae31ce250aa4d64953e14da451c568c [CANDIDATE]: Term 1 election: Requested vote from peers 82c39407898d4033af3c6a1a688a5197 (127.0.99.130:44599), d125d3941ab94bd5b970110e5791e171 (127.0.99.131:44747)
I20240503 01:58:48.507531 713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84bfa8f64219448d8d49a30d50068c22" candidate_uuid: "cae31ce250aa4d64953e14da451c568c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82c39407898d4033af3c6a1a688a5197"
I20240503 01:58:48.507560 843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84bfa8f64219448d8d49a30d50068c22" candidate_uuid: "cae31ce250aa4d64953e14da451c568c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d125d3941ab94bd5b970110e5791e171"
I20240503 01:58:48.507637 843 raft_consensus.cc:3037] T 84bfa8f64219448d8d49a30d50068c22 P d125d3941ab94bd5b970110e5791e171 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:48.507645 713 raft_consensus.cc:3037] T 84bfa8f64219448d8d49a30d50068c22 P 82c39407898d4033af3c6a1a688a5197 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:48.508342 843 raft_consensus.cc:2454] T 84bfa8f64219448d8d49a30d50068c22 P d125d3941ab94bd5b970110e5791e171 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cae31ce250aa4d64953e14da451c568c in term 1.
I20240503 01:58:48.508353 713 raft_consensus.cc:2454] T 84bfa8f64219448d8d49a30d50068c22 P 82c39407898d4033af3c6a1a688a5197 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cae31ce250aa4d64953e14da451c568c in term 1.
I20240503 01:58:48.508574 517 leader_election.cc:302] T 84bfa8f64219448d8d49a30d50068c22 P cae31ce250aa4d64953e14da451c568c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cae31ce250aa4d64953e14da451c568c, d125d3941ab94bd5b970110e5791e171; no voters:
I20240503 01:58:48.508726 914 raft_consensus.cc:2781] T 84bfa8f64219448d8d49a30d50068c22 P cae31ce250aa4d64953e14da451c568c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:58:48.508819 914 raft_consensus.cc:686] T 84bfa8f64219448d8d49a30d50068c22 P cae31ce250aa4d64953e14da451c568c [term 1 LEADER]: Becoming Leader. State: Replica: cae31ce250aa4d64953e14da451c568c, State: Running, Role: LEADER
I20240503 01:58:48.508945 914 consensus_queue.cc:237] T 84bfa8f64219448d8d49a30d50068c22 P cae31ce250aa4d64953e14da451c568c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.510579 438 catalog_manager.cc:5484] T 84bfa8f64219448d8d49a30d50068c22 P cae31ce250aa4d64953e14da451c568c reported cstate change: term changed from 0 to 1, leader changed from <none> to cae31ce250aa4d64953e14da451c568c (127.0.99.129). New cstate: current_term: 1 leader_uuid: "cae31ce250aa4d64953e14da451c568c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } health_report { overall_health: UNKNOWN } } }
I20240503 01:58:48.576627 980 raft_consensus.cc:482] T 1581a7ae381c4228bbb4420b84ba432f P 82c39407898d4033af3c6a1a688a5197 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:58:48.576794 980 raft_consensus.cc:504] T 1581a7ae381c4228bbb4420b84ba432f P 82c39407898d4033af3c6a1a688a5197 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.577278 980 leader_election.cc:288] T 1581a7ae381c4228bbb4420b84ba432f P 82c39407898d4033af3c6a1a688a5197 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cae31ce250aa4d64953e14da451c568c (127.0.99.129:44403), d125d3941ab94bd5b970110e5791e171 (127.0.99.131:44747)
I20240503 01:58:48.577733 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1581a7ae381c4228bbb4420b84ba432f" candidate_uuid: "82c39407898d4033af3c6a1a688a5197" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cae31ce250aa4d64953e14da451c568c" is_pre_election: true
I20240503 01:58:48.577786 843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1581a7ae381c4228bbb4420b84ba432f" candidate_uuid: "82c39407898d4033af3c6a1a688a5197" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d125d3941ab94bd5b970110e5791e171" is_pre_election: true
I20240503 01:58:48.577867 583 raft_consensus.cc:2454] T 1581a7ae381c4228bbb4420b84ba432f P cae31ce250aa4d64953e14da451c568c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82c39407898d4033af3c6a1a688a5197 in term 0.
I20240503 01:58:48.577975 843 raft_consensus.cc:2454] T 1581a7ae381c4228bbb4420b84ba432f P d125d3941ab94bd5b970110e5791e171 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82c39407898d4033af3c6a1a688a5197 in term 0.
I20240503 01:58:48.578148 649 leader_election.cc:302] T 1581a7ae381c4228bbb4420b84ba432f P 82c39407898d4033af3c6a1a688a5197 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82c39407898d4033af3c6a1a688a5197, cae31ce250aa4d64953e14da451c568c; no voters:
I20240503 01:58:48.578315 980 raft_consensus.cc:2781] T 1581a7ae381c4228bbb4420b84ba432f P 82c39407898d4033af3c6a1a688a5197 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:58:48.578372 980 raft_consensus.cc:482] T 1581a7ae381c4228bbb4420b84ba432f P 82c39407898d4033af3c6a1a688a5197 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:58:48.578406 980 raft_consensus.cc:3037] T 1581a7ae381c4228bbb4420b84ba432f P 82c39407898d4033af3c6a1a688a5197 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:48.605275 980 raft_consensus.cc:504] T 1581a7ae381c4228bbb4420b84ba432f P 82c39407898d4033af3c6a1a688a5197 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.605794 980 leader_election.cc:288] T 1581a7ae381c4228bbb4420b84ba432f P 82c39407898d4033af3c6a1a688a5197 [CANDIDATE]: Term 1 election: Requested vote from peers cae31ce250aa4d64953e14da451c568c (127.0.99.129:44403), d125d3941ab94bd5b970110e5791e171 (127.0.99.131:44747)
I20240503 01:58:48.606261 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1581a7ae381c4228bbb4420b84ba432f" candidate_uuid: "82c39407898d4033af3c6a1a688a5197" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cae31ce250aa4d64953e14da451c568c"
I20240503 01:58:48.606287 843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1581a7ae381c4228bbb4420b84ba432f" candidate_uuid: "82c39407898d4033af3c6a1a688a5197" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d125d3941ab94bd5b970110e5791e171"
I20240503 01:58:48.606391 583 raft_consensus.cc:3037] T 1581a7ae381c4228bbb4420b84ba432f P cae31ce250aa4d64953e14da451c568c [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:48.606494 843 raft_consensus.cc:3037] T 1581a7ae381c4228bbb4420b84ba432f P d125d3941ab94bd5b970110e5791e171 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:48.607193 583 raft_consensus.cc:2454] T 1581a7ae381c4228bbb4420b84ba432f P cae31ce250aa4d64953e14da451c568c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82c39407898d4033af3c6a1a688a5197 in term 1.
I20240503 01:58:48.607507 649 leader_election.cc:302] T 1581a7ae381c4228bbb4420b84ba432f P 82c39407898d4033af3c6a1a688a5197 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82c39407898d4033af3c6a1a688a5197, cae31ce250aa4d64953e14da451c568c; no voters:
I20240503 01:58:48.607654 843 raft_consensus.cc:2454] T 1581a7ae381c4228bbb4420b84ba432f P d125d3941ab94bd5b970110e5791e171 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82c39407898d4033af3c6a1a688a5197 in term 1.
I20240503 01:58:48.607687 980 raft_consensus.cc:2781] T 1581a7ae381c4228bbb4420b84ba432f P 82c39407898d4033af3c6a1a688a5197 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:58:48.607847 980 raft_consensus.cc:686] T 1581a7ae381c4228bbb4420b84ba432f P 82c39407898d4033af3c6a1a688a5197 [term 1 LEADER]: Becoming Leader. State: Replica: 82c39407898d4033af3c6a1a688a5197, State: Running, Role: LEADER
I20240503 01:58:48.608008 980 consensus_queue.cc:237] T 1581a7ae381c4228bbb4420b84ba432f P 82c39407898d4033af3c6a1a688a5197 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } } peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } }
I20240503 01:58:48.610349 438 catalog_manager.cc:5484] T 1581a7ae381c4228bbb4420b84ba432f P 82c39407898d4033af3c6a1a688a5197 reported cstate change: term changed from 0 to 1, leader changed from <none> to 82c39407898d4033af3c6a1a688a5197 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "82c39407898d4033af3c6a1a688a5197" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 } health_report { overall_health: UNKNOWN } } }
I20240503 01:58:48.673271 914 consensus_queue.cc:1035] T a7090b6ae5394aecb146a7fa25b89e79 P cae31ce250aa4d64953e14da451c568c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:48.678366 912 consensus_queue.cc:1035] T a7090b6ae5394aecb146a7fa25b89e79 P cae31ce250aa4d64953e14da451c568c [LEADER]: Connected to new peer: Peer: permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:48.687332 935 consensus_queue.cc:1035] T 86f9a4762bf34b52a8d000a487226e5b P d125d3941ab94bd5b970110e5791e171 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:48.691598 962 consensus_queue.cc:1035] T 86f9a4762bf34b52a8d000a487226e5b P d125d3941ab94bd5b970110e5791e171 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:48.698297 962 consensus_queue.cc:1035] T 2f1cd41c7fc8437598f73938ae3f2a1d P d125d3941ab94bd5b970110e5791e171 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:48.701381 929 consensus_queue.cc:1035] T 2f1cd41c7fc8437598f73938ae3f2a1d P d125d3941ab94bd5b970110e5791e171 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:48.735394 912 consensus_queue.cc:1035] T 2e5dc6a7de704dcdb0aec15124300b61 P cae31ce250aa4d64953e14da451c568c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:48.738698 921 consensus_queue.cc:1035] T 2e5dc6a7de704dcdb0aec15124300b61 P cae31ce250aa4d64953e14da451c568c [LEADER]: Connected to new peer: Peer: permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:48.752622 973 consensus_queue.cc:1035] T 8d1ad90f476a4b2b97f59259ef4bff9f P 82c39407898d4033af3c6a1a688a5197 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:48.755695 973 consensus_queue.cc:1035] T 8d1ad90f476a4b2b97f59259ef4bff9f P 82c39407898d4033af3c6a1a688a5197 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:48.780141 973 consensus_queue.cc:1035] T 5a547015b4514b8286adb5c2e6356ba1 P 82c39407898d4033af3c6a1a688a5197 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:48.785719 973 consensus_queue.cc:1035] T 5a547015b4514b8286adb5c2e6356ba1 P 82c39407898d4033af3c6a1a688a5197 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:48.795259 912 consensus_queue.cc:1035] T c54888ed43eb4ab4bc79a9ab0a2bcf51 P cae31ce250aa4d64953e14da451c568c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:48.799504 912 consensus_queue.cc:1035] T c54888ed43eb4ab4bc79a9ab0a2bcf51 P cae31ce250aa4d64953e14da451c568c [LEADER]: Connected to new peer: Peer: permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:48.810459 912 consensus_queue.cc:1035] T 0488ca07075c4daca7b8f4d5f827ea56 P cae31ce250aa4d64953e14da451c568c [LEADER]: Connected to new peer: Peer: permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:48.813635 914 consensus_queue.cc:1035] T 0488ca07075c4daca7b8f4d5f827ea56 P cae31ce250aa4d64953e14da451c568c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 01:58:48.831580 1000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:58:48.831846 1000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:58:48.835079 1000 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 01:58:48.849241 914 consensus_queue.cc:1035] T 42a69d697389414a973d995a763d8219 P cae31ce250aa4d64953e14da451c568c [LEADER]: Connected to new peer: Peer: permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:48.853025 912 consensus_queue.cc:1035] T 42a69d697389414a973d995a763d8219 P cae31ce250aa4d64953e14da451c568c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_0'
I20240503 01:58:48.872686 935 consensus_queue.cc:1035] T e0970cda8db54a8593f7d482e861da46 P d125d3941ab94bd5b970110e5791e171 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:48.874958 973 consensus_queue.cc:1035] T ccaa4613da8546f88fb844a6711baabc P 82c39407898d4033af3c6a1a688a5197 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:48.908910 935 consensus_queue.cc:1035] T e0970cda8db54a8593f7d482e861da46 P d125d3941ab94bd5b970110e5791e171 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:48.914426 928 consensus_queue.cc:1035] T ccaa4613da8546f88fb844a6711baabc P 82c39407898d4033af3c6a1a688a5197 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:48.937546 928 consensus_queue.cc:1035] T e38e1de36d82402c82b72dce9e765505 P 82c39407898d4033af3c6a1a688a5197 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:48.970819 935 consensus_queue.cc:1035] T 9f1fb01218234ec4a13fd6f9dfb3fdd6 P d125d3941ab94bd5b970110e5791e171 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:48.977221 928 consensus_queue.cc:1035] T e38e1de36d82402c82b72dce9e765505 P 82c39407898d4033af3c6a1a688a5197 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:48.985415 994 mvcc.cc:205] Tried to move back new op lower bound from 7023417462638981120 to 7023417460591312896. Current Snapshot: MvccSnapshot[applied={T|T < 7023417462638981120}]
I20240503 01:58:49.008518 912 consensus_queue.cc:1035] T d2a69cc1ebd8418f85677c18ef9d80ef P cae31ce250aa4d64953e14da451c568c [LEADER]: Connected to new peer: Peer: permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:49.010737 935 consensus_queue.cc:1035] T 9f1fb01218234ec4a13fd6f9dfb3fdd6 P d125d3941ab94bd5b970110e5791e171 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:49.032285 921 consensus_queue.cc:1035] T d2a69cc1ebd8418f85677c18ef9d80ef P cae31ce250aa4d64953e14da451c568c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:49.047052 921 consensus_queue.cc:1035] T 84bfa8f64219448d8d49a30d50068c22 P cae31ce250aa4d64953e14da451c568c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:49.047667 928 consensus_queue.cc:1035] T 1581a7ae381c4228bbb4420b84ba432f P 82c39407898d4033af3c6a1a688a5197 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d125d3941ab94bd5b970110e5791e171" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:49.056342 921 consensus_queue.cc:1035] T 84bfa8f64219448d8d49a30d50068c22 P cae31ce250aa4d64953e14da451c568c [LEADER]: Connected to new peer: Peer: permanent_uuid: "82c39407898d4033af3c6a1a688a5197" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 196.136 ms
time per row: 0.098068 ms
I20240503 01:58:49.068202 973 consensus_queue.cc:1035] T 1581a7ae381c4228bbb4420b84ba432f P 82c39407898d4033af3c6a1a688a5197 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cae31ce250aa4d64953e14da451c568c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:49.288954 557 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 01:58:49.288990 689 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 01:58:49.289126 816 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 01:58:49.392596 689 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 01:58:49.393149 816 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 01:58:49.393522 557 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 01:58:49.515625 689 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 01:58:49.516979 557 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 01:58:49.517280 816 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 01:58:49.749413 689 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 01:58:49.749892 557 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 01:58:49.749981 816 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 01:58:49.843313 816 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 01:58:49.844020 689 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 01:58:49.844422 557 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 01:58:49.876722 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task42rE7p/build/debug/bin/kudu with pid 501
I20240503 01:58:49.884104 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task42rE7p/build/debug/bin/kudu with pid 632
I20240503 01:58:49.890556 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task42rE7p/build/debug/bin/kudu with pid 763
I20240503 01:58:49.896791 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task42rE7p/build/debug/bin/kudu with pid 410
2024-05-03T01:58:49Z chronyd exiting
[ OK ] TableRangeRebalancingTest.Basic (6391 ms)
[----------] 1 test from TableRangeRebalancingTest (6391 ms total)
[----------] 1 test from RebalanceStartSafetyTest
[ RUN ] RebalanceStartSafetyTest.TooManyIgnoredTservers/1
I20240503 01:58:49.925943 398 test_util.cc:268] Using random seed: 261507379
I20240503 01:58:49.926663 398 ts_itest-base.cc:112] Starting cluster with:
I20240503 01:58:49.926704 398 ts_itest-base.cc:113] --------------
I20240503 01:58:49.926724 398 ts_itest-base.cc:114] 5 tablet servers
I20240503 01:58:49.926740 398 ts_itest-base.cc:115] 3 replicas per TS
I20240503 01:58:49.926766 398 ts_itest-base.cc:116] --------------
2024-05-03T01:58:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T01:58:49Z Disabled control of system clock
I20240503 01:58:49.934187 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task42rE7p/build/debug/bin/kudu
/tmp/dist-test-task42rE7p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:43117
--webserver_interface=127.0.99.190
--webserver_port=0
--builtin_ntp_servers=127.0.99.148:33437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.99.190:43117
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240503 01:58:50.036499 1241 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:58:50.036764 1241 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:58:50.036813 1241 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:58:50.039749 1241 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:58:50.039808 1241 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 01:58:50.039827 1241 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:58:50.039845 1241 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 01:58:50.039877 1241 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 01:58:50.043404 1241 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:33437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.0.99.190:43117
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:43117
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.99.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.1241
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:58:50.044252 1241 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:58:50.045377 1241 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:58:50.053819 1246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:58:50.054839 1247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:58:50.056058 1249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:58:50.057032 1241 server_base.cc:1034] running on GCE node
I20240503 01:58:50.057543 1241 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:58:50.058437 1241 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:58:50.059599 1241 hybrid_clock.cc:648] HybridClock initialized: now 1714701530059584 us; error 35 us; skew 500 ppm
I20240503 01:58:50.061511 1241 webserver.cc:460] Webserver started at http://127.0.99.190:42781/ using document root <none> and password file <none>
I20240503 01:58:50.062036 1241 fs_manager.cc:359] Metadata directory not provided
I20240503 01:58:50.062099 1241 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:58:50.062268 1241 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:58:50.063859 1241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "dcc80b2f1d764d1eaf3a60f882217b92"
format_stamp: "Formatted at 2024-05-03 01:58:50 on dist-test-slave-dtz3"
I20240503 01:58:50.064322 1241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "dcc80b2f1d764d1eaf3a60f882217b92"
format_stamp: "Formatted at 2024-05-03 01:58:50 on dist-test-slave-dtz3"
I20240503 01:58:50.067484 1241 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 01:58:50.069566 1255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:50.070518 1241 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 01:58:50.070628 1241 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/wal
uuid: "dcc80b2f1d764d1eaf3a60f882217b92"
format_stamp: "Formatted at 2024-05-03 01:58:50 on dist-test-slave-dtz3"
I20240503 01:58:50.070706 1241 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:58:50.093358 1241 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:58:50.093930 1241 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:58:50.094085 1241 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:58:50.100488 1241 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.190:43117
I20240503 01:58:50.100517 1307 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.190:43117 every 8 connection(s)
I20240503 01:58:50.101461 1241 server_base.cc:1166] Dumped server information to /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 01:58:50.103844 1308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:50.108610 1308 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcc80b2f1d764d1eaf3a60f882217b92: Bootstrap starting.
I20240503 01:58:50.108888 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task42rE7p/build/debug/bin/kudu as pid 1241
I20240503 01:58:50.108986 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 01:58:50.110837 1308 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dcc80b2f1d764d1eaf3a60f882217b92: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:50.111455 1308 log.cc:824] T 00000000000000000000000000000000 P dcc80b2f1d764d1eaf3a60f882217b92: Log is configured to *not* fsync() on all Append() calls
I20240503 01:58:50.113124 1308 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcc80b2f1d764d1eaf3a60f882217b92: No bootstrap required, opened a new log
I20240503 01:58:50.116096 1308 raft_consensus.cc:348] T 00000000000000000000000000000000 P dcc80b2f1d764d1eaf3a60f882217b92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcc80b2f1d764d1eaf3a60f882217b92" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 43117 } }
I20240503 01:58:50.116297 1308 raft_consensus.cc:374] T 00000000000000000000000000000000 P dcc80b2f1d764d1eaf3a60f882217b92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:50.116364 1308 raft_consensus.cc:729] T 00000000000000000000000000000000 P dcc80b2f1d764d1eaf3a60f882217b92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcc80b2f1d764d1eaf3a60f882217b92, State: Initialized, Role: FOLLOWER
I20240503 01:58:50.116776 1308 consensus_queue.cc:260] T 00000000000000000000000000000000 P dcc80b2f1d764d1eaf3a60f882217b92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcc80b2f1d764d1eaf3a60f882217b92" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 43117 } }
I20240503 01:58:50.116925 1308 raft_consensus.cc:388] T 00000000000000000000000000000000 P dcc80b2f1d764d1eaf3a60f882217b92 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 01:58:50.116986 1308 raft_consensus.cc:482] T 00000000000000000000000000000000 P dcc80b2f1d764d1eaf3a60f882217b92 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 01:58:50.117086 1308 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dcc80b2f1d764d1eaf3a60f882217b92 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:50.117950 1308 raft_consensus.cc:504] T 00000000000000000000000000000000 P dcc80b2f1d764d1eaf3a60f882217b92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcc80b2f1d764d1eaf3a60f882217b92" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 43117 } }
I20240503 01:58:50.118218 1308 leader_election.cc:302] T 00000000000000000000000000000000 P dcc80b2f1d764d1eaf3a60f882217b92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dcc80b2f1d764d1eaf3a60f882217b92; no voters:
I20240503 01:58:50.118429 1308 leader_election.cc:288] T 00000000000000000000000000000000 P dcc80b2f1d764d1eaf3a60f882217b92 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 01:58:50.118563 1313 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dcc80b2f1d764d1eaf3a60f882217b92 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:58:50.118785 1313 raft_consensus.cc:686] T 00000000000000000000000000000000 P dcc80b2f1d764d1eaf3a60f882217b92 [term 1 LEADER]: Becoming Leader. State: Replica: dcc80b2f1d764d1eaf3a60f882217b92, State: Running, Role: LEADER
I20240503 01:58:50.119048 1313 consensus_queue.cc:237] T 00000000000000000000000000000000 P dcc80b2f1d764d1eaf3a60f882217b92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcc80b2f1d764d1eaf3a60f882217b92" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 43117 } }
I20240503 01:58:50.119393 1308 sys_catalog.cc:564] T 00000000000000000000000000000000 P dcc80b2f1d764d1eaf3a60f882217b92 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 01:58:50.120409 1314 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcc80b2f1d764d1eaf3a60f882217b92 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dcc80b2f1d764d1eaf3a60f882217b92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcc80b2f1d764d1eaf3a60f882217b92" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 43117 } } }
I20240503 01:58:50.120441 1315 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcc80b2f1d764d1eaf3a60f882217b92 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dcc80b2f1d764d1eaf3a60f882217b92. Latest consensus state: current_term: 1 leader_uuid: "dcc80b2f1d764d1eaf3a60f882217b92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcc80b2f1d764d1eaf3a60f882217b92" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 43117 } } }
I20240503 01:58:50.120661 1315 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcc80b2f1d764d1eaf3a60f882217b92 [sys.catalog]: This master's current role is: LEADER
I20240503 01:58:50.120937 1314 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcc80b2f1d764d1eaf3a60f882217b92 [sys.catalog]: This master's current role is: LEADER
W20240503 01:58:50.123842 1329 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dcc80b2f1d764d1eaf3a60f882217b92: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240503 01:58:50.123907 1329 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240503 01:58:50.123983 1321 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 01:58:50.124615 1321 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 01:58:50.128970 1321 catalog_manager.cc:1347] Generated new cluster ID: 79e6756120894537a8b1a4d653ed02e7
I20240503 01:58:50.129029 1321 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 01:58:50.160925 1321 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 01:58:50.161860 1321 catalog_manager.cc:1504] Loading token signing keys...
I20240503 01:58:50.168296 1321 catalog_manager.cc:5857] T 00000000000000000000000000000000 P dcc80b2f1d764d1eaf3a60f882217b92: Generated new TSK 0
I20240503 01:58:50.168968 1321 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 01:58:50.175290 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task42rE7p/build/debug/bin/kudu
/tmp/dist-test-task42rE7p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.129:0
--local_ip_for_outbound_sockets=127.0.99.129
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43117
--builtin_ntp_servers=127.0.99.148:33437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 01:58:50.277177 1332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:58:50.277382 1332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:58:50.277426 1332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:58:50.280412 1332 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:58:50.280469 1332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:58:50.280558 1332 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.129
I20240503 01:58:50.283991 1332 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:33437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43117
--never_fsync=true
--heap_profile_path=/tmp/kudu.1332
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.129
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:58:50.284827 1332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:58:50.285916 1332 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:58:50.294446 1340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:58:50.295598 1337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:58:50.296813 1338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:58:50.297700 1332 server_base.cc:1034] running on GCE node
I20240503 01:58:50.298030 1332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:58:50.298537 1332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:58:50.299692 1332 hybrid_clock.cc:648] HybridClock initialized: now 1714701530299677 us; error 38 us; skew 500 ppm
I20240503 01:58:50.301496 1332 webserver.cc:460] Webserver started at http://127.0.99.129:37455/ using document root <none> and password file <none>
I20240503 01:58:50.301992 1332 fs_manager.cc:359] Metadata directory not provided
I20240503 01:58:50.302043 1332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:58:50.302183 1332 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:58:50.303700 1332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "b0f2431cd6474ba4847d2928fe7b7bec"
format_stamp: "Formatted at 2024-05-03 01:58:50 on dist-test-slave-dtz3"
I20240503 01:58:50.304123 1332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "b0f2431cd6474ba4847d2928fe7b7bec"
format_stamp: "Formatted at 2024-05-03 01:58:50 on dist-test-slave-dtz3"
I20240503 01:58:50.307194 1332 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 01:58:50.309175 1346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:50.310122 1332 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 01:58:50.310223 1332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "b0f2431cd6474ba4847d2928fe7b7bec"
format_stamp: "Formatted at 2024-05-03 01:58:50 on dist-test-slave-dtz3"
I20240503 01:58:50.310293 1332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:58:50.332139 1332 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:58:50.332722 1332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:58:50.332856 1332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:58:50.333300 1332 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:58:50.334121 1332 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:58:50.334167 1332 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:50.334204 1332 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:58:50.334224 1332 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:50.342867 1332 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.129:45239
I20240503 01:58:50.342911 1459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.129:45239 every 8 connection(s)
I20240503 01:58:50.343739 1332 server_base.cc:1166] Dumped server information to /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 01:58:50.350800 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task42rE7p/build/debug/bin/kudu as pid 1332
I20240503 01:58:50.350929 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 01:58:50.351832 1460 heartbeater.cc:347] Connected to a master server at 127.0.99.190:43117
I20240503 01:58:50.352023 1460 heartbeater.cc:464] Registering TS with master...
I20240503 01:58:50.352599 1460 heartbeater.cc:511] Master 127.0.99.190:43117 requested a full tablet report, sending...
I20240503 01:58:50.353024 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task42rE7p/build/debug/bin/kudu
/tmp/dist-test-task42rE7p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.130:0
--local_ip_for_outbound_sockets=127.0.99.130
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43117
--builtin_ntp_servers=127.0.99.148:33437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 01:58:50.354048 1272 ts_manager.cc:194] Registered new tserver with Master: b0f2431cd6474ba4847d2928fe7b7bec (127.0.99.129:45239)
I20240503 01:58:50.355406 1272 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.129:55191
W20240503 01:58:50.461364 1463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:58:50.461582 1463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:58:50.461648 1463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:58:50.464622 1463 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:58:50.464735 1463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:58:50.464833 1463 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.130
I20240503 01:58:50.468349 1463 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:33437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43117
--never_fsync=true
--heap_profile_path=/tmp/kudu.1463
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.130
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:58:50.469223 1463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:58:50.470310 1463 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:58:50.479437 1468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:58:50.480427 1469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:58:50.481429 1471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:58:50.482543 1463 server_base.cc:1034] running on GCE node
I20240503 01:58:50.482918 1463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:58:50.483446 1463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:58:50.484584 1463 hybrid_clock.cc:648] HybridClock initialized: now 1714701530484568 us; error 39 us; skew 500 ppm
I20240503 01:58:50.486469 1463 webserver.cc:460] Webserver started at http://127.0.99.130:43497/ using document root <none> and password file <none>
I20240503 01:58:50.487004 1463 fs_manager.cc:359] Metadata directory not provided
I20240503 01:58:50.487066 1463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:58:50.487236 1463 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:58:50.488894 1463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "e5f177cddfc247719e2df5714fde27fe"
format_stamp: "Formatted at 2024-05-03 01:58:50 on dist-test-slave-dtz3"
I20240503 01:58:50.489351 1463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "e5f177cddfc247719e2df5714fde27fe"
format_stamp: "Formatted at 2024-05-03 01:58:50 on dist-test-slave-dtz3"
I20240503 01:58:50.492486 1463 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240503 01:58:50.494633 1477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:50.495574 1463 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 01:58:50.495699 1463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "e5f177cddfc247719e2df5714fde27fe"
format_stamp: "Formatted at 2024-05-03 01:58:50 on dist-test-slave-dtz3"
I20240503 01:58:50.495779 1463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:58:50.527611 1463 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:58:50.528246 1463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:58:50.528412 1463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:58:50.529013 1463 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:58:50.529913 1463 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:58:50.529968 1463 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:50.530025 1463 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:58:50.530058 1463 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:50.538686 1463 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.130:45739
I20240503 01:58:50.538728 1590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.130:45739 every 8 connection(s)
I20240503 01:58:50.539566 1463 server_base.cc:1166] Dumped server information to /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 01:58:50.548075 1591 heartbeater.cc:347] Connected to a master server at 127.0.99.190:43117
I20240503 01:58:50.548275 1591 heartbeater.cc:464] Registering TS with master...
I20240503 01:58:50.548853 1591 heartbeater.cc:511] Master 127.0.99.190:43117 requested a full tablet report, sending...
I20240503 01:58:50.549391 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task42rE7p/build/debug/bin/kudu as pid 1463
I20240503 01:58:50.549510 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 01:58:50.549901 1272 ts_manager.cc:194] Registered new tserver with Master: e5f177cddfc247719e2df5714fde27fe (127.0.99.130:45739)
I20240503 01:58:50.550458 1272 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.130:37577
I20240503 01:58:50.551610 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task42rE7p/build/debug/bin/kudu
/tmp/dist-test-task42rE7p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.131:0
--local_ip_for_outbound_sockets=127.0.99.131
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43117
--builtin_ntp_servers=127.0.99.148:33437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 01:58:50.655905 1594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:58:50.656106 1594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:58:50.656176 1594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:58:50.659261 1594 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:58:50.659340 1594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:58:50.659447 1594 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.131
I20240503 01:58:50.663616 1594 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:33437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43117
--never_fsync=true
--heap_profile_path=/tmp/kudu.1594
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.131
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:58:50.664453 1594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:58:50.665542 1594 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:58:50.674505 1600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:58:50.675668 1599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:58:50.676494 1602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:58:50.677523 1594 server_base.cc:1034] running on GCE node
I20240503 01:58:50.677889 1594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:58:50.678408 1594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:58:50.679554 1594 hybrid_clock.cc:648] HybridClock initialized: now 1714701530679534 us; error 44 us; skew 500 ppm
I20240503 01:58:50.681385 1594 webserver.cc:460] Webserver started at http://127.0.99.131:46081/ using document root <none> and password file <none>
I20240503 01:58:50.681910 1594 fs_manager.cc:359] Metadata directory not provided
I20240503 01:58:50.681972 1594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:58:50.682137 1594 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:58:50.683687 1594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "5f65278376b54501804d47bdbe7ab295"
format_stamp: "Formatted at 2024-05-03 01:58:50 on dist-test-slave-dtz3"
I20240503 01:58:50.684134 1594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "5f65278376b54501804d47bdbe7ab295"
format_stamp: "Formatted at 2024-05-03 01:58:50 on dist-test-slave-dtz3"
I20240503 01:58:50.687328 1594 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 01:58:50.689397 1608 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:50.690339 1594 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 01:58:50.690449 1594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "5f65278376b54501804d47bdbe7ab295"
format_stamp: "Formatted at 2024-05-03 01:58:50 on dist-test-slave-dtz3"
I20240503 01:58:50.690531 1594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:58:50.726667 1594 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:58:50.727262 1594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:58:50.727427 1594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:58:50.727941 1594 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:58:50.728814 1594 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:58:50.728864 1594 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:50.728921 1594 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:58:50.728955 1594 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:50.736858 1594 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.131:45469
I20240503 01:58:50.736905 1721 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.131:45469 every 8 connection(s)
I20240503 01:58:50.737709 1594 server_base.cc:1166] Dumped server information to /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 01:58:50.745682 1722 heartbeater.cc:347] Connected to a master server at 127.0.99.190:43117
I20240503 01:58:50.745857 1722 heartbeater.cc:464] Registering TS with master...
I20240503 01:58:50.746305 1722 heartbeater.cc:511] Master 127.0.99.190:43117 requested a full tablet report, sending...
I20240503 01:58:50.747175 1271 ts_manager.cc:194] Registered new tserver with Master: 5f65278376b54501804d47bdbe7ab295 (127.0.99.131:45469)
I20240503 01:58:50.747236 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task42rE7p/build/debug/bin/kudu as pid 1594
I20240503 01:58:50.747341 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 01:58:50.747725 1271 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.131:35025
I20240503 01:58:50.749136 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task42rE7p/build/debug/bin/kudu
/tmp/dist-test-task42rE7p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.132:0
--local_ip_for_outbound_sockets=127.0.99.132
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43117
--builtin_ntp_servers=127.0.99.148:33437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 01:58:50.850308 1725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:58:50.850525 1725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:58:50.850569 1725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:58:50.853546 1725 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:58:50.853601 1725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:58:50.853706 1725 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.132
I20240503 01:58:50.857201 1725 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:33437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43117
--never_fsync=true
--heap_profile_path=/tmp/kudu.1725
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.132
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:58:50.858016 1725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:58:50.859046 1725 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:58:50.867482 1733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:58:50.868321 1730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:58:50.869452 1731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:58:50.870462 1725 server_base.cc:1034] running on GCE node
I20240503 01:58:50.870808 1725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:58:50.871307 1725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:58:50.872462 1725 hybrid_clock.cc:648] HybridClock initialized: now 1714701530872438 us; error 36 us; skew 500 ppm
I20240503 01:58:50.874467 1725 webserver.cc:460] Webserver started at http://127.0.99.132:35927/ using document root <none> and password file <none>
I20240503 01:58:50.874995 1725 fs_manager.cc:359] Metadata directory not provided
I20240503 01:58:50.875051 1725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:58:50.875227 1725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:58:50.876924 1725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "dded5f72041648618722dd2132c1c4d4"
format_stamp: "Formatted at 2024-05-03 01:58:50 on dist-test-slave-dtz3"
I20240503 01:58:50.877391 1725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "dded5f72041648618722dd2132c1c4d4"
format_stamp: "Formatted at 2024-05-03 01:58:50 on dist-test-slave-dtz3"
I20240503 01:58:50.880434 1725 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240503 01:58:50.882445 1739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:50.883371 1725 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 01:58:50.883460 1725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "dded5f72041648618722dd2132c1c4d4"
format_stamp: "Formatted at 2024-05-03 01:58:50 on dist-test-slave-dtz3"
I20240503 01:58:50.883541 1725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:58:50.905627 1725 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:58:50.906169 1725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:58:50.906312 1725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:58:50.906814 1725 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:58:50.907634 1725 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:58:50.907680 1725 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:50.907721 1725 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:58:50.907752 1725 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:50.916517 1725 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.132:43873
I20240503 01:58:50.916539 1852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.132:43873 every 8 connection(s)
I20240503 01:58:50.917407 1725 server_base.cc:1166] Dumped server information to /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 01:58:50.924348 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task42rE7p/build/debug/bin/kudu as pid 1725
I20240503 01:58:50.924480 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 01:58:50.926328 1853 heartbeater.cc:347] Connected to a master server at 127.0.99.190:43117
I20240503 01:58:50.926548 1853 heartbeater.cc:464] Registering TS with master...
I20240503 01:58:50.926635 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task42rE7p/build/debug/bin/kudu
/tmp/dist-test-task42rE7p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:0
--local_ip_for_outbound_sockets=127.0.99.133
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43117
--builtin_ntp_servers=127.0.99.148:33437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 01:58:50.927098 1853 heartbeater.cc:511] Master 127.0.99.190:43117 requested a full tablet report, sending...
I20240503 01:58:50.928030 1272 ts_manager.cc:194] Registered new tserver with Master: dded5f72041648618722dd2132c1c4d4 (127.0.99.132:43873)
I20240503 01:58:50.928754 1272 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.132:52247
W20240503 01:58:51.029590 1856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:58:51.029780 1856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:58:51.029824 1856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:58:51.032742 1856 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:58:51.032796 1856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:58:51.032856 1856 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240503 01:58:51.036275 1856 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:33437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43117
--never_fsync=true
--heap_profile_path=/tmp/kudu.1856
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:58:51.037067 1856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:58:51.038080 1856 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:58:51.046896 1861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:58:51.047993 1862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:58:51.048851 1864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:58:51.049841 1856 server_base.cc:1034] running on GCE node
I20240503 01:58:51.050173 1856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:58:51.050711 1856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:58:51.051863 1856 hybrid_clock.cc:648] HybridClock initialized: now 1714701531051834 us; error 41 us; skew 500 ppm
I20240503 01:58:51.053681 1856 webserver.cc:460] Webserver started at http://127.0.99.133:35469/ using document root <none> and password file <none>
I20240503 01:58:51.054165 1856 fs_manager.cc:359] Metadata directory not provided
I20240503 01:58:51.054215 1856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:58:51.054395 1856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:58:51.055933 1856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "49b0ad6eed1a42fc805b4196a8355147"
format_stamp: "Formatted at 2024-05-03 01:58:51 on dist-test-slave-dtz3"
I20240503 01:58:51.056375 1856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "49b0ad6eed1a42fc805b4196a8355147"
format_stamp: "Formatted at 2024-05-03 01:58:51 on dist-test-slave-dtz3"
I20240503 01:58:51.059513 1856 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 01:58:51.061529 1870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:51.062566 1856 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 01:58:51.062669 1856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "49b0ad6eed1a42fc805b4196a8355147"
format_stamp: "Formatted at 2024-05-03 01:58:51 on dist-test-slave-dtz3"
I20240503 01:58:51.062765 1856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:58:51.083738 1856 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:58:51.084264 1856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:58:51.084414 1856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:58:51.084982 1856 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:58:51.085784 1856 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:58:51.085829 1856 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:51.085873 1856 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:58:51.085903 1856 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:51.095049 1856 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:35813
I20240503 01:58:51.095091 1983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:35813 every 8 connection(s)
I20240503 01:58:51.095901 1856 server_base.cc:1166] Dumped server information to /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 01:58:51.102458 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task42rE7p/build/debug/bin/kudu as pid 1856
I20240503 01:58:51.102574 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 01:58:51.104223 1984 heartbeater.cc:347] Connected to a master server at 127.0.99.190:43117
I20240503 01:58:51.104418 1984 heartbeater.cc:464] Registering TS with master...
I20240503 01:58:51.104918 1984 heartbeater.cc:511] Master 127.0.99.190:43117 requested a full tablet report, sending...
I20240503 01:58:51.105813 1272 ts_manager.cc:194] Registered new tserver with Master: 49b0ad6eed1a42fc805b4196a8355147 (127.0.99.133:35813)
I20240503 01:58:51.106453 1272 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:45547
I20240503 01:58:51.116083 398 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240503 01:58:51.123768 1272 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52892:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240503 01:58:51.139729 1656 tablet_service.cc:1467] Processing CreateTablet for tablet 2f80f5aed82a4659bed63a131dba3ca3 (DEFAULT_TABLE table=TestTable [id=3b1a99ebf3574f54b484fd519fcc1526]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:51.139710 1918 tablet_service.cc:1467] Processing CreateTablet for tablet 2f80f5aed82a4659bed63a131dba3ca3 (DEFAULT_TABLE table=TestTable [id=3b1a99ebf3574f54b484fd519fcc1526]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:51.140156 1525 tablet_service.cc:1467] Processing CreateTablet for tablet 2f80f5aed82a4659bed63a131dba3ca3 (DEFAULT_TABLE table=TestTable [id=3b1a99ebf3574f54b484fd519fcc1526]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:51.140574 1918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f80f5aed82a4659bed63a131dba3ca3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:51.140594 1656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f80f5aed82a4659bed63a131dba3ca3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:51.140882 1525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f80f5aed82a4659bed63a131dba3ca3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:51.145356 2001 tablet_bootstrap.cc:492] T 2f80f5aed82a4659bed63a131dba3ca3 P 49b0ad6eed1a42fc805b4196a8355147: Bootstrap starting.
I20240503 01:58:51.146441 2002 tablet_bootstrap.cc:492] T 2f80f5aed82a4659bed63a131dba3ca3 P e5f177cddfc247719e2df5714fde27fe: Bootstrap starting.
I20240503 01:58:51.147779 2001 tablet_bootstrap.cc:654] T 2f80f5aed82a4659bed63a131dba3ca3 P 49b0ad6eed1a42fc805b4196a8355147: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:51.148094 2003 tablet_bootstrap.cc:492] T 2f80f5aed82a4659bed63a131dba3ca3 P 5f65278376b54501804d47bdbe7ab295: Bootstrap starting.
I20240503 01:58:51.148244 2002 tablet_bootstrap.cc:654] T 2f80f5aed82a4659bed63a131dba3ca3 P e5f177cddfc247719e2df5714fde27fe: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:51.148614 2001 log.cc:824] T 2f80f5aed82a4659bed63a131dba3ca3 P 49b0ad6eed1a42fc805b4196a8355147: Log is configured to *not* fsync() on all Append() calls
I20240503 01:58:51.148991 2002 log.cc:824] T 2f80f5aed82a4659bed63a131dba3ca3 P e5f177cddfc247719e2df5714fde27fe: Log is configured to *not* fsync() on all Append() calls
I20240503 01:58:51.150039 2003 tablet_bootstrap.cc:654] T 2f80f5aed82a4659bed63a131dba3ca3 P 5f65278376b54501804d47bdbe7ab295: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:51.150157 2001 tablet_bootstrap.cc:492] T 2f80f5aed82a4659bed63a131dba3ca3 P 49b0ad6eed1a42fc805b4196a8355147: No bootstrap required, opened a new log
I20240503 01:58:51.150312 2001 ts_tablet_manager.cc:1367] T 2f80f5aed82a4659bed63a131dba3ca3 P 49b0ad6eed1a42fc805b4196a8355147: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 01:58:51.150467 2002 tablet_bootstrap.cc:492] T 2f80f5aed82a4659bed63a131dba3ca3 P e5f177cddfc247719e2df5714fde27fe: No bootstrap required, opened a new log
I20240503 01:58:51.150625 2002 ts_tablet_manager.cc:1367] T 2f80f5aed82a4659bed63a131dba3ca3 P e5f177cddfc247719e2df5714fde27fe: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 01:58:51.150707 2003 log.cc:824] T 2f80f5aed82a4659bed63a131dba3ca3 P 5f65278376b54501804d47bdbe7ab295: Log is configured to *not* fsync() on all Append() calls
I20240503 01:58:51.152191 2003 tablet_bootstrap.cc:492] T 2f80f5aed82a4659bed63a131dba3ca3 P 5f65278376b54501804d47bdbe7ab295: No bootstrap required, opened a new log
I20240503 01:58:51.152355 2003 ts_tablet_manager.cc:1367] T 2f80f5aed82a4659bed63a131dba3ca3 P 5f65278376b54501804d47bdbe7ab295: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 01:58:51.153350 2001 raft_consensus.cc:348] T 2f80f5aed82a4659bed63a131dba3ca3 P 49b0ad6eed1a42fc805b4196a8355147 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f65278376b54501804d47bdbe7ab295" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45469 } } peers { permanent_uuid: "e5f177cddfc247719e2df5714fde27fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45739 } } peers { permanent_uuid: "49b0ad6eed1a42fc805b4196a8355147" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35813 } }
I20240503 01:58:51.153522 2001 raft_consensus.cc:374] T 2f80f5aed82a4659bed63a131dba3ca3 P 49b0ad6eed1a42fc805b4196a8355147 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:51.153576 2001 raft_consensus.cc:729] T 2f80f5aed82a4659bed63a131dba3ca3 P 49b0ad6eed1a42fc805b4196a8355147 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49b0ad6eed1a42fc805b4196a8355147, State: Initialized, Role: FOLLOWER
I20240503 01:58:51.153956 2001 consensus_queue.cc:260] T 2f80f5aed82a4659bed63a131dba3ca3 P 49b0ad6eed1a42fc805b4196a8355147 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f65278376b54501804d47bdbe7ab295" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45469 } } peers { permanent_uuid: "e5f177cddfc247719e2df5714fde27fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45739 } } peers { permanent_uuid: "49b0ad6eed1a42fc805b4196a8355147" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35813 } }
I20240503 01:58:51.154203 2002 raft_consensus.cc:348] T 2f80f5aed82a4659bed63a131dba3ca3 P e5f177cddfc247719e2df5714fde27fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f65278376b54501804d47bdbe7ab295" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45469 } } peers { permanent_uuid: "e5f177cddfc247719e2df5714fde27fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45739 } } peers { permanent_uuid: "49b0ad6eed1a42fc805b4196a8355147" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35813 } }
I20240503 01:58:51.154470 2002 raft_consensus.cc:374] T 2f80f5aed82a4659bed63a131dba3ca3 P e5f177cddfc247719e2df5714fde27fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:51.154547 2002 raft_consensus.cc:729] T 2f80f5aed82a4659bed63a131dba3ca3 P e5f177cddfc247719e2df5714fde27fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5f177cddfc247719e2df5714fde27fe, State: Initialized, Role: FOLLOWER
I20240503 01:58:51.154821 2001 ts_tablet_manager.cc:1398] T 2f80f5aed82a4659bed63a131dba3ca3 P 49b0ad6eed1a42fc805b4196a8355147: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20240503 01:58:51.154963 1984 heartbeater.cc:503] Master 127.0.99.190:43117 was elected leader, sending a full tablet report...
I20240503 01:58:51.154973 2002 consensus_queue.cc:260] T 2f80f5aed82a4659bed63a131dba3ca3 P e5f177cddfc247719e2df5714fde27fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f65278376b54501804d47bdbe7ab295" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45469 } } peers { permanent_uuid: "e5f177cddfc247719e2df5714fde27fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45739 } } peers { permanent_uuid: "49b0ad6eed1a42fc805b4196a8355147" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35813 } }
I20240503 01:58:51.155577 1591 heartbeater.cc:503] Master 127.0.99.190:43117 was elected leader, sending a full tablet report...
I20240503 01:58:51.155735 2002 ts_tablet_manager.cc:1398] T 2f80f5aed82a4659bed63a131dba3ca3 P e5f177cddfc247719e2df5714fde27fe: Time spent starting tablet: real 0.005s user 0.001s sys 0.004s
I20240503 01:58:51.156046 2003 raft_consensus.cc:348] T 2f80f5aed82a4659bed63a131dba3ca3 P 5f65278376b54501804d47bdbe7ab295 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f65278376b54501804d47bdbe7ab295" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45469 } } peers { permanent_uuid: "e5f177cddfc247719e2df5714fde27fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45739 } } peers { permanent_uuid: "49b0ad6eed1a42fc805b4196a8355147" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35813 } }
I20240503 01:58:51.156183 2003 raft_consensus.cc:374] T 2f80f5aed82a4659bed63a131dba3ca3 P 5f65278376b54501804d47bdbe7ab295 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:51.156215 2003 raft_consensus.cc:729] T 2f80f5aed82a4659bed63a131dba3ca3 P 5f65278376b54501804d47bdbe7ab295 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f65278376b54501804d47bdbe7ab295, State: Initialized, Role: FOLLOWER
I20240503 01:58:51.156555 2003 consensus_queue.cc:260] T 2f80f5aed82a4659bed63a131dba3ca3 P 5f65278376b54501804d47bdbe7ab295 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f65278376b54501804d47bdbe7ab295" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45469 } } peers { permanent_uuid: "e5f177cddfc247719e2df5714fde27fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45739 } } peers { permanent_uuid: "49b0ad6eed1a42fc805b4196a8355147" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35813 } }
I20240503 01:58:51.157112 1722 heartbeater.cc:503] Master 127.0.99.190:43117 was elected leader, sending a full tablet report...
I20240503 01:58:51.158622 2003 ts_tablet_manager.cc:1398] T 2f80f5aed82a4659bed63a131dba3ca3 P 5f65278376b54501804d47bdbe7ab295: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20240503 01:58:51.238775 1723 tablet.cc:2369] T 2f80f5aed82a4659bed63a131dba3ca3 P 5f65278376b54501804d47bdbe7ab295: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 01:58:51.290836 1592 tablet.cc:2369] T 2f80f5aed82a4659bed63a131dba3ca3 P e5f177cddfc247719e2df5714fde27fe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 01:58:51.302754 2007 raft_consensus.cc:482] T 2f80f5aed82a4659bed63a131dba3ca3 P 49b0ad6eed1a42fc805b4196a8355147 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:58:51.302950 2007 raft_consensus.cc:504] T 2f80f5aed82a4659bed63a131dba3ca3 P 49b0ad6eed1a42fc805b4196a8355147 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f65278376b54501804d47bdbe7ab295" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45469 } } peers { permanent_uuid: "e5f177cddfc247719e2df5714fde27fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45739 } } peers { permanent_uuid: "49b0ad6eed1a42fc805b4196a8355147" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35813 } }
I20240503 01:58:51.304030 2007 leader_election.cc:288] T 2f80f5aed82a4659bed63a131dba3ca3 P 49b0ad6eed1a42fc805b4196a8355147 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f65278376b54501804d47bdbe7ab295 (127.0.99.131:45469), e5f177cddfc247719e2df5714fde27fe (127.0.99.130:45739)
I20240503 01:58:51.307762 1676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f80f5aed82a4659bed63a131dba3ca3" candidate_uuid: "49b0ad6eed1a42fc805b4196a8355147" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f65278376b54501804d47bdbe7ab295" is_pre_election: true
I20240503 01:58:51.307775 1545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f80f5aed82a4659bed63a131dba3ca3" candidate_uuid: "49b0ad6eed1a42fc805b4196a8355147" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5f177cddfc247719e2df5714fde27fe" is_pre_election: true
I20240503 01:58:51.308048 1676 raft_consensus.cc:2454] T 2f80f5aed82a4659bed63a131dba3ca3 P 5f65278376b54501804d47bdbe7ab295 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49b0ad6eed1a42fc805b4196a8355147 in term 0.
I20240503 01:58:51.308048 1545 raft_consensus.cc:2454] T 2f80f5aed82a4659bed63a131dba3ca3 P e5f177cddfc247719e2df5714fde27fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49b0ad6eed1a42fc805b4196a8355147 in term 0.
I20240503 01:58:51.308471 1874 leader_election.cc:302] T 2f80f5aed82a4659bed63a131dba3ca3 P 49b0ad6eed1a42fc805b4196a8355147 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49b0ad6eed1a42fc805b4196a8355147, e5f177cddfc247719e2df5714fde27fe; no voters:
I20240503 01:58:51.308729 2007 raft_consensus.cc:2781] T 2f80f5aed82a4659bed63a131dba3ca3 P 49b0ad6eed1a42fc805b4196a8355147 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:58:51.308810 2007 raft_consensus.cc:482] T 2f80f5aed82a4659bed63a131dba3ca3 P 49b0ad6eed1a42fc805b4196a8355147 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:58:51.308871 2007 raft_consensus.cc:3037] T 2f80f5aed82a4659bed63a131dba3ca3 P 49b0ad6eed1a42fc805b4196a8355147 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:51.309741 2007 raft_consensus.cc:504] T 2f80f5aed82a4659bed63a131dba3ca3 P 49b0ad6eed1a42fc805b4196a8355147 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f65278376b54501804d47bdbe7ab295" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45469 } } peers { permanent_uuid: "e5f177cddfc247719e2df5714fde27fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45739 } } peers { permanent_uuid: "49b0ad6eed1a42fc805b4196a8355147" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35813 } }
I20240503 01:58:51.310088 2007 leader_election.cc:288] T 2f80f5aed82a4659bed63a131dba3ca3 P 49b0ad6eed1a42fc805b4196a8355147 [CANDIDATE]: Term 1 election: Requested vote from peers 5f65278376b54501804d47bdbe7ab295 (127.0.99.131:45469), e5f177cddfc247719e2df5714fde27fe (127.0.99.130:45739)
I20240503 01:58:51.310395 1545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f80f5aed82a4659bed63a131dba3ca3" candidate_uuid: "49b0ad6eed1a42fc805b4196a8355147" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5f177cddfc247719e2df5714fde27fe"
I20240503 01:58:51.310478 1676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f80f5aed82a4659bed63a131dba3ca3" candidate_uuid: "49b0ad6eed1a42fc805b4196a8355147" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f65278376b54501804d47bdbe7ab295"
I20240503 01:58:51.310503 1545 raft_consensus.cc:3037] T 2f80f5aed82a4659bed63a131dba3ca3 P e5f177cddfc247719e2df5714fde27fe [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:51.310598 1676 raft_consensus.cc:3037] T 2f80f5aed82a4659bed63a131dba3ca3 P 5f65278376b54501804d47bdbe7ab295 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:51.311349 1545 raft_consensus.cc:2454] T 2f80f5aed82a4659bed63a131dba3ca3 P e5f177cddfc247719e2df5714fde27fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49b0ad6eed1a42fc805b4196a8355147 in term 1.
I20240503 01:58:51.311403 1676 raft_consensus.cc:2454] T 2f80f5aed82a4659bed63a131dba3ca3 P 5f65278376b54501804d47bdbe7ab295 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49b0ad6eed1a42fc805b4196a8355147 in term 1.
I20240503 01:58:51.311594 1874 leader_election.cc:302] T 2f80f5aed82a4659bed63a131dba3ca3 P 49b0ad6eed1a42fc805b4196a8355147 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49b0ad6eed1a42fc805b4196a8355147, e5f177cddfc247719e2df5714fde27fe; no voters:
I20240503 01:58:51.311789 2007 raft_consensus.cc:2781] T 2f80f5aed82a4659bed63a131dba3ca3 P 49b0ad6eed1a42fc805b4196a8355147 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:58:51.311985 2007 raft_consensus.cc:686] T 2f80f5aed82a4659bed63a131dba3ca3 P 49b0ad6eed1a42fc805b4196a8355147 [term 1 LEADER]: Becoming Leader. State: Replica: 49b0ad6eed1a42fc805b4196a8355147, State: Running, Role: LEADER
I20240503 01:58:51.312258 2007 consensus_queue.cc:237] T 2f80f5aed82a4659bed63a131dba3ca3 P 49b0ad6eed1a42fc805b4196a8355147 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f65278376b54501804d47bdbe7ab295" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45469 } } peers { permanent_uuid: "e5f177cddfc247719e2df5714fde27fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45739 } } peers { permanent_uuid: "49b0ad6eed1a42fc805b4196a8355147" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35813 } }
I20240503 01:58:51.315341 1271 catalog_manager.cc:5484] T 2f80f5aed82a4659bed63a131dba3ca3 P 49b0ad6eed1a42fc805b4196a8355147 reported cstate change: term changed from 0 to 1, leader changed from <none> to 49b0ad6eed1a42fc805b4196a8355147 (127.0.99.133). New cstate: current_term: 1 leader_uuid: "49b0ad6eed1a42fc805b4196a8355147" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f65278376b54501804d47bdbe7ab295" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e5f177cddfc247719e2df5714fde27fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45739 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49b0ad6eed1a42fc805b4196a8355147" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35813 } health_report { overall_health: HEALTHY } } }
W20240503 01:58:51.346706 1985 tablet.cc:2369] T 2f80f5aed82a4659bed63a131dba3ca3 P 49b0ad6eed1a42fc805b4196a8355147: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 01:58:51.357409 1460 heartbeater.cc:503] Master 127.0.99.190:43117 was elected leader, sending a full tablet report...
I20240503 01:58:51.366255 398 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240503 01:58:51.367938 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver b0f2431cd6474ba4847d2928fe7b7bec to finish bootstrapping
I20240503 01:58:51.373780 398 ts_itest-base.cc:233] Waiting for 1 tablets on tserver e5f177cddfc247719e2df5714fde27fe to finish bootstrapping
I20240503 01:58:51.378253 398 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 5f65278376b54501804d47bdbe7ab295 to finish bootstrapping
I20240503 01:58:51.382494 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver dded5f72041648618722dd2132c1c4d4 to finish bootstrapping
I20240503 01:58:51.387934 398 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 49b0ad6eed1a42fc805b4196a8355147 to finish bootstrapping
I20240503 01:58:51.568356 1918 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:58:51.568461 1394 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:58:51.570142 1656 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:58:51.571249 1787 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:58:51.583878 1525 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:58:51.662652 1787 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:58:51.663715 1918 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:58:51.663904 1394 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:58:51.670437 1525 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:58:51.671367 1656 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:58:51.766822 2007 consensus_queue.cc:1035] T 2f80f5aed82a4659bed63a131dba3ca3 P 49b0ad6eed1a42fc805b4196a8355147 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5f177cddfc247719e2df5714fde27fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:51.772055 1918 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:58:51.772676 1525 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:58:51.775154 2007 consensus_queue.cc:1035] T 2f80f5aed82a4659bed63a131dba3ca3 P 49b0ad6eed1a42fc805b4196a8355147 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f65278376b54501804d47bdbe7ab295" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:51.775228 1787 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:58:51.778492 1394 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:58:51.785542 1656 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:58:51.930536 1853 heartbeater.cc:503] Master 127.0.99.190:43117 was elected leader, sending a full tablet report...
I20240503 01:58:51.994915 1918 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:58:51.995266 1656 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:58:51.997390 1525 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:58:51.997692 1394 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:58:52.012965 1787 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:58:52.086040 1394 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:58:52.087177 1918 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:58:52.087273 1525 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:58:52.094152 1787 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:58:52.094609 1656 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:58:52.209007 1787 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:58:52.209087 1918 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:58:52.211815 1656 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:58:52.213128 1525 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:58:52.214035 1394 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:58:52.251163 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task42rE7p/build/debug/bin/kudu with pid 1332
I20240503 01:58:52.256835 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task42rE7p/build/debug/bin/kudu with pid 1463
I20240503 01:58:52.262727 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task42rE7p/build/debug/bin/kudu with pid 1594
I20240503 01:58:52.268853 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task42rE7p/build/debug/bin/kudu with pid 1725
I20240503 01:58:52.274071 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task42rE7p/build/debug/bin/kudu with pid 1856
I20240503 01:58:52.279681 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task42rE7p/build/debug/bin/kudu with pid 1241
2024-05-03T01:58:52Z chronyd exiting
[ OK ] RebalanceStartSafetyTest.TooManyIgnoredTservers/1 (2376 ms)
[----------] 1 test from RebalanceStartSafetyTest (2376 ms total)
[----------] 1 test from RebalanceParamTest
[ RUN ] RebalanceParamTest.Rebalance/7
I20240503 01:58:52.302500 398 test_util.cc:268] Using random seed: 263883935
I20240503 01:58:52.303264 398 ts_itest-base.cc:112] Starting cluster with:
I20240503 01:58:52.303305 398 ts_itest-base.cc:113] --------------
I20240503 01:58:52.303326 398 ts_itest-base.cc:114] 7 tablet servers
I20240503 01:58:52.303344 398 ts_itest-base.cc:115] 5 replicas per TS
I20240503 01:58:52.303369 398 ts_itest-base.cc:116] --------------
2024-05-03T01:58:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T01:58:52Z Disabled control of system clock
I20240503 01:58:52.310921 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task42rE7p/build/debug/bin/kudu
/tmp/dist-test-task42rE7p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:38489
--webserver_interface=127.0.99.190
--webserver_port=0
--builtin_ntp_servers=127.0.99.148:35375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.99.190:38489
--allow_unsafe_replication_factor
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--tserver_unresponsive_timeout_ms=3000 with env {}
W20240503 01:58:52.415506 2227 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240503 01:58:52.415745 2227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:58:52.415803 2227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:58:52.415885 2227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:58:52.418895 2227 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:58:52.418983 2227 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 01:58:52.419029 2227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:58:52.419067 2227 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 01:58:52.419106 2227 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 01:58:52.422660 2227 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.0.99.190:38489
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:38489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.99.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.2227
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:58:52.423542 2227 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:58:52.424629 2227 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:58:52.432396 2232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:58:52.433138 2233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:58:52.434293 2235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:58:52.435285 2227 server_base.cc:1034] running on GCE node
I20240503 01:58:52.435727 2227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:58:52.436542 2227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:58:52.437709 2227 hybrid_clock.cc:648] HybridClock initialized: now 1714701532437701 us; error 24 us; skew 500 ppm
I20240503 01:58:52.439482 2227 webserver.cc:460] Webserver started at http://127.0.99.190:39129/ using document root <none> and password file <none>
I20240503 01:58:52.439998 2227 fs_manager.cc:359] Metadata directory not provided
I20240503 01:58:52.440057 2227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:58:52.440222 2227 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:58:52.441846 2227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "ab13918717ec4184908d11fa45b8012a"
format_stamp: "Formatted at 2024-05-03 01:58:52 on dist-test-slave-dtz3"
I20240503 01:58:52.442304 2227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "ab13918717ec4184908d11fa45b8012a"
format_stamp: "Formatted at 2024-05-03 01:58:52 on dist-test-slave-dtz3"
I20240503 01:58:52.446071 2227 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240503 01:58:52.449030 2241 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:52.450212 2227 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 01:58:52.450320 2227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/wal
uuid: "ab13918717ec4184908d11fa45b8012a"
format_stamp: "Formatted at 2024-05-03 01:58:52 on dist-test-slave-dtz3"
I20240503 01:58:52.450412 2227 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:58:52.473891 2227 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:58:52.474506 2227 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:58:52.474670 2227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:58:52.481195 2227 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.190:38489
I20240503 01:58:52.481222 2293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.190:38489 every 8 connection(s)
I20240503 01:58:52.482120 2227 server_base.cc:1166] Dumped server information to /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 01:58:52.484552 2294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:52.485752 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task42rE7p/build/debug/bin/kudu as pid 2227
I20240503 01:58:52.485843 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 01:58:52.490335 2294 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab13918717ec4184908d11fa45b8012a: Bootstrap starting.
I20240503 01:58:52.492616 2294 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ab13918717ec4184908d11fa45b8012a: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:52.493487 2294 log.cc:824] T 00000000000000000000000000000000 P ab13918717ec4184908d11fa45b8012a: Log is configured to *not* fsync() on all Append() calls
I20240503 01:58:52.495172 2294 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab13918717ec4184908d11fa45b8012a: No bootstrap required, opened a new log
I20240503 01:58:52.497560 2294 raft_consensus.cc:348] T 00000000000000000000000000000000 P ab13918717ec4184908d11fa45b8012a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab13918717ec4184908d11fa45b8012a" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 38489 } }
I20240503 01:58:52.497732 2294 raft_consensus.cc:374] T 00000000000000000000000000000000 P ab13918717ec4184908d11fa45b8012a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:52.497779 2294 raft_consensus.cc:729] T 00000000000000000000000000000000 P ab13918717ec4184908d11fa45b8012a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab13918717ec4184908d11fa45b8012a, State: Initialized, Role: FOLLOWER
I20240503 01:58:52.498131 2294 consensus_queue.cc:260] T 00000000000000000000000000000000 P ab13918717ec4184908d11fa45b8012a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab13918717ec4184908d11fa45b8012a" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 38489 } }
I20240503 01:58:52.498239 2294 raft_consensus.cc:388] T 00000000000000000000000000000000 P ab13918717ec4184908d11fa45b8012a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 01:58:52.498281 2294 raft_consensus.cc:482] T 00000000000000000000000000000000 P ab13918717ec4184908d11fa45b8012a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 01:58:52.498355 2294 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ab13918717ec4184908d11fa45b8012a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:52.499055 2294 raft_consensus.cc:504] T 00000000000000000000000000000000 P ab13918717ec4184908d11fa45b8012a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab13918717ec4184908d11fa45b8012a" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 38489 } }
I20240503 01:58:52.499327 2294 leader_election.cc:302] T 00000000000000000000000000000000 P ab13918717ec4184908d11fa45b8012a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ab13918717ec4184908d11fa45b8012a; no voters:
I20240503 01:58:52.499558 2294 leader_election.cc:288] T 00000000000000000000000000000000 P ab13918717ec4184908d11fa45b8012a [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 01:58:52.499658 2299 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ab13918717ec4184908d11fa45b8012a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:58:52.500280 2299 raft_consensus.cc:686] T 00000000000000000000000000000000 P ab13918717ec4184908d11fa45b8012a [term 1 LEADER]: Becoming Leader. State: Replica: ab13918717ec4184908d11fa45b8012a, State: Running, Role: LEADER
I20240503 01:58:52.500574 2299 consensus_queue.cc:237] T 00000000000000000000000000000000 P ab13918717ec4184908d11fa45b8012a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab13918717ec4184908d11fa45b8012a" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 38489 } }
I20240503 01:58:52.500659 2294 sys_catalog.cc:564] T 00000000000000000000000000000000 P ab13918717ec4184908d11fa45b8012a [sys.catalog]: configured and running, proceeding with master startup.
I20240503 01:58:52.502688 2300 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab13918717ec4184908d11fa45b8012a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ab13918717ec4184908d11fa45b8012a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab13918717ec4184908d11fa45b8012a" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 38489 } } }
I20240503 01:58:52.502812 2300 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab13918717ec4184908d11fa45b8012a [sys.catalog]: This master's current role is: LEADER
I20240503 01:58:52.502696 2301 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab13918717ec4184908d11fa45b8012a [sys.catalog]: SysCatalogTable state changed. Reason: New leader ab13918717ec4184908d11fa45b8012a. Latest consensus state: current_term: 1 leader_uuid: "ab13918717ec4184908d11fa45b8012a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab13918717ec4184908d11fa45b8012a" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 38489 } } }
I20240503 01:58:52.502866 2301 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab13918717ec4184908d11fa45b8012a [sys.catalog]: This master's current role is: LEADER
I20240503 01:58:52.503381 2309 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 01:58:52.505802 2309 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 01:58:52.510535 2309 catalog_manager.cc:1347] Generated new cluster ID: 96ab4a85c1ce482ca4ca9d99f28487a0
I20240503 01:58:52.510607 2309 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 01:58:52.545933 2309 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 01:58:52.546861 2309 catalog_manager.cc:1504] Loading token signing keys...
I20240503 01:58:52.556558 2309 catalog_manager.cc:5857] T 00000000000000000000000000000000 P ab13918717ec4184908d11fa45b8012a: Generated new TSK 0
I20240503 01:58:52.557327 2309 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 01:58:52.568060 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task42rE7p/build/debug/bin/kudu
/tmp/dist-test-task42rE7p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.129:0
--local_ip_for_outbound_sockets=127.0.99.129
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:38489
--builtin_ntp_servers=127.0.99.148:35375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 01:58:52.673245 2318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:58:52.673439 2318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:58:52.673481 2318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:58:52.676440 2318 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:58:52.676499 2318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:58:52.676564 2318 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.129
I20240503 01:58:52.680056 2318 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:38489
--never_fsync=true
--heap_profile_path=/tmp/kudu.2318
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.129
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:58:52.680935 2318 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:58:52.682014 2318 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:58:52.690840 2324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:58:52.691594 2323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:58:52.692819 2326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:58:52.693713 2318 server_base.cc:1034] running on GCE node
I20240503 01:58:52.694053 2318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:58:52.694586 2318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:58:52.695751 2318 hybrid_clock.cc:648] HybridClock initialized: now 1714701532695727 us; error 38 us; skew 500 ppm
I20240503 01:58:52.697522 2318 webserver.cc:460] Webserver started at http://127.0.99.129:37059/ using document root <none> and password file <none>
I20240503 01:58:52.698053 2318 fs_manager.cc:359] Metadata directory not provided
I20240503 01:58:52.698117 2318 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:58:52.698289 2318 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:58:52.699855 2318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "b63928177fa54227a5ff9929265ab9dd"
format_stamp: "Formatted at 2024-05-03 01:58:52 on dist-test-slave-dtz3"
I20240503 01:58:52.700310 2318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "b63928177fa54227a5ff9929265ab9dd"
format_stamp: "Formatted at 2024-05-03 01:58:52 on dist-test-slave-dtz3"
I20240503 01:58:52.703835 2318 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 01:58:52.705932 2332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:52.706871 2318 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 01:58:52.706975 2318 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "b63928177fa54227a5ff9929265ab9dd"
format_stamp: "Formatted at 2024-05-03 01:58:52 on dist-test-slave-dtz3"
I20240503 01:58:52.707056 2318 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:58:52.722978 2318 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:58:52.723551 2318 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:58:52.723713 2318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:58:52.724200 2318 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:58:52.725059 2318 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:58:52.725116 2318 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:52.725174 2318 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:58:52.725205 2318 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:52.733611 2318 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.129:36425
I20240503 01:58:52.733635 2445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.129:36425 every 8 connection(s)
I20240503 01:58:52.734488 2318 server_base.cc:1166] Dumped server information to /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 01:58:52.742743 2446 heartbeater.cc:347] Connected to a master server at 127.0.99.190:38489
I20240503 01:58:52.742944 2446 heartbeater.cc:464] Registering TS with master...
I20240503 01:58:52.743312 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task42rE7p/build/debug/bin/kudu as pid 2318
I20240503 01:58:52.743427 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 01:58:52.743558 2446 heartbeater.cc:511] Master 127.0.99.190:38489 requested a full tablet report, sending...
I20240503 01:58:52.745030 2257 ts_manager.cc:194] Registered new tserver with Master: b63928177fa54227a5ff9929265ab9dd (127.0.99.129:36425)
I20240503 01:58:52.745429 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task42rE7p/build/debug/bin/kudu
/tmp/dist-test-task42rE7p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.130:0
--local_ip_for_outbound_sockets=127.0.99.130
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:38489
--builtin_ntp_servers=127.0.99.148:35375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 01:58:52.746259 2257 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.129:50573
W20240503 01:58:52.848860 2449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:58:52.849090 2449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:58:52.849162 2449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:58:52.852187 2449 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:58:52.852258 2449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:58:52.852349 2449 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.130
I20240503 01:58:52.856006 2449 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:38489
--never_fsync=true
--heap_profile_path=/tmp/kudu.2449
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.130
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:58:52.856918 2449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:58:52.858000 2449 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:58:52.866752 2455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:58:52.867561 2457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:58:52.868748 2454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:58:52.869696 2449 server_base.cc:1034] running on GCE node
I20240503 01:58:52.870074 2449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:58:52.870599 2449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:58:52.871747 2449 hybrid_clock.cc:648] HybridClock initialized: now 1714701532871732 us; error 31 us; skew 500 ppm
I20240503 01:58:52.873611 2449 webserver.cc:460] Webserver started at http://127.0.99.130:37955/ using document root <none> and password file <none>
I20240503 01:58:52.874150 2449 fs_manager.cc:359] Metadata directory not provided
I20240503 01:58:52.874207 2449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:58:52.874395 2449 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:58:52.876039 2449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "a1a844ecd71145e4a362f4c42500fbcb"
format_stamp: "Formatted at 2024-05-03 01:58:52 on dist-test-slave-dtz3"
I20240503 01:58:52.876498 2449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "a1a844ecd71145e4a362f4c42500fbcb"
format_stamp: "Formatted at 2024-05-03 01:58:52 on dist-test-slave-dtz3"
I20240503 01:58:52.879724 2449 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20240503 01:58:52.881831 2463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:52.882773 2449 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 01:58:52.882884 2449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "a1a844ecd71145e4a362f4c42500fbcb"
format_stamp: "Formatted at 2024-05-03 01:58:52 on dist-test-slave-dtz3"
I20240503 01:58:52.882963 2449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:58:52.896297 2449 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:58:52.896910 2449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:58:52.897068 2449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:58:52.897572 2449 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:58:52.898411 2449 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:58:52.898461 2449 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:52.898521 2449 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:58:52.898552 2449 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:52.907553 2449 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.130:46033
I20240503 01:58:52.907589 2576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.130:46033 every 8 connection(s)
I20240503 01:58:52.908443 2449 server_base.cc:1166] Dumped server information to /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 01:58:52.910787 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task42rE7p/build/debug/bin/kudu as pid 2449
I20240503 01:58:52.910904 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 01:58:52.913193 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task42rE7p/build/debug/bin/kudu
/tmp/dist-test-task42rE7p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.131:0
--local_ip_for_outbound_sockets=127.0.99.131
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:38489
--builtin_ntp_servers=127.0.99.148:35375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 01:58:52.917102 2577 heartbeater.cc:347] Connected to a master server at 127.0.99.190:38489
I20240503 01:58:52.917302 2577 heartbeater.cc:464] Registering TS with master...
I20240503 01:58:52.917786 2577 heartbeater.cc:511] Master 127.0.99.190:38489 requested a full tablet report, sending...
I20240503 01:58:52.918737 2257 ts_manager.cc:194] Registered new tserver with Master: a1a844ecd71145e4a362f4c42500fbcb (127.0.99.130:46033)
I20240503 01:58:52.919255 2257 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.130:39189
W20240503 01:58:53.015750 2580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:58:53.015946 2580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:58:53.016017 2580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:58:53.019021 2580 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:58:53.019088 2580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:58:53.019179 2580 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.131
I20240503 01:58:53.022681 2580 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:38489
--never_fsync=true
--heap_profile_path=/tmp/kudu.2580
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.131
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:58:53.023490 2580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:58:53.024542 2580 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:58:53.033042 2586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:58:53.034068 2585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:58:53.034987 2588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:58:53.036049 2580 server_base.cc:1034] running on GCE node
I20240503 01:58:53.036406 2580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:58:53.036953 2580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:58:53.038112 2580 hybrid_clock.cc:648] HybridClock initialized: now 1714701533038083 us; error 42 us; skew 500 ppm
I20240503 01:58:53.039886 2580 webserver.cc:460] Webserver started at http://127.0.99.131:39365/ using document root <none> and password file <none>
I20240503 01:58:53.040426 2580 fs_manager.cc:359] Metadata directory not provided
I20240503 01:58:53.040488 2580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:58:53.040736 2580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:58:53.042361 2580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "b40a9a7b30d34a6db007ed8febd8a759"
format_stamp: "Formatted at 2024-05-03 01:58:53 on dist-test-slave-dtz3"
I20240503 01:58:53.043005 2580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "b40a9a7b30d34a6db007ed8febd8a759"
format_stamp: "Formatted at 2024-05-03 01:58:53 on dist-test-slave-dtz3"
I20240503 01:58:53.046320 2580 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 01:58:53.048491 2594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:53.049454 2580 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 01:58:53.049569 2580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "b40a9a7b30d34a6db007ed8febd8a759"
format_stamp: "Formatted at 2024-05-03 01:58:53 on dist-test-slave-dtz3"
I20240503 01:58:53.049662 2580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:58:53.065498 2580 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:58:53.066092 2580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:58:53.066236 2580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:58:53.066742 2580 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:58:53.067575 2580 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:58:53.067625 2580 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:53.067687 2580 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:58:53.067713 2580 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:53.076412 2580 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.131:34079
I20240503 01:58:53.076444 2707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.131:34079 every 8 connection(s)
I20240503 01:58:53.077337 2580 server_base.cc:1166] Dumped server information to /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 01:58:53.078214 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task42rE7p/build/debug/bin/kudu as pid 2580
I20240503 01:58:53.078297 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 01:58:53.080541 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task42rE7p/build/debug/bin/kudu
/tmp/dist-test-task42rE7p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.132:0
--local_ip_for_outbound_sockets=127.0.99.132
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:38489
--builtin_ntp_servers=127.0.99.148:35375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 01:58:53.087103 2708 heartbeater.cc:347] Connected to a master server at 127.0.99.190:38489
I20240503 01:58:53.087308 2708 heartbeater.cc:464] Registering TS with master...
I20240503 01:58:53.087884 2708 heartbeater.cc:511] Master 127.0.99.190:38489 requested a full tablet report, sending...
I20240503 01:58:53.088866 2257 ts_manager.cc:194] Registered new tserver with Master: b40a9a7b30d34a6db007ed8febd8a759 (127.0.99.131:34079)
I20240503 01:58:53.089404 2257 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.131:57025
W20240503 01:58:53.182941 2711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:58:53.183125 2711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:58:53.183166 2711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:58:53.186172 2711 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:58:53.186242 2711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:58:53.186342 2711 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.132
I20240503 01:58:53.189862 2711 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:38489
--never_fsync=true
--heap_profile_path=/tmp/kudu.2711
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.132
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:58:53.190618 2711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:58:53.191650 2711 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:58:53.199750 2716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:58:53.200896 2719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:58:53.201603 2717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:58:53.202744 2711 server_base.cc:1034] running on GCE node
I20240503 01:58:53.203096 2711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:58:53.203630 2711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:58:53.204784 2711 hybrid_clock.cc:648] HybridClock initialized: now 1714701533204748 us; error 49 us; skew 500 ppm
I20240503 01:58:53.206552 2711 webserver.cc:460] Webserver started at http://127.0.99.132:46225/ using document root <none> and password file <none>
I20240503 01:58:53.207041 2711 fs_manager.cc:359] Metadata directory not provided
I20240503 01:58:53.207090 2711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:58:53.207232 2711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:58:53.208750 2711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "52c196c15e3348b985c689652bc49918"
format_stamp: "Formatted at 2024-05-03 01:58:53 on dist-test-slave-dtz3"
I20240503 01:58:53.209170 2711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "52c196c15e3348b985c689652bc49918"
format_stamp: "Formatted at 2024-05-03 01:58:53 on dist-test-slave-dtz3"
I20240503 01:58:53.212214 2711 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240503 01:58:53.214157 2725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:53.215010 2711 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 01:58:53.215094 2711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "52c196c15e3348b985c689652bc49918"
format_stamp: "Formatted at 2024-05-03 01:58:53 on dist-test-slave-dtz3"
I20240503 01:58:53.215157 2711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:58:53.239269 2711 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:58:53.239809 2711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:58:53.239943 2711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:58:53.240366 2711 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:58:53.241219 2711 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:58:53.241263 2711 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:53.241299 2711 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:58:53.241318 2711 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:53.249526 2711 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.132:42887
I20240503 01:58:53.249557 2838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.132:42887 every 8 connection(s)
I20240503 01:58:53.250358 2711 server_base.cc:1166] Dumped server information to /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 01:58:53.256338 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task42rE7p/build/debug/bin/kudu as pid 2711
I20240503 01:58:53.256451 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 01:58:53.258229 2839 heartbeater.cc:347] Connected to a master server at 127.0.99.190:38489
I20240503 01:58:53.258435 2839 heartbeater.cc:464] Registering TS with master...
I20240503 01:58:53.258481 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task42rE7p/build/debug/bin/kudu
/tmp/dist-test-task42rE7p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:0
--local_ip_for_outbound_sockets=127.0.99.133
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:38489
--builtin_ntp_servers=127.0.99.148:35375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 01:58:53.259028 2839 heartbeater.cc:511] Master 127.0.99.190:38489 requested a full tablet report, sending...
I20240503 01:58:53.259961 2257 ts_manager.cc:194] Registered new tserver with Master: 52c196c15e3348b985c689652bc49918 (127.0.99.132:42887)
I20240503 01:58:53.260462 2257 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.132:40113
W20240503 01:58:53.360221 2842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:58:53.360396 2842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:58:53.360435 2842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:58:53.363337 2842 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:58:53.363389 2842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:58:53.363449 2842 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240503 01:58:53.367023 2842 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:38489
--never_fsync=true
--heap_profile_path=/tmp/kudu.2842
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:58:53.367771 2842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:58:53.368795 2842 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:58:53.377136 2847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:58:53.378162 2848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:58:53.379045 2850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:58:53.380064 2842 server_base.cc:1034] running on GCE node
I20240503 01:58:53.380414 2842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:58:53.380928 2842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:58:53.382079 2842 hybrid_clock.cc:648] HybridClock initialized: now 1714701533382049 us; error 42 us; skew 500 ppm
I20240503 01:58:53.383749 2842 webserver.cc:460] Webserver started at http://127.0.99.133:38845/ using document root <none> and password file <none>
I20240503 01:58:53.384271 2842 fs_manager.cc:359] Metadata directory not provided
I20240503 01:58:53.384327 2842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:58:53.384501 2842 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:58:53.386198 2842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "327aa26a8e434a609ebcb382aa665fce"
format_stamp: "Formatted at 2024-05-03 01:58:53 on dist-test-slave-dtz3"
I20240503 01:58:53.386641 2842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "327aa26a8e434a609ebcb382aa665fce"
format_stamp: "Formatted at 2024-05-03 01:58:53 on dist-test-slave-dtz3"
I20240503 01:58:53.389816 2842 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 01:58:53.391849 2856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:53.392814 2842 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 01:58:53.392918 2842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "327aa26a8e434a609ebcb382aa665fce"
format_stamp: "Formatted at 2024-05-03 01:58:53 on dist-test-slave-dtz3"
I20240503 01:58:53.393011 2842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:58:53.411314 2842 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:58:53.411896 2842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:58:53.412051 2842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:58:53.412544 2842 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:58:53.413378 2842 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:58:53.413420 2842 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:53.413455 2842 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:58:53.413470 2842 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:53.421969 2842 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:38853
I20240503 01:58:53.422009 2969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:38853 every 8 connection(s)
I20240503 01:58:53.422801 2842 server_base.cc:1166] Dumped server information to /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 01:58:53.423676 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task42rE7p/build/debug/bin/kudu as pid 2842
I20240503 01:58:53.423779 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 01:58:53.425984 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task42rE7p/build/debug/bin/kudu
/tmp/dist-test-task42rE7p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.134:0
--local_ip_for_outbound_sockets=127.0.99.134
--webserver_interface=127.0.99.134
--webserver_port=0
--tserver_master_addrs=127.0.99.190:38489
--builtin_ntp_servers=127.0.99.148:35375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 01:58:53.432319 2970 heartbeater.cc:347] Connected to a master server at 127.0.99.190:38489
I20240503 01:58:53.432518 2970 heartbeater.cc:464] Registering TS with master...
I20240503 01:58:53.433028 2970 heartbeater.cc:511] Master 127.0.99.190:38489 requested a full tablet report, sending...
I20240503 01:58:53.433971 2257 ts_manager.cc:194] Registered new tserver with Master: 327aa26a8e434a609ebcb382aa665fce (127.0.99.133:38853)
I20240503 01:58:53.434478 2257 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:44447
W20240503 01:58:53.534080 2973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:58:53.534296 2973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:58:53.534368 2973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:58:53.537359 2973 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:58:53.537425 2973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:58:53.537528 2973 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.134
I20240503 01:58:53.541112 2973 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.134:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.0.99.134
--webserver_port=0
--tserver_master_addrs=127.0.99.190:38489
--never_fsync=true
--heap_profile_path=/tmp/kudu.2973
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.134
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:58:53.541975 2973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:58:53.543048 2973 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:58:53.551606 2981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:58:53.552779 2979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:58:53.553659 2978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:58:53.554847 2973 server_base.cc:1034] running on GCE node
I20240503 01:58:53.555225 2973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:58:53.556032 2973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:58:53.557202 2973 hybrid_clock.cc:648] HybridClock initialized: now 1714701533557184 us; error 38 us; skew 500 ppm
I20240503 01:58:53.559012 2973 webserver.cc:460] Webserver started at http://127.0.99.134:34263/ using document root <none> and password file <none>
I20240503 01:58:53.559525 2973 fs_manager.cc:359] Metadata directory not provided
I20240503 01:58:53.559587 2973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:58:53.559763 2973 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:58:53.561378 2973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "6568f69a78c44bb985162d2b7d0a9376"
format_stamp: "Formatted at 2024-05-03 01:58:53 on dist-test-slave-dtz3"
I20240503 01:58:53.561832 2973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "6568f69a78c44bb985162d2b7d0a9376"
format_stamp: "Formatted at 2024-05-03 01:58:53 on dist-test-slave-dtz3"
I20240503 01:58:53.564908 2973 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 01:58:53.566859 2987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:53.567766 2973 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 01:58:53.567857 2973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "6568f69a78c44bb985162d2b7d0a9376"
format_stamp: "Formatted at 2024-05-03 01:58:53 on dist-test-slave-dtz3"
I20240503 01:58:53.567945 2973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:58:53.580377 2973 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:58:53.580983 2973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:58:53.581130 2973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:58:53.581604 2973 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:58:53.582407 2973 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:58:53.582453 2973 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:53.582509 2973 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:58:53.582540 2973 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:53.591337 2973 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.134:43603
I20240503 01:58:53.591392 3100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.134:43603 every 8 connection(s)
I20240503 01:58:53.592187 2973 server_base.cc:1166] Dumped server information to /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 01:58:53.600502 3101 heartbeater.cc:347] Connected to a master server at 127.0.99.190:38489
I20240503 01:58:53.600726 3101 heartbeater.cc:464] Registering TS with master...
I20240503 01:58:53.601228 3101 heartbeater.cc:511] Master 127.0.99.190:38489 requested a full tablet report, sending...
I20240503 01:58:53.601547 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task42rE7p/build/debug/bin/kudu as pid 2973
I20240503 01:58:53.601665 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240503 01:58:53.602269 2257 ts_manager.cc:194] Registered new tserver with Master: 6568f69a78c44bb985162d2b7d0a9376 (127.0.99.134:43603)
I20240503 01:58:53.602968 2257 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.134:53967
I20240503 01:58:53.603744 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task42rE7p/build/debug/bin/kudu
/tmp/dist-test-task42rE7p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.135:0
--local_ip_for_outbound_sockets=127.0.99.135
--webserver_interface=127.0.99.135
--webserver_port=0
--tserver_master_addrs=127.0.99.190:38489
--builtin_ntp_servers=127.0.99.148:35375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 01:58:53.710268 3104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:58:53.710500 3104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:58:53.710552 3104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:58:53.715852 3104 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:58:53.715957 3104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:58:53.716136 3104 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.135
I20240503 01:58:53.722847 3104 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.135:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.0.99.135
--webserver_port=0
--tserver_master_addrs=127.0.99.190:38489
--never_fsync=true
--heap_profile_path=/tmp/kudu.3104
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.135
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:58:53.723932 3104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:58:53.725418 3104 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:58:53.736949 3112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:58:53.738088 3110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:58:53.738819 3109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:58:53.739912 3104 server_base.cc:1034] running on GCE node
I20240503 01:58:53.740397 3104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:58:53.741132 3104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:58:53.742316 3104 hybrid_clock.cc:648] HybridClock initialized: now 1714701533742284 us; error 43 us; skew 500 ppm
I20240503 01:58:53.745174 3104 webserver.cc:460] Webserver started at http://127.0.99.135:32861/ using document root <none> and password file <none>
I20240503 01:58:53.745864 3104 fs_manager.cc:359] Metadata directory not provided
I20240503 01:58:53.745949 3104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:58:53.746173 3104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:58:53.748219 2446 heartbeater.cc:503] Master 127.0.99.190:38489 was elected leader, sending a full tablet report...
I20240503 01:58:53.748497 3104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "4125b1a70e864e129bd54bdc2720d7fa"
format_stamp: "Formatted at 2024-05-03 01:58:53 on dist-test-slave-dtz3"
I20240503 01:58:53.749254 3104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "4125b1a70e864e129bd54bdc2720d7fa"
format_stamp: "Formatted at 2024-05-03 01:58:53 on dist-test-slave-dtz3"
I20240503 01:58:53.753878 3104 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20240503 01:58:53.756753 3118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:53.757938 3104 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20240503 01:58:53.758061 3104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "4125b1a70e864e129bd54bdc2720d7fa"
format_stamp: "Formatted at 2024-05-03 01:58:53 on dist-test-slave-dtz3"
I20240503 01:58:53.758162 3104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:58:53.786001 3104 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:58:53.786988 3104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:58:53.787175 3104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:58:53.787992 3104 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:58:53.789398 3104 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:58:53.789479 3104 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:53.789552 3104 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:58:53.789590 3104 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:53.810362 3104 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.135:39501
I20240503 01:58:53.811411 3104 server_base.cc:1166] Dumped server information to /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 01:58:53.813772 3231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.135:39501 every 8 connection(s)
I20240503 01:58:53.821547 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task42rE7p/build/debug/bin/kudu as pid 3104
I20240503 01:58:53.821677 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240503 01:58:53.822086 3232 heartbeater.cc:347] Connected to a master server at 127.0.99.190:38489
I20240503 01:58:53.822299 3232 heartbeater.cc:464] Registering TS with master...
I20240503 01:58:53.822793 3232 heartbeater.cc:511] Master 127.0.99.190:38489 requested a full tablet report, sending...
I20240503 01:58:53.823698 2258 ts_manager.cc:194] Registered new tserver with Master: 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135:39501)
I20240503 01:58:53.824378 2258 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.135:44869
I20240503 01:58:53.824946 398 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240503 01:58:53.835484 2258 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53804:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 5
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240503 01:58:53.855118 2773 tablet_service.cc:1467] Processing CreateTablet for tablet 240749b0904f4bb2a4c79b7713473411 (DEFAULT_TABLE table=TestTable [id=e423c6476d1b43779e0a839bf55a520d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:53.856114 2773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 240749b0904f4bb2a4c79b7713473411. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:53.856590 2642 tablet_service.cc:1467] Processing CreateTablet for tablet 240749b0904f4bb2a4c79b7713473411 (DEFAULT_TABLE table=TestTable [id=e423c6476d1b43779e0a839bf55a520d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:53.857430 2642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 240749b0904f4bb2a4c79b7713473411. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:53.858304 3035 tablet_service.cc:1467] Processing CreateTablet for tablet 240749b0904f4bb2a4c79b7713473411 (DEFAULT_TABLE table=TestTable [id=e423c6476d1b43779e0a839bf55a520d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:53.859094 3035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 240749b0904f4bb2a4c79b7713473411. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:53.863317 3254 tablet_bootstrap.cc:492] T 240749b0904f4bb2a4c79b7713473411 P b40a9a7b30d34a6db007ed8febd8a759: Bootstrap starting.
I20240503 01:58:53.863425 2380 tablet_service.cc:1467] Processing CreateTablet for tablet 240749b0904f4bb2a4c79b7713473411 (DEFAULT_TABLE table=TestTable [id=e423c6476d1b43779e0a839bf55a520d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:53.864159 2380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 240749b0904f4bb2a4c79b7713473411. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:53.865186 3255 tablet_bootstrap.cc:492] T 240749b0904f4bb2a4c79b7713473411 P 6568f69a78c44bb985162d2b7d0a9376: Bootstrap starting.
I20240503 01:58:53.865494 3254 tablet_bootstrap.cc:654] T 240749b0904f4bb2a4c79b7713473411 P b40a9a7b30d34a6db007ed8febd8a759: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:53.866281 3254 log.cc:824] T 240749b0904f4bb2a4c79b7713473411 P b40a9a7b30d34a6db007ed8febd8a759: Log is configured to *not* fsync() on all Append() calls
I20240503 01:58:53.867254 3255 tablet_bootstrap.cc:654] T 240749b0904f4bb2a4c79b7713473411 P 6568f69a78c44bb985162d2b7d0a9376: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:53.868183 3255 log.cc:824] T 240749b0904f4bb2a4c79b7713473411 P 6568f69a78c44bb985162d2b7d0a9376: Log is configured to *not* fsync() on all Append() calls
I20240503 01:58:53.869612 3258 tablet_bootstrap.cc:492] T 240749b0904f4bb2a4c79b7713473411 P b63928177fa54227a5ff9929265ab9dd: Bootstrap starting.
I20240503 01:58:53.869961 3255 tablet_bootstrap.cc:492] T 240749b0904f4bb2a4c79b7713473411 P 6568f69a78c44bb985162d2b7d0a9376: No bootstrap required, opened a new log
I20240503 01:58:53.870169 3255 ts_tablet_manager.cc:1367] T 240749b0904f4bb2a4c79b7713473411 P 6568f69a78c44bb985162d2b7d0a9376: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20240503 01:58:53.870182 3166 tablet_service.cc:1467] Processing CreateTablet for tablet 240749b0904f4bb2a4c79b7713473411 (DEFAULT_TABLE table=TestTable [id=e423c6476d1b43779e0a839bf55a520d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:53.871026 3166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 240749b0904f4bb2a4c79b7713473411. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:53.871554 3258 tablet_bootstrap.cc:654] T 240749b0904f4bb2a4c79b7713473411 P b63928177fa54227a5ff9929265ab9dd: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:53.872406 3258 log.cc:824] T 240749b0904f4bb2a4c79b7713473411 P b63928177fa54227a5ff9929265ab9dd: Log is configured to *not* fsync() on all Append() calls
I20240503 01:58:53.874297 3258 tablet_bootstrap.cc:492] T 240749b0904f4bb2a4c79b7713473411 P b63928177fa54227a5ff9929265ab9dd: No bootstrap required, opened a new log
I20240503 01:58:53.874482 3258 ts_tablet_manager.cc:1367] T 240749b0904f4bb2a4c79b7713473411 P b63928177fa54227a5ff9929265ab9dd: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20240503 01:58:53.874614 3255 raft_consensus.cc:348] T 240749b0904f4bb2a4c79b7713473411 P 6568f69a78c44bb985162d2b7d0a9376 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } }
I20240503 01:58:53.874804 3255 raft_consensus.cc:374] T 240749b0904f4bb2a4c79b7713473411 P 6568f69a78c44bb985162d2b7d0a9376 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:53.874902 3255 raft_consensus.cc:729] T 240749b0904f4bb2a4c79b7713473411 P 6568f69a78c44bb985162d2b7d0a9376 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6568f69a78c44bb985162d2b7d0a9376, State: Initialized, Role: FOLLOWER
I20240503 01:58:53.875322 3255 consensus_queue.cc:260] T 240749b0904f4bb2a4c79b7713473411 P 6568f69a78c44bb985162d2b7d0a9376 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } }
I20240503 01:58:53.876271 3255 ts_tablet_manager.cc:1398] T 240749b0904f4bb2a4c79b7713473411 P 6568f69a78c44bb985162d2b7d0a9376: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240503 01:58:53.876724 3101 heartbeater.cc:503] Master 127.0.99.190:38489 was elected leader, sending a full tablet report...
I20240503 01:58:53.877040 3259 tablet_bootstrap.cc:492] T 240749b0904f4bb2a4c79b7713473411 P 52c196c15e3348b985c689652bc49918: Bootstrap starting.
I20240503 01:58:53.878391 3262 tablet_bootstrap.cc:492] T 240749b0904f4bb2a4c79b7713473411 P 4125b1a70e864e129bd54bdc2720d7fa: Bootstrap starting.
I20240503 01:58:53.879344 3259 tablet_bootstrap.cc:654] T 240749b0904f4bb2a4c79b7713473411 P 52c196c15e3348b985c689652bc49918: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:53.880172 3262 tablet_bootstrap.cc:654] T 240749b0904f4bb2a4c79b7713473411 P 4125b1a70e864e129bd54bdc2720d7fa: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:53.880225 3259 log.cc:824] T 240749b0904f4bb2a4c79b7713473411 P 52c196c15e3348b985c689652bc49918: Log is configured to *not* fsync() on all Append() calls
I20240503 01:58:53.881004 3262 log.cc:824] T 240749b0904f4bb2a4c79b7713473411 P 4125b1a70e864e129bd54bdc2720d7fa: Log is configured to *not* fsync() on all Append() calls
I20240503 01:58:53.882608 3258 raft_consensus.cc:348] T 240749b0904f4bb2a4c79b7713473411 P b63928177fa54227a5ff9929265ab9dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } }
I20240503 01:58:53.882807 3262 tablet_bootstrap.cc:492] T 240749b0904f4bb2a4c79b7713473411 P 4125b1a70e864e129bd54bdc2720d7fa: No bootstrap required, opened a new log
I20240503 01:58:53.882921 3262 ts_tablet_manager.cc:1367] T 240749b0904f4bb2a4c79b7713473411 P 4125b1a70e864e129bd54bdc2720d7fa: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 01:58:53.882923 3258 raft_consensus.cc:374] T 240749b0904f4bb2a4c79b7713473411 P b63928177fa54227a5ff9929265ab9dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:53.882997 3258 raft_consensus.cc:729] T 240749b0904f4bb2a4c79b7713473411 P b63928177fa54227a5ff9929265ab9dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b63928177fa54227a5ff9929265ab9dd, State: Initialized, Role: FOLLOWER
I20240503 01:58:53.883468 3258 consensus_queue.cc:260] T 240749b0904f4bb2a4c79b7713473411 P b63928177fa54227a5ff9929265ab9dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } }
I20240503 01:58:53.884441 3258 ts_tablet_manager.cc:1398] T 240749b0904f4bb2a4c79b7713473411 P b63928177fa54227a5ff9929265ab9dd: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20240503 01:58:53.886664 3254 tablet_bootstrap.cc:492] T 240749b0904f4bb2a4c79b7713473411 P b40a9a7b30d34a6db007ed8febd8a759: No bootstrap required, opened a new log
I20240503 01:58:53.886855 3254 ts_tablet_manager.cc:1367] T 240749b0904f4bb2a4c79b7713473411 P b40a9a7b30d34a6db007ed8febd8a759: Time spent bootstrapping tablet: real 0.024s user 0.005s sys 0.000s
I20240503 01:58:53.888073 3262 raft_consensus.cc:348] T 240749b0904f4bb2a4c79b7713473411 P 4125b1a70e864e129bd54bdc2720d7fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } }
I20240503 01:58:53.888209 3262 raft_consensus.cc:374] T 240749b0904f4bb2a4c79b7713473411 P 4125b1a70e864e129bd54bdc2720d7fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:53.888238 3262 raft_consensus.cc:729] T 240749b0904f4bb2a4c79b7713473411 P 4125b1a70e864e129bd54bdc2720d7fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4125b1a70e864e129bd54bdc2720d7fa, State: Initialized, Role: FOLLOWER
I20240503 01:58:53.888540 3262 consensus_queue.cc:260] T 240749b0904f4bb2a4c79b7713473411 P 4125b1a70e864e129bd54bdc2720d7fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } }
I20240503 01:58:53.889240 3232 heartbeater.cc:503] Master 127.0.99.190:38489 was elected leader, sending a full tablet report...
I20240503 01:58:53.890266 3262 ts_tablet_manager.cc:1398] T 240749b0904f4bb2a4c79b7713473411 P 4125b1a70e864e129bd54bdc2720d7fa: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20240503 01:58:53.891436 3254 raft_consensus.cc:348] T 240749b0904f4bb2a4c79b7713473411 P b40a9a7b30d34a6db007ed8febd8a759 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } }
I20240503 01:58:53.891706 3254 raft_consensus.cc:374] T 240749b0904f4bb2a4c79b7713473411 P b40a9a7b30d34a6db007ed8febd8a759 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:53.891781 3254 raft_consensus.cc:729] T 240749b0904f4bb2a4c79b7713473411 P b40a9a7b30d34a6db007ed8febd8a759 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b40a9a7b30d34a6db007ed8febd8a759, State: Initialized, Role: FOLLOWER
I20240503 01:58:53.892222 3254 consensus_queue.cc:260] T 240749b0904f4bb2a4c79b7713473411 P b40a9a7b30d34a6db007ed8febd8a759 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } }
I20240503 01:58:53.893049 2708 heartbeater.cc:503] Master 127.0.99.190:38489 was elected leader, sending a full tablet report...
I20240503 01:58:53.893815 3254 ts_tablet_manager.cc:1398] T 240749b0904f4bb2a4c79b7713473411 P b40a9a7b30d34a6db007ed8febd8a759: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20240503 01:58:53.898274 3259 tablet_bootstrap.cc:492] T 240749b0904f4bb2a4c79b7713473411 P 52c196c15e3348b985c689652bc49918: No bootstrap required, opened a new log
I20240503 01:58:53.898466 3259 ts_tablet_manager.cc:1367] T 240749b0904f4bb2a4c79b7713473411 P 52c196c15e3348b985c689652bc49918: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.000s
I20240503 01:58:53.903198 3259 raft_consensus.cc:348] T 240749b0904f4bb2a4c79b7713473411 P 52c196c15e3348b985c689652bc49918 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } }
I20240503 01:58:53.903542 3259 raft_consensus.cc:374] T 240749b0904f4bb2a4c79b7713473411 P 52c196c15e3348b985c689652bc49918 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:53.903621 3259 raft_consensus.cc:729] T 240749b0904f4bb2a4c79b7713473411 P 52c196c15e3348b985c689652bc49918 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52c196c15e3348b985c689652bc49918, State: Initialized, Role: FOLLOWER
I20240503 01:58:53.904101 3259 consensus_queue.cc:260] T 240749b0904f4bb2a4c79b7713473411 P 52c196c15e3348b985c689652bc49918 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } }
I20240503 01:58:53.904858 2839 heartbeater.cc:503] Master 127.0.99.190:38489 was elected leader, sending a full tablet report...
I20240503 01:58:53.905190 3259 ts_tablet_manager.cc:1398] T 240749b0904f4bb2a4c79b7713473411 P 52c196c15e3348b985c689652bc49918: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20240503 01:58:53.921037 2577 heartbeater.cc:503] Master 127.0.99.190:38489 was elected leader, sending a full tablet report...
W20240503 01:58:53.986025 2447 tablet.cc:2369] T 240749b0904f4bb2a4c79b7713473411 P b63928177fa54227a5ff9929265ab9dd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 01:58:54.001797 2840 tablet.cc:2369] T 240749b0904f4bb2a4c79b7713473411 P 52c196c15e3348b985c689652bc49918: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 01:58:54.028734 3267 raft_consensus.cc:482] T 240749b0904f4bb2a4c79b7713473411 P b40a9a7b30d34a6db007ed8febd8a759 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:58:54.028955 3267 raft_consensus.cc:504] T 240749b0904f4bb2a4c79b7713473411 P b40a9a7b30d34a6db007ed8febd8a759 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } }
I20240503 01:58:54.030213 3267 leader_election.cc:288] T 240749b0904f4bb2a4c79b7713473411 P b40a9a7b30d34a6db007ed8febd8a759 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52c196c15e3348b985c689652bc49918 (127.0.99.132:42887), 6568f69a78c44bb985162d2b7d0a9376 (127.0.99.134:43603), b63928177fa54227a5ff9929265ab9dd (127.0.99.129:36425), 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135:39501)
I20240503 01:58:54.035424 2793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "240749b0904f4bb2a4c79b7713473411" candidate_uuid: "b40a9a7b30d34a6db007ed8febd8a759" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52c196c15e3348b985c689652bc49918" is_pre_election: true
I20240503 01:58:54.035586 3055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "240749b0904f4bb2a4c79b7713473411" candidate_uuid: "b40a9a7b30d34a6db007ed8febd8a759" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6568f69a78c44bb985162d2b7d0a9376" is_pre_election: true
I20240503 01:58:54.035727 2793 raft_consensus.cc:2454] T 240749b0904f4bb2a4c79b7713473411 P 52c196c15e3348b985c689652bc49918 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b40a9a7b30d34a6db007ed8febd8a759 in term 0.
I20240503 01:58:54.035862 3055 raft_consensus.cc:2454] T 240749b0904f4bb2a4c79b7713473411 P 6568f69a78c44bb985162d2b7d0a9376 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b40a9a7b30d34a6db007ed8febd8a759 in term 0.
I20240503 01:58:54.036096 2400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "240749b0904f4bb2a4c79b7713473411" candidate_uuid: "b40a9a7b30d34a6db007ed8febd8a759" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b63928177fa54227a5ff9929265ab9dd" is_pre_election: true
I20240503 01:58:54.036355 2400 raft_consensus.cc:2454] T 240749b0904f4bb2a4c79b7713473411 P b63928177fa54227a5ff9929265ab9dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b40a9a7b30d34a6db007ed8febd8a759 in term 0.
I20240503 01:58:54.036430 2598 leader_election.cc:302] T 240749b0904f4bb2a4c79b7713473411 P b40a9a7b30d34a6db007ed8febd8a759 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 52c196c15e3348b985c689652bc49918, 6568f69a78c44bb985162d2b7d0a9376, b40a9a7b30d34a6db007ed8febd8a759; no voters:
I20240503 01:58:54.036686 3267 raft_consensus.cc:2781] T 240749b0904f4bb2a4c79b7713473411 P b40a9a7b30d34a6db007ed8febd8a759 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:58:54.036777 3267 raft_consensus.cc:482] T 240749b0904f4bb2a4c79b7713473411 P b40a9a7b30d34a6db007ed8febd8a759 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:58:54.036839 3267 raft_consensus.cc:3037] T 240749b0904f4bb2a4c79b7713473411 P b40a9a7b30d34a6db007ed8febd8a759 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:54.037156 3185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "240749b0904f4bb2a4c79b7713473411" candidate_uuid: "b40a9a7b30d34a6db007ed8febd8a759" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4125b1a70e864e129bd54bdc2720d7fa" is_pre_election: true
I20240503 01:58:54.037405 3185 raft_consensus.cc:2454] T 240749b0904f4bb2a4c79b7713473411 P 4125b1a70e864e129bd54bdc2720d7fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b40a9a7b30d34a6db007ed8febd8a759 in term 0.
I20240503 01:58:54.038064 3267 raft_consensus.cc:504] T 240749b0904f4bb2a4c79b7713473411 P b40a9a7b30d34a6db007ed8febd8a759 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } }
I20240503 01:58:54.038458 3267 leader_election.cc:288] T 240749b0904f4bb2a4c79b7713473411 P b40a9a7b30d34a6db007ed8febd8a759 [CANDIDATE]: Term 1 election: Requested vote from peers 52c196c15e3348b985c689652bc49918 (127.0.99.132:42887), 6568f69a78c44bb985162d2b7d0a9376 (127.0.99.134:43603), b63928177fa54227a5ff9929265ab9dd (127.0.99.129:36425), 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135:39501)
I20240503 01:58:54.038731 3055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "240749b0904f4bb2a4c79b7713473411" candidate_uuid: "b40a9a7b30d34a6db007ed8febd8a759" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6568f69a78c44bb985162d2b7d0a9376"
I20240503 01:58:54.038771 2793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "240749b0904f4bb2a4c79b7713473411" candidate_uuid: "b40a9a7b30d34a6db007ed8febd8a759" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52c196c15e3348b985c689652bc49918"
I20240503 01:58:54.038851 3055 raft_consensus.cc:3037] T 240749b0904f4bb2a4c79b7713473411 P 6568f69a78c44bb985162d2b7d0a9376 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:54.038887 2793 raft_consensus.cc:3037] T 240749b0904f4bb2a4c79b7713473411 P 52c196c15e3348b985c689652bc49918 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:54.039858 3055 raft_consensus.cc:2454] T 240749b0904f4bb2a4c79b7713473411 P 6568f69a78c44bb985162d2b7d0a9376 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b40a9a7b30d34a6db007ed8febd8a759 in term 1.
I20240503 01:58:54.040123 2400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "240749b0904f4bb2a4c79b7713473411" candidate_uuid: "b40a9a7b30d34a6db007ed8febd8a759" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b63928177fa54227a5ff9929265ab9dd"
I20240503 01:58:54.040242 2400 raft_consensus.cc:3037] T 240749b0904f4bb2a4c79b7713473411 P b63928177fa54227a5ff9929265ab9dd [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:54.040258 2793 raft_consensus.cc:2454] T 240749b0904f4bb2a4c79b7713473411 P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b40a9a7b30d34a6db007ed8febd8a759 in term 1.
I20240503 01:58:54.040757 2596 leader_election.cc:302] T 240749b0904f4bb2a4c79b7713473411 P b40a9a7b30d34a6db007ed8febd8a759 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 52c196c15e3348b985c689652bc49918, 6568f69a78c44bb985162d2b7d0a9376, b40a9a7b30d34a6db007ed8febd8a759; no voters:
I20240503 01:58:54.040910 3267 raft_consensus.cc:2781] T 240749b0904f4bb2a4c79b7713473411 P b40a9a7b30d34a6db007ed8febd8a759 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:58:54.040964 3185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "240749b0904f4bb2a4c79b7713473411" candidate_uuid: "b40a9a7b30d34a6db007ed8febd8a759" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4125b1a70e864e129bd54bdc2720d7fa"
I20240503 01:58:54.041075 3185 raft_consensus.cc:3037] T 240749b0904f4bb2a4c79b7713473411 P 4125b1a70e864e129bd54bdc2720d7fa [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:54.041203 3267 raft_consensus.cc:686] T 240749b0904f4bb2a4c79b7713473411 P b40a9a7b30d34a6db007ed8febd8a759 [term 1 LEADER]: Becoming Leader. State: Replica: b40a9a7b30d34a6db007ed8febd8a759, State: Running, Role: LEADER
I20240503 01:58:54.041246 2400 raft_consensus.cc:2454] T 240749b0904f4bb2a4c79b7713473411 P b63928177fa54227a5ff9929265ab9dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b40a9a7b30d34a6db007ed8febd8a759 in term 1.
I20240503 01:58:54.041535 3267 consensus_queue.cc:237] T 240749b0904f4bb2a4c79b7713473411 P b40a9a7b30d34a6db007ed8febd8a759 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } }
I20240503 01:58:54.042256 3185 raft_consensus.cc:2454] T 240749b0904f4bb2a4c79b7713473411 P 4125b1a70e864e129bd54bdc2720d7fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b40a9a7b30d34a6db007ed8febd8a759 in term 1.
I20240503 01:58:54.044828 2258 catalog_manager.cc:5484] T 240749b0904f4bb2a4c79b7713473411 P b40a9a7b30d34a6db007ed8febd8a759 reported cstate change: term changed from 0 to 1, leader changed from <none> to b40a9a7b30d34a6db007ed8febd8a759 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "b40a9a7b30d34a6db007ed8febd8a759" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } health_report { overall_health: UNKNOWN } } }
W20240503 01:58:54.063853 3233 tablet.cc:2369] T 240749b0904f4bb2a4c79b7713473411 P 4125b1a70e864e129bd54bdc2720d7fa: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 01:58:54.078680 2709 tablet.cc:2369] T 240749b0904f4bb2a4c79b7713473411 P b40a9a7b30d34a6db007ed8febd8a759: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 01:58:54.088594 398 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240503 01:58:54.090133 398 ts_itest-base.cc:233] Waiting for 1 tablets on tserver b63928177fa54227a5ff9929265ab9dd to finish bootstrapping
W20240503 01:58:54.093232 3102 tablet.cc:2369] T 240749b0904f4bb2a4c79b7713473411 P 6568f69a78c44bb985162d2b7d0a9376: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 01:58:54.095469 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver a1a844ecd71145e4a362f4c42500fbcb to finish bootstrapping
I20240503 01:58:54.101050 398 ts_itest-base.cc:233] Waiting for 1 tablets on tserver b40a9a7b30d34a6db007ed8febd8a759 to finish bootstrapping
I20240503 01:58:54.105480 398 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 52c196c15e3348b985c689652bc49918 to finish bootstrapping
I20240503 01:58:54.109555 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 327aa26a8e434a609ebcb382aa665fce to finish bootstrapping
I20240503 01:58:54.115036 398 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 6568f69a78c44bb985162d2b7d0a9376 to finish bootstrapping
I20240503 01:58:54.119591 398 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 4125b1a70e864e129bd54bdc2720d7fa to finish bootstrapping
I20240503 01:58:54.123920 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task42rE7p/build/debug/bin/kudu with pid 3104
I20240503 01:58:54.436240 2970 heartbeater.cc:503] Master 127.0.99.190:38489 was elected leader, sending a full tablet report...
I20240503 01:58:54.489653 3267 consensus_queue.cc:1035] T 240749b0904f4bb2a4c79b7713473411 P b40a9a7b30d34a6db007ed8febd8a759 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:54.517853 3267 consensus_queue.cc:1035] T 240749b0904f4bb2a4c79b7713473411 P b40a9a7b30d34a6db007ed8febd8a759 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 01:58:54.523350 2598 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:39501: connect: Connection refused (error 111)
I20240503 01:58:54.523847 3273 consensus_queue.cc:1035] T 240749b0904f4bb2a4c79b7713473411 P b40a9a7b30d34a6db007ed8febd8a759 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 01:58:54.524937 2598 consensus_peers.cc:485] T 240749b0904f4bb2a4c79b7713473411 P b40a9a7b30d34a6db007ed8febd8a759 -> Peer 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135:39501): Couldn't send request to peer 4125b1a70e864e129bd54bdc2720d7fa. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:39501: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 01:58:56.768915 2598 consensus_peers.cc:485] T 240749b0904f4bb2a4c79b7713473411 P b40a9a7b30d34a6db007ed8febd8a759 -> Peer 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135:39501): Couldn't send request to peer 4125b1a70e864e129bd54bdc2720d7fa. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:39501: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 01:58:57.881333 2257 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53804:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 5
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:58:57.888981 2642 tablet_service.cc:1467] Processing CreateTablet for tablet 568a05972208414c90f7f4e2ea2c10a9 (DEFAULT_TABLE table=rebalance_test_table_0 [id=2de29504ce33484ba12dd9985ab742a9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:57.889374 2642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 568a05972208414c90f7f4e2ea2c10a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:57.889636 2641 tablet_service.cc:1467] Processing CreateTablet for tablet 43008abe74604666b330d78ec0f51c7c (DEFAULT_TABLE table=rebalance_test_table_0 [id=2de29504ce33484ba12dd9985ab742a9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:57.889874 2641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43008abe74604666b330d78ec0f51c7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:57.890426 2380 tablet_service.cc:1467] Processing CreateTablet for tablet 568a05972208414c90f7f4e2ea2c10a9 (DEFAULT_TABLE table=rebalance_test_table_0 [id=2de29504ce33484ba12dd9985ab742a9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:57.890733 2380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 568a05972208414c90f7f4e2ea2c10a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:57.891019 2379 tablet_service.cc:1467] Processing CreateTablet for tablet 43008abe74604666b330d78ec0f51c7c (DEFAULT_TABLE table=rebalance_test_table_0 [id=2de29504ce33484ba12dd9985ab742a9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:57.891247 2379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43008abe74604666b330d78ec0f51c7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:57.892530 3035 tablet_service.cc:1467] Processing CreateTablet for tablet 43008abe74604666b330d78ec0f51c7c (DEFAULT_TABLE table=rebalance_test_table_0 [id=2de29504ce33484ba12dd9985ab742a9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:57.892606 2773 tablet_service.cc:1467] Processing CreateTablet for tablet 568a05972208414c90f7f4e2ea2c10a9 (DEFAULT_TABLE table=rebalance_test_table_0 [id=2de29504ce33484ba12dd9985ab742a9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:57.892853 3035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43008abe74604666b330d78ec0f51c7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:57.892880 2773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 568a05972208414c90f7f4e2ea2c10a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:57.893272 3034 tablet_service.cc:1467] Processing CreateTablet for tablet c9787dcf6d674611a2f06c143a291bb5 (DEFAULT_TABLE table=rebalance_test_table_0 [id=2de29504ce33484ba12dd9985ab742a9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:57.893290 2378 tablet_service.cc:1467] Processing CreateTablet for tablet c9787dcf6d674611a2f06c143a291bb5 (DEFAULT_TABLE table=rebalance_test_table_0 [id=2de29504ce33484ba12dd9985ab742a9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:57.893497 2378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9787dcf6d674611a2f06c143a291bb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:57.893501 3034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9787dcf6d674611a2f06c143a291bb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:57.895228 2772 tablet_service.cc:1467] Processing CreateTablet for tablet c9787dcf6d674611a2f06c143a291bb5 (DEFAULT_TABLE table=rebalance_test_table_0 [id=2de29504ce33484ba12dd9985ab742a9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:57.895483 2772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9787dcf6d674611a2f06c143a291bb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:57.895717 3307 tablet_bootstrap.cc:492] T 568a05972208414c90f7f4e2ea2c10a9 P b63928177fa54227a5ff9929265ab9dd: Bootstrap starting.
I20240503 01:58:57.895937 3309 tablet_bootstrap.cc:492] T 568a05972208414c90f7f4e2ea2c10a9 P 52c196c15e3348b985c689652bc49918: Bootstrap starting.
I20240503 01:58:57.896795 3309 tablet_bootstrap.cc:654] T 568a05972208414c90f7f4e2ea2c10a9 P 52c196c15e3348b985c689652bc49918: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:57.896852 3308 tablet_bootstrap.cc:492] T 43008abe74604666b330d78ec0f51c7c P 6568f69a78c44bb985162d2b7d0a9376: Bootstrap starting.
I20240503 01:58:57.897748 3308 tablet_bootstrap.cc:654] T 43008abe74604666b330d78ec0f51c7c P 6568f69a78c44bb985162d2b7d0a9376: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:57.897754 3307 tablet_bootstrap.cc:654] T 568a05972208414c90f7f4e2ea2c10a9 P b63928177fa54227a5ff9929265ab9dd: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:57.898630 3306 tablet_bootstrap.cc:492] T 568a05972208414c90f7f4e2ea2c10a9 P b40a9a7b30d34a6db007ed8febd8a759: Bootstrap starting.
I20240503 01:58:57.898869 3307 tablet_bootstrap.cc:492] T 568a05972208414c90f7f4e2ea2c10a9 P b63928177fa54227a5ff9929265ab9dd: No bootstrap required, opened a new log
I20240503 01:58:57.899008 3307 ts_tablet_manager.cc:1367] T 568a05972208414c90f7f4e2ea2c10a9 P b63928177fa54227a5ff9929265ab9dd: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:58:57.900446 3306 tablet_bootstrap.cc:654] T 568a05972208414c90f7f4e2ea2c10a9 P b40a9a7b30d34a6db007ed8febd8a759: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:57.900926 3307 raft_consensus.cc:348] T 568a05972208414c90f7f4e2ea2c10a9 P b63928177fa54227a5ff9929265ab9dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } }
I20240503 01:58:57.901201 3307 raft_consensus.cc:374] T 568a05972208414c90f7f4e2ea2c10a9 P b63928177fa54227a5ff9929265ab9dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:57.901285 3307 raft_consensus.cc:729] T 568a05972208414c90f7f4e2ea2c10a9 P b63928177fa54227a5ff9929265ab9dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b63928177fa54227a5ff9929265ab9dd, State: Initialized, Role: FOLLOWER
I20240503 01:58:57.901470 3307 consensus_queue.cc:260] T 568a05972208414c90f7f4e2ea2c10a9 P b63928177fa54227a5ff9929265ab9dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } }
I20240503 01:58:57.901669 3309 tablet_bootstrap.cc:492] T 568a05972208414c90f7f4e2ea2c10a9 P 52c196c15e3348b985c689652bc49918: No bootstrap required, opened a new log
I20240503 01:58:57.901809 3309 ts_tablet_manager.cc:1367] T 568a05972208414c90f7f4e2ea2c10a9 P 52c196c15e3348b985c689652bc49918: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20240503 01:58:57.901849 3307 ts_tablet_manager.cc:1398] T 568a05972208414c90f7f4e2ea2c10a9 P b63928177fa54227a5ff9929265ab9dd: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 01:58:57.902000 2511 tablet_service.cc:1467] Processing CreateTablet for tablet 568a05972208414c90f7f4e2ea2c10a9 (DEFAULT_TABLE table=rebalance_test_table_0 [id=2de29504ce33484ba12dd9985ab742a9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:57.902421 3309 raft_consensus.cc:348] T 568a05972208414c90f7f4e2ea2c10a9 P 52c196c15e3348b985c689652bc49918 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } }
I20240503 01:58:57.902534 3309 raft_consensus.cc:374] T 568a05972208414c90f7f4e2ea2c10a9 P 52c196c15e3348b985c689652bc49918 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:57.902575 3309 raft_consensus.cc:729] T 568a05972208414c90f7f4e2ea2c10a9 P 52c196c15e3348b985c689652bc49918 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52c196c15e3348b985c689652bc49918, State: Initialized, Role: FOLLOWER
I20240503 01:58:57.902776 3309 consensus_queue.cc:260] T 568a05972208414c90f7f4e2ea2c10a9 P 52c196c15e3348b985c689652bc49918 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } }
I20240503 01:58:57.902995 2511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 568a05972208414c90f7f4e2ea2c10a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:57.903151 3309 ts_tablet_manager.cc:1398] T 568a05972208414c90f7f4e2ea2c10a9 P 52c196c15e3348b985c689652bc49918: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:58:57.903380 3309 tablet_bootstrap.cc:492] T c9787dcf6d674611a2f06c143a291bb5 P 52c196c15e3348b985c689652bc49918: Bootstrap starting.
I20240503 01:58:57.903594 3308 tablet_bootstrap.cc:492] T 43008abe74604666b330d78ec0f51c7c P 6568f69a78c44bb985162d2b7d0a9376: No bootstrap required, opened a new log
I20240503 01:58:57.903668 3308 ts_tablet_manager.cc:1367] T 43008abe74604666b330d78ec0f51c7c P 6568f69a78c44bb985162d2b7d0a9376: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 01:58:57.904261 3309 tablet_bootstrap.cc:654] T c9787dcf6d674611a2f06c143a291bb5 P 52c196c15e3348b985c689652bc49918: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:57.904191 3308 raft_consensus.cc:348] T 43008abe74604666b330d78ec0f51c7c P 6568f69a78c44bb985162d2b7d0a9376 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } }
I20240503 01:58:57.904322 3308 raft_consensus.cc:374] T 43008abe74604666b330d78ec0f51c7c P 6568f69a78c44bb985162d2b7d0a9376 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:57.904357 3308 raft_consensus.cc:729] T 43008abe74604666b330d78ec0f51c7c P 6568f69a78c44bb985162d2b7d0a9376 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6568f69a78c44bb985162d2b7d0a9376, State: Initialized, Role: FOLLOWER
I20240503 01:58:57.904493 3308 consensus_queue.cc:260] T 43008abe74604666b330d78ec0f51c7c P 6568f69a78c44bb985162d2b7d0a9376 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } }
I20240503 01:58:57.904824 3308 ts_tablet_manager.cc:1398] T 43008abe74604666b330d78ec0f51c7c P 6568f69a78c44bb985162d2b7d0a9376: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:58:57.904985 3308 tablet_bootstrap.cc:492] T c9787dcf6d674611a2f06c143a291bb5 P 6568f69a78c44bb985162d2b7d0a9376: Bootstrap starting.
I20240503 01:58:57.905623 2509 tablet_service.cc:1467] Processing CreateTablet for tablet c9787dcf6d674611a2f06c143a291bb5 (DEFAULT_TABLE table=rebalance_test_table_0 [id=2de29504ce33484ba12dd9985ab742a9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:57.905925 2509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9787dcf6d674611a2f06c143a291bb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:57.906064 3307 tablet_bootstrap.cc:492] T 43008abe74604666b330d78ec0f51c7c P b63928177fa54227a5ff9929265ab9dd: Bootstrap starting.
I20240503 01:58:57.906524 3308 tablet_bootstrap.cc:654] T c9787dcf6d674611a2f06c143a291bb5 P 6568f69a78c44bb985162d2b7d0a9376: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:57.906222 2904 tablet_service.cc:1467] Processing CreateTablet for tablet 568a05972208414c90f7f4e2ea2c10a9 (DEFAULT_TABLE table=rebalance_test_table_0 [id=2de29504ce33484ba12dd9985ab742a9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:57.906670 2510 tablet_service.cc:1467] Processing CreateTablet for tablet 43008abe74604666b330d78ec0f51c7c (DEFAULT_TABLE table=rebalance_test_table_0 [id=2de29504ce33484ba12dd9985ab742a9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:57.906951 2510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43008abe74604666b330d78ec0f51c7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:57.907104 2904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 568a05972208414c90f7f4e2ea2c10a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:57.907610 3307 tablet_bootstrap.cc:654] T 43008abe74604666b330d78ec0f51c7c P b63928177fa54227a5ff9929265ab9dd: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:57.907828 2902 tablet_service.cc:1467] Processing CreateTablet for tablet c9787dcf6d674611a2f06c143a291bb5 (DEFAULT_TABLE table=rebalance_test_table_0 [id=2de29504ce33484ba12dd9985ab742a9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:57.908249 2903 tablet_service.cc:1467] Processing CreateTablet for tablet 43008abe74604666b330d78ec0f51c7c (DEFAULT_TABLE table=rebalance_test_table_0 [id=2de29504ce33484ba12dd9985ab742a9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:57.908546 2902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9787dcf6d674611a2f06c143a291bb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:57.908987 2903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43008abe74604666b330d78ec0f51c7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:57.914410 3319 tablet_bootstrap.cc:492] T 43008abe74604666b330d78ec0f51c7c P 327aa26a8e434a609ebcb382aa665fce: Bootstrap starting.
I20240503 01:58:57.914657 3307 tablet_bootstrap.cc:492] T 43008abe74604666b330d78ec0f51c7c P b63928177fa54227a5ff9929265ab9dd: No bootstrap required, opened a new log
I20240503 01:58:57.914741 3307 ts_tablet_manager.cc:1367] T 43008abe74604666b330d78ec0f51c7c P b63928177fa54227a5ff9929265ab9dd: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 01:58:57.915181 3307 raft_consensus.cc:348] T 43008abe74604666b330d78ec0f51c7c P b63928177fa54227a5ff9929265ab9dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } }
I20240503 01:58:57.915282 3307 raft_consensus.cc:374] T 43008abe74604666b330d78ec0f51c7c P b63928177fa54227a5ff9929265ab9dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:57.915321 3307 raft_consensus.cc:729] T 43008abe74604666b330d78ec0f51c7c P b63928177fa54227a5ff9929265ab9dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b63928177fa54227a5ff9929265ab9dd, State: Initialized, Role: FOLLOWER
I20240503 01:58:57.915593 3318 tablet_bootstrap.cc:492] T c9787dcf6d674611a2f06c143a291bb5 P a1a844ecd71145e4a362f4c42500fbcb: Bootstrap starting.
I20240503 01:58:57.915591 3307 consensus_queue.cc:260] T 43008abe74604666b330d78ec0f51c7c P b63928177fa54227a5ff9929265ab9dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } }
I20240503 01:58:57.916072 3307 ts_tablet_manager.cc:1398] T 43008abe74604666b330d78ec0f51c7c P b63928177fa54227a5ff9929265ab9dd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:58:57.916519 3319 tablet_bootstrap.cc:654] T 43008abe74604666b330d78ec0f51c7c P 327aa26a8e434a609ebcb382aa665fce: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:57.916940 3307 tablet_bootstrap.cc:492] T c9787dcf6d674611a2f06c143a291bb5 P b63928177fa54227a5ff9929265ab9dd: Bootstrap starting.
I20240503 01:58:57.917294 3319 log.cc:824] T 43008abe74604666b330d78ec0f51c7c P 327aa26a8e434a609ebcb382aa665fce: Log is configured to *not* fsync() on all Append() calls
I20240503 01:58:57.917856 3318 tablet_bootstrap.cc:654] T c9787dcf6d674611a2f06c143a291bb5 P a1a844ecd71145e4a362f4c42500fbcb: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:57.918469 3308 tablet_bootstrap.cc:492] T c9787dcf6d674611a2f06c143a291bb5 P 6568f69a78c44bb985162d2b7d0a9376: No bootstrap required, opened a new log
I20240503 01:58:57.918571 3308 ts_tablet_manager.cc:1367] T c9787dcf6d674611a2f06c143a291bb5 P 6568f69a78c44bb985162d2b7d0a9376: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.002s
I20240503 01:58:57.918634 3318 log.cc:824] T c9787dcf6d674611a2f06c143a291bb5 P a1a844ecd71145e4a362f4c42500fbcb: Log is configured to *not* fsync() on all Append() calls
I20240503 01:58:57.919054 3308 raft_consensus.cc:348] T c9787dcf6d674611a2f06c143a291bb5 P 6568f69a78c44bb985162d2b7d0a9376 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } }
I20240503 01:58:57.919164 3308 raft_consensus.cc:374] T c9787dcf6d674611a2f06c143a291bb5 P 6568f69a78c44bb985162d2b7d0a9376 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:57.919198 3308 raft_consensus.cc:729] T c9787dcf6d674611a2f06c143a291bb5 P 6568f69a78c44bb985162d2b7d0a9376 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6568f69a78c44bb985162d2b7d0a9376, State: Initialized, Role: FOLLOWER
I20240503 01:58:57.919376 3307 tablet_bootstrap.cc:654] T c9787dcf6d674611a2f06c143a291bb5 P b63928177fa54227a5ff9929265ab9dd: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:57.919312 3308 consensus_queue.cc:260] T c9787dcf6d674611a2f06c143a291bb5 P 6568f69a78c44bb985162d2b7d0a9376 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } }
I20240503 01:58:57.919615 3308 ts_tablet_manager.cc:1398] T c9787dcf6d674611a2f06c143a291bb5 P 6568f69a78c44bb985162d2b7d0a9376: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:58:57.920382 3307 tablet_bootstrap.cc:492] T c9787dcf6d674611a2f06c143a291bb5 P b63928177fa54227a5ff9929265ab9dd: No bootstrap required, opened a new log
I20240503 01:58:57.920483 3307 ts_tablet_manager.cc:1367] T c9787dcf6d674611a2f06c143a291bb5 P b63928177fa54227a5ff9929265ab9dd: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 01:58:57.920521 3309 tablet_bootstrap.cc:492] T c9787dcf6d674611a2f06c143a291bb5 P 52c196c15e3348b985c689652bc49918: No bootstrap required, opened a new log
I20240503 01:58:57.920608 3309 ts_tablet_manager.cc:1367] T c9787dcf6d674611a2f06c143a291bb5 P 52c196c15e3348b985c689652bc49918: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.000s
I20240503 01:58:57.920825 3318 tablet_bootstrap.cc:492] T c9787dcf6d674611a2f06c143a291bb5 P a1a844ecd71145e4a362f4c42500fbcb: No bootstrap required, opened a new log
I20240503 01:58:57.920974 3318 ts_tablet_manager.cc:1367] T c9787dcf6d674611a2f06c143a291bb5 P a1a844ecd71145e4a362f4c42500fbcb: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240503 01:58:57.920939 3307 raft_consensus.cc:348] T c9787dcf6d674611a2f06c143a291bb5 P b63928177fa54227a5ff9929265ab9dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } }
I20240503 01:58:57.921046 3307 raft_consensus.cc:374] T c9787dcf6d674611a2f06c143a291bb5 P b63928177fa54227a5ff9929265ab9dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:57.921082 3307 raft_consensus.cc:729] T c9787dcf6d674611a2f06c143a291bb5 P b63928177fa54227a5ff9929265ab9dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b63928177fa54227a5ff9929265ab9dd, State: Initialized, Role: FOLLOWER
I20240503 01:58:57.921104 3309 raft_consensus.cc:348] T c9787dcf6d674611a2f06c143a291bb5 P 52c196c15e3348b985c689652bc49918 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } }
I20240503 01:58:57.921226 3309 raft_consensus.cc:374] T c9787dcf6d674611a2f06c143a291bb5 P 52c196c15e3348b985c689652bc49918 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:57.921257 3309 raft_consensus.cc:729] T c9787dcf6d674611a2f06c143a291bb5 P 52c196c15e3348b985c689652bc49918 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52c196c15e3348b985c689652bc49918, State: Initialized, Role: FOLLOWER
I20240503 01:58:57.921203 3307 consensus_queue.cc:260] T c9787dcf6d674611a2f06c143a291bb5 P b63928177fa54227a5ff9929265ab9dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } }
I20240503 01:58:57.921368 3309 consensus_queue.cc:260] T c9787dcf6d674611a2f06c143a291bb5 P 52c196c15e3348b985c689652bc49918 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } }
I20240503 01:58:57.921511 3307 ts_tablet_manager.cc:1398] T c9787dcf6d674611a2f06c143a291bb5 P b63928177fa54227a5ff9929265ab9dd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:58:57.921608 3309 ts_tablet_manager.cc:1398] T c9787dcf6d674611a2f06c143a291bb5 P 52c196c15e3348b985c689652bc49918: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:58:57.922494 3306 tablet_bootstrap.cc:492] T 568a05972208414c90f7f4e2ea2c10a9 P b40a9a7b30d34a6db007ed8febd8a759: No bootstrap required, opened a new log
I20240503 01:58:57.922601 3306 ts_tablet_manager.cc:1367] T 568a05972208414c90f7f4e2ea2c10a9 P b40a9a7b30d34a6db007ed8febd8a759: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.000s
I20240503 01:58:57.923110 3306 raft_consensus.cc:348] T 568a05972208414c90f7f4e2ea2c10a9 P b40a9a7b30d34a6db007ed8febd8a759 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } }
I20240503 01:58:57.923229 3306 raft_consensus.cc:374] T 568a05972208414c90f7f4e2ea2c10a9 P b40a9a7b30d34a6db007ed8febd8a759 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:57.923261 3306 raft_consensus.cc:729] T 568a05972208414c90f7f4e2ea2c10a9 P b40a9a7b30d34a6db007ed8febd8a759 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b40a9a7b30d34a6db007ed8febd8a759, State: Initialized, Role: FOLLOWER
I20240503 01:58:57.923395 3306 consensus_queue.cc:260] T 568a05972208414c90f7f4e2ea2c10a9 P b40a9a7b30d34a6db007ed8febd8a759 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } }
I20240503 01:58:57.923530 3319 tablet_bootstrap.cc:492] T 43008abe74604666b330d78ec0f51c7c P 327aa26a8e434a609ebcb382aa665fce: No bootstrap required, opened a new log
I20240503 01:58:57.923669 3306 ts_tablet_manager.cc:1398] T 568a05972208414c90f7f4e2ea2c10a9 P b40a9a7b30d34a6db007ed8febd8a759: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:58:57.923676 3319 ts_tablet_manager.cc:1367] T 43008abe74604666b330d78ec0f51c7c P 327aa26a8e434a609ebcb382aa665fce: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20240503 01:58:57.923830 3306 tablet_bootstrap.cc:492] T 43008abe74604666b330d78ec0f51c7c P b40a9a7b30d34a6db007ed8febd8a759: Bootstrap starting.
I20240503 01:58:57.924728 3306 tablet_bootstrap.cc:654] T 43008abe74604666b330d78ec0f51c7c P b40a9a7b30d34a6db007ed8febd8a759: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:57.926321 3306 tablet_bootstrap.cc:492] T 43008abe74604666b330d78ec0f51c7c P b40a9a7b30d34a6db007ed8febd8a759: No bootstrap required, opened a new log
I20240503 01:58:57.926457 3306 ts_tablet_manager.cc:1367] T 43008abe74604666b330d78ec0f51c7c P b40a9a7b30d34a6db007ed8febd8a759: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 01:58:57.926921 3306 raft_consensus.cc:348] T 43008abe74604666b330d78ec0f51c7c P b40a9a7b30d34a6db007ed8febd8a759 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } }
I20240503 01:58:57.927111 3306 raft_consensus.cc:374] T 43008abe74604666b330d78ec0f51c7c P b40a9a7b30d34a6db007ed8febd8a759 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:57.927194 3306 raft_consensus.cc:729] T 43008abe74604666b330d78ec0f51c7c P b40a9a7b30d34a6db007ed8febd8a759 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b40a9a7b30d34a6db007ed8febd8a759, State: Initialized, Role: FOLLOWER
I20240503 01:58:57.927349 3306 consensus_queue.cc:260] T 43008abe74604666b330d78ec0f51c7c P b40a9a7b30d34a6db007ed8febd8a759 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } }
I20240503 01:58:57.927326 3319 raft_consensus.cc:348] T 43008abe74604666b330d78ec0f51c7c P 327aa26a8e434a609ebcb382aa665fce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } }
I20240503 01:58:57.927544 3319 raft_consensus.cc:374] T 43008abe74604666b330d78ec0f51c7c P 327aa26a8e434a609ebcb382aa665fce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:57.927603 3319 raft_consensus.cc:729] T 43008abe74604666b330d78ec0f51c7c P 327aa26a8e434a609ebcb382aa665fce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 327aa26a8e434a609ebcb382aa665fce, State: Initialized, Role: FOLLOWER
I20240503 01:58:57.927692 3306 ts_tablet_manager.cc:1398] T 43008abe74604666b330d78ec0f51c7c P b40a9a7b30d34a6db007ed8febd8a759: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:58:57.928123 3319 consensus_queue.cc:260] T 43008abe74604666b330d78ec0f51c7c P 327aa26a8e434a609ebcb382aa665fce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } }
I20240503 01:58:57.929005 3319 ts_tablet_manager.cc:1398] T 43008abe74604666b330d78ec0f51c7c P 327aa26a8e434a609ebcb382aa665fce: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20240503 01:58:57.929262 3319 tablet_bootstrap.cc:492] T 568a05972208414c90f7f4e2ea2c10a9 P 327aa26a8e434a609ebcb382aa665fce: Bootstrap starting.
I20240503 01:58:57.930176 3319 tablet_bootstrap.cc:654] T 568a05972208414c90f7f4e2ea2c10a9 P 327aa26a8e434a609ebcb382aa665fce: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:57.930253 3318 raft_consensus.cc:348] T c9787dcf6d674611a2f06c143a291bb5 P a1a844ecd71145e4a362f4c42500fbcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } }
I20240503 01:58:57.930454 3318 raft_consensus.cc:374] T c9787dcf6d674611a2f06c143a291bb5 P a1a844ecd71145e4a362f4c42500fbcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:57.930516 3318 raft_consensus.cc:729] T c9787dcf6d674611a2f06c143a291bb5 P a1a844ecd71145e4a362f4c42500fbcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1a844ecd71145e4a362f4c42500fbcb, State: Initialized, Role: FOLLOWER
I20240503 01:58:57.930955 3318 consensus_queue.cc:260] T c9787dcf6d674611a2f06c143a291bb5 P a1a844ecd71145e4a362f4c42500fbcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } }
I20240503 01:58:57.931524 3319 tablet_bootstrap.cc:492] T 568a05972208414c90f7f4e2ea2c10a9 P 327aa26a8e434a609ebcb382aa665fce: No bootstrap required, opened a new log
I20240503 01:58:57.931612 3319 ts_tablet_manager.cc:1367] T 568a05972208414c90f7f4e2ea2c10a9 P 327aa26a8e434a609ebcb382aa665fce: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:58:57.931802 3318 ts_tablet_manager.cc:1398] T c9787dcf6d674611a2f06c143a291bb5 P a1a844ecd71145e4a362f4c42500fbcb: Time spent starting tablet: real 0.011s user 0.005s sys 0.000s
I20240503 01:58:57.932024 3318 tablet_bootstrap.cc:492] T 43008abe74604666b330d78ec0f51c7c P a1a844ecd71145e4a362f4c42500fbcb: Bootstrap starting.
I20240503 01:58:57.932158 3319 raft_consensus.cc:348] T 568a05972208414c90f7f4e2ea2c10a9 P 327aa26a8e434a609ebcb382aa665fce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } }
I20240503 01:58:57.932309 3319 raft_consensus.cc:374] T 568a05972208414c90f7f4e2ea2c10a9 P 327aa26a8e434a609ebcb382aa665fce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:57.932377 3319 raft_consensus.cc:729] T 568a05972208414c90f7f4e2ea2c10a9 P 327aa26a8e434a609ebcb382aa665fce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 327aa26a8e434a609ebcb382aa665fce, State: Initialized, Role: FOLLOWER
I20240503 01:58:57.932548 3319 consensus_queue.cc:260] T 568a05972208414c90f7f4e2ea2c10a9 P 327aa26a8e434a609ebcb382aa665fce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } }
I20240503 01:58:57.932987 3318 tablet_bootstrap.cc:654] T 43008abe74604666b330d78ec0f51c7c P a1a844ecd71145e4a362f4c42500fbcb: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:57.933079 3319 ts_tablet_manager.cc:1398] T 568a05972208414c90f7f4e2ea2c10a9 P 327aa26a8e434a609ebcb382aa665fce: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:58:57.933307 3319 tablet_bootstrap.cc:492] T c9787dcf6d674611a2f06c143a291bb5 P 327aa26a8e434a609ebcb382aa665fce: Bootstrap starting.
I20240503 01:58:57.934125 3318 tablet_bootstrap.cc:492] T 43008abe74604666b330d78ec0f51c7c P a1a844ecd71145e4a362f4c42500fbcb: No bootstrap required, opened a new log
I20240503 01:58:57.934219 3318 ts_tablet_manager.cc:1367] T 43008abe74604666b330d78ec0f51c7c P a1a844ecd71145e4a362f4c42500fbcb: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:58:57.934238 3319 tablet_bootstrap.cc:654] T c9787dcf6d674611a2f06c143a291bb5 P 327aa26a8e434a609ebcb382aa665fce: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:57.934732 3318 raft_consensus.cc:348] T 43008abe74604666b330d78ec0f51c7c P a1a844ecd71145e4a362f4c42500fbcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } }
I20240503 01:58:57.934890 3318 raft_consensus.cc:374] T 43008abe74604666b330d78ec0f51c7c P a1a844ecd71145e4a362f4c42500fbcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:57.934942 3318 raft_consensus.cc:729] T 43008abe74604666b330d78ec0f51c7c P a1a844ecd71145e4a362f4c42500fbcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1a844ecd71145e4a362f4c42500fbcb, State: Initialized, Role: FOLLOWER
I20240503 01:58:57.935089 3318 consensus_queue.cc:260] T 43008abe74604666b330d78ec0f51c7c P a1a844ecd71145e4a362f4c42500fbcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } }
I20240503 01:58:57.935329 3318 ts_tablet_manager.cc:1398] T 43008abe74604666b330d78ec0f51c7c P a1a844ecd71145e4a362f4c42500fbcb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:58:57.935490 3318 tablet_bootstrap.cc:492] T 568a05972208414c90f7f4e2ea2c10a9 P a1a844ecd71145e4a362f4c42500fbcb: Bootstrap starting.
I20240503 01:58:57.935930 3319 tablet_bootstrap.cc:492] T c9787dcf6d674611a2f06c143a291bb5 P 327aa26a8e434a609ebcb382aa665fce: No bootstrap required, opened a new log
I20240503 01:58:57.936024 3319 ts_tablet_manager.cc:1367] T c9787dcf6d674611a2f06c143a291bb5 P 327aa26a8e434a609ebcb382aa665fce: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 01:58:57.936370 3318 tablet_bootstrap.cc:654] T 568a05972208414c90f7f4e2ea2c10a9 P a1a844ecd71145e4a362f4c42500fbcb: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:57.936619 3319 raft_consensus.cc:348] T c9787dcf6d674611a2f06c143a291bb5 P 327aa26a8e434a609ebcb382aa665fce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } }
I20240503 01:58:57.936766 3319 raft_consensus.cc:374] T c9787dcf6d674611a2f06c143a291bb5 P 327aa26a8e434a609ebcb382aa665fce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:57.936830 3319 raft_consensus.cc:729] T c9787dcf6d674611a2f06c143a291bb5 P 327aa26a8e434a609ebcb382aa665fce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 327aa26a8e434a609ebcb382aa665fce, State: Initialized, Role: FOLLOWER
I20240503 01:58:57.936954 3319 consensus_queue.cc:260] T c9787dcf6d674611a2f06c143a291bb5 P 327aa26a8e434a609ebcb382aa665fce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } }
I20240503 01:58:57.937273 3319 ts_tablet_manager.cc:1398] T c9787dcf6d674611a2f06c143a291bb5 P 327aa26a8e434a609ebcb382aa665fce: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:58:57.938095 3318 tablet_bootstrap.cc:492] T 568a05972208414c90f7f4e2ea2c10a9 P a1a844ecd71145e4a362f4c42500fbcb: No bootstrap required, opened a new log
I20240503 01:58:57.938187 3318 ts_tablet_manager.cc:1367] T 568a05972208414c90f7f4e2ea2c10a9 P a1a844ecd71145e4a362f4c42500fbcb: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 01:58:57.938529 3318 raft_consensus.cc:348] T 568a05972208414c90f7f4e2ea2c10a9 P a1a844ecd71145e4a362f4c42500fbcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } }
I20240503 01:58:57.938616 3318 raft_consensus.cc:374] T 568a05972208414c90f7f4e2ea2c10a9 P a1a844ecd71145e4a362f4c42500fbcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:57.938650 3318 raft_consensus.cc:729] T 568a05972208414c90f7f4e2ea2c10a9 P a1a844ecd71145e4a362f4c42500fbcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1a844ecd71145e4a362f4c42500fbcb, State: Initialized, Role: FOLLOWER
I20240503 01:58:57.938794 3318 consensus_queue.cc:260] T 568a05972208414c90f7f4e2ea2c10a9 P a1a844ecd71145e4a362f4c42500fbcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } }
I20240503 01:58:57.939043 3318 ts_tablet_manager.cc:1398] T 568a05972208414c90f7f4e2ea2c10a9 P a1a844ecd71145e4a362f4c42500fbcb: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:58:57.983639 3292 raft_consensus.cc:482] T 568a05972208414c90f7f4e2ea2c10a9 P b40a9a7b30d34a6db007ed8febd8a759 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:58:57.983762 3292 raft_consensus.cc:504] T 568a05972208414c90f7f4e2ea2c10a9 P b40a9a7b30d34a6db007ed8febd8a759 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } }
I20240503 01:58:57.984333 3292 leader_election.cc:288] T 568a05972208414c90f7f4e2ea2c10a9 P b40a9a7b30d34a6db007ed8febd8a759 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a1a844ecd71145e4a362f4c42500fbcb (127.0.99.130:46033), 327aa26a8e434a609ebcb382aa665fce (127.0.99.133:38853), b63928177fa54227a5ff9929265ab9dd (127.0.99.129:36425), 52c196c15e3348b985c689652bc49918 (127.0.99.132:42887)
I20240503 01:58:57.985044 2400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568a05972208414c90f7f4e2ea2c10a9" candidate_uuid: "b40a9a7b30d34a6db007ed8febd8a759" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b63928177fa54227a5ff9929265ab9dd" is_pre_election: true
I20240503 01:58:57.985252 2400 raft_consensus.cc:2454] T 568a05972208414c90f7f4e2ea2c10a9 P b63928177fa54227a5ff9929265ab9dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b40a9a7b30d34a6db007ed8febd8a759 in term 0.
I20240503 01:58:57.985246 2793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568a05972208414c90f7f4e2ea2c10a9" candidate_uuid: "b40a9a7b30d34a6db007ed8febd8a759" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52c196c15e3348b985c689652bc49918" is_pre_election: true
I20240503 01:58:57.985539 2793 raft_consensus.cc:2454] T 568a05972208414c90f7f4e2ea2c10a9 P 52c196c15e3348b985c689652bc49918 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b40a9a7b30d34a6db007ed8febd8a759 in term 0.
I20240503 01:58:57.985816 2598 leader_election.cc:302] T 568a05972208414c90f7f4e2ea2c10a9 P b40a9a7b30d34a6db007ed8febd8a759 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 52c196c15e3348b985c689652bc49918, b40a9a7b30d34a6db007ed8febd8a759, b63928177fa54227a5ff9929265ab9dd; no voters:
I20240503 01:58:57.985980 3292 raft_consensus.cc:2781] T 568a05972208414c90f7f4e2ea2c10a9 P b40a9a7b30d34a6db007ed8febd8a759 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:58:57.986023 3292 raft_consensus.cc:482] T 568a05972208414c90f7f4e2ea2c10a9 P b40a9a7b30d34a6db007ed8febd8a759 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:58:57.986056 3292 raft_consensus.cc:3037] T 568a05972208414c90f7f4e2ea2c10a9 P b40a9a7b30d34a6db007ed8febd8a759 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:57.987174 3292 raft_consensus.cc:504] T 568a05972208414c90f7f4e2ea2c10a9 P b40a9a7b30d34a6db007ed8febd8a759 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } }
I20240503 01:58:57.987671 3292 leader_election.cc:288] T 568a05972208414c90f7f4e2ea2c10a9 P b40a9a7b30d34a6db007ed8febd8a759 [CANDIDATE]: Term 1 election: Requested vote from peers a1a844ecd71145e4a362f4c42500fbcb (127.0.99.130:46033), 327aa26a8e434a609ebcb382aa665fce (127.0.99.133:38853), b63928177fa54227a5ff9929265ab9dd (127.0.99.129:36425), 52c196c15e3348b985c689652bc49918 (127.0.99.132:42887)
I20240503 01:58:57.987995 2793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568a05972208414c90f7f4e2ea2c10a9" candidate_uuid: "b40a9a7b30d34a6db007ed8febd8a759" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52c196c15e3348b985c689652bc49918"
I20240503 01:58:57.988003 2400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568a05972208414c90f7f4e2ea2c10a9" candidate_uuid: "b40a9a7b30d34a6db007ed8febd8a759" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b63928177fa54227a5ff9929265ab9dd"
I20240503 01:58:57.988094 2793 raft_consensus.cc:3037] T 568a05972208414c90f7f4e2ea2c10a9 P 52c196c15e3348b985c689652bc49918 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:57.988094 2400 raft_consensus.cc:3037] T 568a05972208414c90f7f4e2ea2c10a9 P b63928177fa54227a5ff9929265ab9dd [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:57.989039 2793 raft_consensus.cc:2454] T 568a05972208414c90f7f4e2ea2c10a9 P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b40a9a7b30d34a6db007ed8febd8a759 in term 1.
I20240503 01:58:57.989039 2400 raft_consensus.cc:2454] T 568a05972208414c90f7f4e2ea2c10a9 P b63928177fa54227a5ff9929265ab9dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b40a9a7b30d34a6db007ed8febd8a759 in term 1.
I20240503 01:58:57.989363 2598 leader_election.cc:302] T 568a05972208414c90f7f4e2ea2c10a9 P b40a9a7b30d34a6db007ed8febd8a759 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 52c196c15e3348b985c689652bc49918, b40a9a7b30d34a6db007ed8febd8a759, b63928177fa54227a5ff9929265ab9dd; no voters:
I20240503 01:58:57.989449 2923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568a05972208414c90f7f4e2ea2c10a9" candidate_uuid: "b40a9a7b30d34a6db007ed8febd8a759" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "327aa26a8e434a609ebcb382aa665fce"
I20240503 01:58:57.989482 2924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568a05972208414c90f7f4e2ea2c10a9" candidate_uuid: "b40a9a7b30d34a6db007ed8febd8a759" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "327aa26a8e434a609ebcb382aa665fce" is_pre_election: true
I20240503 01:58:57.989601 3292 raft_consensus.cc:2781] T 568a05972208414c90f7f4e2ea2c10a9 P b40a9a7b30d34a6db007ed8febd8a759 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:58:57.989681 3292 raft_consensus.cc:686] T 568a05972208414c90f7f4e2ea2c10a9 P b40a9a7b30d34a6db007ed8febd8a759 [term 1 LEADER]: Becoming Leader. State: Replica: b40a9a7b30d34a6db007ed8febd8a759, State: Running, Role: LEADER
I20240503 01:58:57.989758 2924 raft_consensus.cc:2454] T 568a05972208414c90f7f4e2ea2c10a9 P 327aa26a8e434a609ebcb382aa665fce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b40a9a7b30d34a6db007ed8febd8a759 in term 0.
I20240503 01:58:57.989820 3292 consensus_queue.cc:237] T 568a05972208414c90f7f4e2ea2c10a9 P b40a9a7b30d34a6db007ed8febd8a759 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } }
I20240503 01:58:57.990039 2531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568a05972208414c90f7f4e2ea2c10a9" candidate_uuid: "b40a9a7b30d34a6db007ed8febd8a759" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1a844ecd71145e4a362f4c42500fbcb" is_pre_election: true
I20240503 01:58:57.990228 2530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568a05972208414c90f7f4e2ea2c10a9" candidate_uuid: "b40a9a7b30d34a6db007ed8febd8a759" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1a844ecd71145e4a362f4c42500fbcb"
I20240503 01:58:57.990307 2531 raft_consensus.cc:2454] T 568a05972208414c90f7f4e2ea2c10a9 P a1a844ecd71145e4a362f4c42500fbcb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b40a9a7b30d34a6db007ed8febd8a759 in term 0.
I20240503 01:58:57.991650 2258 catalog_manager.cc:5484] T 568a05972208414c90f7f4e2ea2c10a9 P b40a9a7b30d34a6db007ed8febd8a759 reported cstate change: term changed from 0 to 1, leader changed from <none> to b40a9a7b30d34a6db007ed8febd8a759 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "b40a9a7b30d34a6db007ed8febd8a759" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } health_report { overall_health: UNKNOWN } } }
I20240503 01:58:58.013816 3329 raft_consensus.cc:482] T 43008abe74604666b330d78ec0f51c7c P a1a844ecd71145e4a362f4c42500fbcb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:58:58.013947 3329 raft_consensus.cc:504] T 43008abe74604666b330d78ec0f51c7c P a1a844ecd71145e4a362f4c42500fbcb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } }
I20240503 01:58:58.015112 3329 leader_election.cc:288] T 43008abe74604666b330d78ec0f51c7c P a1a844ecd71145e4a362f4c42500fbcb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6568f69a78c44bb985162d2b7d0a9376 (127.0.99.134:43603), 327aa26a8e434a609ebcb382aa665fce (127.0.99.133:38853), b63928177fa54227a5ff9929265ab9dd (127.0.99.129:36425), b40a9a7b30d34a6db007ed8febd8a759 (127.0.99.131:34079)
I20240503 01:58:58.020323 2923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43008abe74604666b330d78ec0f51c7c" candidate_uuid: "a1a844ecd71145e4a362f4c42500fbcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "327aa26a8e434a609ebcb382aa665fce" is_pre_election: true
I20240503 01:58:58.020341 3055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43008abe74604666b330d78ec0f51c7c" candidate_uuid: "a1a844ecd71145e4a362f4c42500fbcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6568f69a78c44bb985162d2b7d0a9376" is_pre_election: true
I20240503 01:58:58.020439 2662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43008abe74604666b330d78ec0f51c7c" candidate_uuid: "a1a844ecd71145e4a362f4c42500fbcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b40a9a7b30d34a6db007ed8febd8a759" is_pre_election: true
I20240503 01:58:58.020484 2923 raft_consensus.cc:2454] T 43008abe74604666b330d78ec0f51c7c P 327aa26a8e434a609ebcb382aa665fce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1a844ecd71145e4a362f4c42500fbcb in term 0.
I20240503 01:58:58.020495 3055 raft_consensus.cc:2454] T 43008abe74604666b330d78ec0f51c7c P 6568f69a78c44bb985162d2b7d0a9376 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1a844ecd71145e4a362f4c42500fbcb in term 0.
I20240503 01:58:58.020661 2662 raft_consensus.cc:2454] T 43008abe74604666b330d78ec0f51c7c P b40a9a7b30d34a6db007ed8febd8a759 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1a844ecd71145e4a362f4c42500fbcb in term 0.
I20240503 01:58:58.021047 2465 leader_election.cc:302] T 43008abe74604666b330d78ec0f51c7c P a1a844ecd71145e4a362f4c42500fbcb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 327aa26a8e434a609ebcb382aa665fce, 6568f69a78c44bb985162d2b7d0a9376, a1a844ecd71145e4a362f4c42500fbcb; no voters:
I20240503 01:58:58.021338 3329 raft_consensus.cc:2781] T 43008abe74604666b330d78ec0f51c7c P a1a844ecd71145e4a362f4c42500fbcb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:58:58.021405 3329 raft_consensus.cc:482] T 43008abe74604666b330d78ec0f51c7c P a1a844ecd71145e4a362f4c42500fbcb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:58:58.021466 3329 raft_consensus.cc:3037] T 43008abe74604666b330d78ec0f51c7c P a1a844ecd71145e4a362f4c42500fbcb [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:58.021437 2400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43008abe74604666b330d78ec0f51c7c" candidate_uuid: "a1a844ecd71145e4a362f4c42500fbcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b63928177fa54227a5ff9929265ab9dd" is_pre_election: true
I20240503 01:58:58.021572 2400 raft_consensus.cc:2454] T 43008abe74604666b330d78ec0f51c7c P b63928177fa54227a5ff9929265ab9dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1a844ecd71145e4a362f4c42500fbcb in term 0.
I20240503 01:58:58.022439 3329 raft_consensus.cc:504] T 43008abe74604666b330d78ec0f51c7c P a1a844ecd71145e4a362f4c42500fbcb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } }
I20240503 01:58:58.022895 3329 leader_election.cc:288] T 43008abe74604666b330d78ec0f51c7c P a1a844ecd71145e4a362f4c42500fbcb [CANDIDATE]: Term 1 election: Requested vote from peers 6568f69a78c44bb985162d2b7d0a9376 (127.0.99.134:43603), 327aa26a8e434a609ebcb382aa665fce (127.0.99.133:38853), b63928177fa54227a5ff9929265ab9dd (127.0.99.129:36425), b40a9a7b30d34a6db007ed8febd8a759 (127.0.99.131:34079)
I20240503 01:58:58.023083 3055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43008abe74604666b330d78ec0f51c7c" candidate_uuid: "a1a844ecd71145e4a362f4c42500fbcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6568f69a78c44bb985162d2b7d0a9376"
I20240503 01:58:58.023137 2923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43008abe74604666b330d78ec0f51c7c" candidate_uuid: "a1a844ecd71145e4a362f4c42500fbcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "327aa26a8e434a609ebcb382aa665fce"
I20240503 01:58:58.023209 2400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43008abe74604666b330d78ec0f51c7c" candidate_uuid: "a1a844ecd71145e4a362f4c42500fbcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b63928177fa54227a5ff9929265ab9dd"
I20240503 01:58:58.023231 3055 raft_consensus.cc:3037] T 43008abe74604666b330d78ec0f51c7c P 6568f69a78c44bb985162d2b7d0a9376 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:58.023262 2923 raft_consensus.cc:3037] T 43008abe74604666b330d78ec0f51c7c P 327aa26a8e434a609ebcb382aa665fce [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:58.023280 2662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43008abe74604666b330d78ec0f51c7c" candidate_uuid: "a1a844ecd71145e4a362f4c42500fbcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b40a9a7b30d34a6db007ed8febd8a759"
I20240503 01:58:58.023308 2400 raft_consensus.cc:3037] T 43008abe74604666b330d78ec0f51c7c P b63928177fa54227a5ff9929265ab9dd [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:58.023376 2662 raft_consensus.cc:3037] T 43008abe74604666b330d78ec0f51c7c P b40a9a7b30d34a6db007ed8febd8a759 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:58.024252 2400 raft_consensus.cc:2454] T 43008abe74604666b330d78ec0f51c7c P b63928177fa54227a5ff9929265ab9dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1a844ecd71145e4a362f4c42500fbcb in term 1.
I20240503 01:58:58.024276 3055 raft_consensus.cc:2454] T 43008abe74604666b330d78ec0f51c7c P 6568f69a78c44bb985162d2b7d0a9376 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1a844ecd71145e4a362f4c42500fbcb in term 1.
I20240503 01:58:58.024307 2923 raft_consensus.cc:2454] T 43008abe74604666b330d78ec0f51c7c P 327aa26a8e434a609ebcb382aa665fce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1a844ecd71145e4a362f4c42500fbcb in term 1.
I20240503 01:58:58.024346 2662 raft_consensus.cc:2454] T 43008abe74604666b330d78ec0f51c7c P b40a9a7b30d34a6db007ed8febd8a759 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1a844ecd71145e4a362f4c42500fbcb in term 1.
I20240503 01:58:58.024616 2465 leader_election.cc:302] T 43008abe74604666b330d78ec0f51c7c P a1a844ecd71145e4a362f4c42500fbcb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 327aa26a8e434a609ebcb382aa665fce, 6568f69a78c44bb985162d2b7d0a9376, a1a844ecd71145e4a362f4c42500fbcb; no voters:
I20240503 01:58:58.024824 3329 raft_consensus.cc:2781] T 43008abe74604666b330d78ec0f51c7c P a1a844ecd71145e4a362f4c42500fbcb [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:58:58.025035 3329 raft_consensus.cc:686] T 43008abe74604666b330d78ec0f51c7c P a1a844ecd71145e4a362f4c42500fbcb [term 1 LEADER]: Becoming Leader. State: Replica: a1a844ecd71145e4a362f4c42500fbcb, State: Running, Role: LEADER
I20240503 01:58:58.025367 3329 consensus_queue.cc:237] T 43008abe74604666b330d78ec0f51c7c P a1a844ecd71145e4a362f4c42500fbcb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } }
I20240503 01:58:58.028366 2258 catalog_manager.cc:5484] T 43008abe74604666b330d78ec0f51c7c P a1a844ecd71145e4a362f4c42500fbcb reported cstate change: term changed from 0 to 1, leader changed from <none> to a1a844ecd71145e4a362f4c42500fbcb (127.0.99.130). New cstate: current_term: 1 leader_uuid: "a1a844ecd71145e4a362f4c42500fbcb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } health_report { overall_health: UNKNOWN } } }
W20240503 01:58:58.163095 2578 tablet.cc:2369] T 43008abe74604666b330d78ec0f51c7c P a1a844ecd71145e4a362f4c42500fbcb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 01:58:58.178740 2971 tablet.cc:2369] T 43008abe74604666b330d78ec0f51c7c P 327aa26a8e434a609ebcb382aa665fce: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 01:58:58.189918 3316 raft_consensus.cc:482] T c9787dcf6d674611a2f06c143a291bb5 P 6568f69a78c44bb985162d2b7d0a9376 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:58:58.190058 3316 raft_consensus.cc:504] T c9787dcf6d674611a2f06c143a291bb5 P 6568f69a78c44bb985162d2b7d0a9376 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } }
I20240503 01:58:58.191239 3316 leader_election.cc:288] T c9787dcf6d674611a2f06c143a291bb5 P 6568f69a78c44bb985162d2b7d0a9376 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 327aa26a8e434a609ebcb382aa665fce (127.0.99.133:38853), a1a844ecd71145e4a362f4c42500fbcb (127.0.99.130:46033), 52c196c15e3348b985c689652bc49918 (127.0.99.132:42887), b63928177fa54227a5ff9929265ab9dd (127.0.99.129:36425)
I20240503 01:58:58.195786 2530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9787dcf6d674611a2f06c143a291bb5" candidate_uuid: "6568f69a78c44bb985162d2b7d0a9376" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1a844ecd71145e4a362f4c42500fbcb" is_pre_election: true
I20240503 01:58:58.195952 2530 raft_consensus.cc:2454] T c9787dcf6d674611a2f06c143a291bb5 P a1a844ecd71145e4a362f4c42500fbcb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6568f69a78c44bb985162d2b7d0a9376 in term 0.
I20240503 01:58:58.196178 2923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9787dcf6d674611a2f06c143a291bb5" candidate_uuid: "6568f69a78c44bb985162d2b7d0a9376" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "327aa26a8e434a609ebcb382aa665fce" is_pre_election: true
I20240503 01:58:58.196307 2923 raft_consensus.cc:2454] T c9787dcf6d674611a2f06c143a291bb5 P 327aa26a8e434a609ebcb382aa665fce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6568f69a78c44bb985162d2b7d0a9376 in term 0.
I20240503 01:58:58.196600 2989 leader_election.cc:302] T c9787dcf6d674611a2f06c143a291bb5 P 6568f69a78c44bb985162d2b7d0a9376 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 327aa26a8e434a609ebcb382aa665fce, 6568f69a78c44bb985162d2b7d0a9376, a1a844ecd71145e4a362f4c42500fbcb; no voters:
I20240503 01:58:58.196859 3316 raft_consensus.cc:2781] T c9787dcf6d674611a2f06c143a291bb5 P 6568f69a78c44bb985162d2b7d0a9376 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:58:58.196928 3316 raft_consensus.cc:482] T c9787dcf6d674611a2f06c143a291bb5 P 6568f69a78c44bb985162d2b7d0a9376 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:58:58.196888 2400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9787dcf6d674611a2f06c143a291bb5" candidate_uuid: "6568f69a78c44bb985162d2b7d0a9376" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b63928177fa54227a5ff9929265ab9dd" is_pre_election: true
I20240503 01:58:58.196966 3316 raft_consensus.cc:3037] T c9787dcf6d674611a2f06c143a291bb5 P 6568f69a78c44bb985162d2b7d0a9376 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:58.197019 2400 raft_consensus.cc:2454] T c9787dcf6d674611a2f06c143a291bb5 P b63928177fa54227a5ff9929265ab9dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6568f69a78c44bb985162d2b7d0a9376 in term 0.
I20240503 01:58:58.197857 2793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9787dcf6d674611a2f06c143a291bb5" candidate_uuid: "6568f69a78c44bb985162d2b7d0a9376" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52c196c15e3348b985c689652bc49918" is_pre_election: true
I20240503 01:58:58.198006 2793 raft_consensus.cc:2454] T c9787dcf6d674611a2f06c143a291bb5 P 52c196c15e3348b985c689652bc49918 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6568f69a78c44bb985162d2b7d0a9376 in term 0.
I20240503 01:58:58.197968 3316 raft_consensus.cc:504] T c9787dcf6d674611a2f06c143a291bb5 P 6568f69a78c44bb985162d2b7d0a9376 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } }
I20240503 01:58:58.198525 3316 leader_election.cc:288] T c9787dcf6d674611a2f06c143a291bb5 P 6568f69a78c44bb985162d2b7d0a9376 [CANDIDATE]: Term 1 election: Requested vote from peers 327aa26a8e434a609ebcb382aa665fce (127.0.99.133:38853), a1a844ecd71145e4a362f4c42500fbcb (127.0.99.130:46033), 52c196c15e3348b985c689652bc49918 (127.0.99.132:42887), b63928177fa54227a5ff9929265ab9dd (127.0.99.129:36425)
I20240503 01:58:58.198789 2923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9787dcf6d674611a2f06c143a291bb5" candidate_uuid: "6568f69a78c44bb985162d2b7d0a9376" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "327aa26a8e434a609ebcb382aa665fce"
I20240503 01:58:58.198848 2530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9787dcf6d674611a2f06c143a291bb5" candidate_uuid: "6568f69a78c44bb985162d2b7d0a9376" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1a844ecd71145e4a362f4c42500fbcb"
I20240503 01:58:58.198912 2923 raft_consensus.cc:3037] T c9787dcf6d674611a2f06c143a291bb5 P 327aa26a8e434a609ebcb382aa665fce [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:58.198940 2530 raft_consensus.cc:3037] T c9787dcf6d674611a2f06c143a291bb5 P a1a844ecd71145e4a362f4c42500fbcb [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:58.198954 2400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9787dcf6d674611a2f06c143a291bb5" candidate_uuid: "6568f69a78c44bb985162d2b7d0a9376" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b63928177fa54227a5ff9929265ab9dd"
I20240503 01:58:58.199050 2400 raft_consensus.cc:3037] T c9787dcf6d674611a2f06c143a291bb5 P b63928177fa54227a5ff9929265ab9dd [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:58.199760 2400 raft_consensus.cc:2454] T c9787dcf6d674611a2f06c143a291bb5 P b63928177fa54227a5ff9929265ab9dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6568f69a78c44bb985162d2b7d0a9376 in term 1.
I20240503 01:58:58.199952 2923 raft_consensus.cc:2454] T c9787dcf6d674611a2f06c143a291bb5 P 327aa26a8e434a609ebcb382aa665fce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6568f69a78c44bb985162d2b7d0a9376 in term 1.
I20240503 01:58:58.199952 2530 raft_consensus.cc:2454] T c9787dcf6d674611a2f06c143a291bb5 P a1a844ecd71145e4a362f4c42500fbcb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6568f69a78c44bb985162d2b7d0a9376 in term 1.
I20240503 01:58:58.199972 2793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9787dcf6d674611a2f06c143a291bb5" candidate_uuid: "6568f69a78c44bb985162d2b7d0a9376" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52c196c15e3348b985c689652bc49918"
I20240503 01:58:58.200071 2793 raft_consensus.cc:3037] T c9787dcf6d674611a2f06c143a291bb5 P 52c196c15e3348b985c689652bc49918 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:58.200919 2793 raft_consensus.cc:2454] T c9787dcf6d674611a2f06c143a291bb5 P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6568f69a78c44bb985162d2b7d0a9376 in term 1.
I20240503 01:58:58.201149 2991 leader_election.cc:302] T c9787dcf6d674611a2f06c143a291bb5 P 6568f69a78c44bb985162d2b7d0a9376 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 327aa26a8e434a609ebcb382aa665fce, 6568f69a78c44bb985162d2b7d0a9376, a1a844ecd71145e4a362f4c42500fbcb; no voters:
I20240503 01:58:58.201315 3316 raft_consensus.cc:2781] T c9787dcf6d674611a2f06c143a291bb5 P 6568f69a78c44bb985162d2b7d0a9376 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:58:58.201557 3316 raft_consensus.cc:686] T c9787dcf6d674611a2f06c143a291bb5 P 6568f69a78c44bb985162d2b7d0a9376 [term 1 LEADER]: Becoming Leader. State: Replica: 6568f69a78c44bb985162d2b7d0a9376, State: Running, Role: LEADER
I20240503 01:58:58.201843 3316 consensus_queue.cc:237] T c9787dcf6d674611a2f06c143a291bb5 P 6568f69a78c44bb985162d2b7d0a9376 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } }
I20240503 01:58:58.204900 2258 catalog_manager.cc:5484] T c9787dcf6d674611a2f06c143a291bb5 P 6568f69a78c44bb985162d2b7d0a9376 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6568f69a78c44bb985162d2b7d0a9376 (127.0.99.134). New cstate: current_term: 1 leader_uuid: "6568f69a78c44bb985162d2b7d0a9376" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } health_report { overall_health: UNKNOWN } } }
W20240503 01:58:58.374400 3351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:58:58.374629 3351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:58:58.377566 3351 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 01:58:58.386087 3292 consensus_queue.cc:1035] T 568a05972208414c90f7f4e2ea2c10a9 P b40a9a7b30d34a6db007ed8febd8a759 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_0'
I20240503 01:58:58.419742 3292 consensus_queue.cc:1035] T 568a05972208414c90f7f4e2ea2c10a9 P b40a9a7b30d34a6db007ed8febd8a759 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:58.424597 2923 raft_consensus.cc:3037] T 568a05972208414c90f7f4e2ea2c10a9 P 327aa26a8e434a609ebcb382aa665fce [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:58.427758 2530 raft_consensus.cc:3037] T 568a05972208414c90f7f4e2ea2c10a9 P a1a844ecd71145e4a362f4c42500fbcb [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:58.432742 3297 consensus_queue.cc:1035] T 568a05972208414c90f7f4e2ea2c10a9 P b40a9a7b30d34a6db007ed8febd8a759 [LEADER]: Connected to new peer: Peer: permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:58.433450 3297 consensus_queue.cc:1035] T 568a05972208414c90f7f4e2ea2c10a9 P b40a9a7b30d34a6db007ed8febd8a759 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:58.435523 3342 consensus_queue.cc:1035] T 43008abe74604666b330d78ec0f51c7c P a1a844ecd71145e4a362f4c42500fbcb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:58.488775 3342 consensus_queue.cc:1035] T 43008abe74604666b330d78ec0f51c7c P a1a844ecd71145e4a362f4c42500fbcb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:58.494195 3329 consensus_queue.cc:1035] T 43008abe74604666b330d78ec0f51c7c P a1a844ecd71145e4a362f4c42500fbcb [LEADER]: Connected to new peer: Peer: permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:58.494433 3329 consensus_queue.cc:1035] T 43008abe74604666b330d78ec0f51c7c P a1a844ecd71145e4a362f4c42500fbcb [LEADER]: Connected to new peer: Peer: permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:58.518438 2792 raft_consensus.cc:1261] T c9787dcf6d674611a2f06c143a291bb5 P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Refusing update from remote peer 6568f69a78c44bb985162d2b7d0a9376: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:58:58.519075 3316 consensus_queue.cc:1035] T c9787dcf6d674611a2f06c143a291bb5 P 6568f69a78c44bb985162d2b7d0a9376 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:58.525668 2530 raft_consensus.cc:1261] T c9787dcf6d674611a2f06c143a291bb5 P a1a844ecd71145e4a362f4c42500fbcb [term 1 FOLLOWER]: Refusing update from remote peer 6568f69a78c44bb985162d2b7d0a9376: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:58:58.528597 2398 raft_consensus.cc:1261] T c9787dcf6d674611a2f06c143a291bb5 P b63928177fa54227a5ff9929265ab9dd [term 1 FOLLOWER]: Refusing update from remote peer 6568f69a78c44bb985162d2b7d0a9376: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:58:58.528784 2924 raft_consensus.cc:1261] T c9787dcf6d674611a2f06c143a291bb5 P 327aa26a8e434a609ebcb382aa665fce [term 1 FOLLOWER]: Refusing update from remote peer 6568f69a78c44bb985162d2b7d0a9376: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:58:58.529465 3316 consensus_queue.cc:1035] T c9787dcf6d674611a2f06c143a291bb5 P 6568f69a78c44bb985162d2b7d0a9376 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:58.529858 3316 consensus_queue.cc:1035] T c9787dcf6d674611a2f06c143a291bb5 P 6568f69a78c44bb985162d2b7d0a9376 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:58.529994 3349 consensus_queue.cc:1035] T c9787dcf6d674611a2f06c143a291bb5 P 6568f69a78c44bb985162d2b7d0a9376 [LEADER]: Connected to new peer: Peer: permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:58.563226 3379 mvcc.cc:205] Tried to move back new op lower bound from 7023417501762564096 to 7023417500479406080. Current Snapshot: MvccSnapshot[applied={T|T < 7023417501762564096}]
I20240503 01:58:58.566890 3385 mvcc.cc:205] Tried to move back new op lower bound from 7023417501762564096 to 7023417500479406080. Current Snapshot: MvccSnapshot[applied={T|T < 7023417501762564096}]
I20240503 01:58:58.571422 3382 mvcc.cc:205] Tried to move back new op lower bound from 7023417501762564096 to 7023417500479406080. Current Snapshot: MvccSnapshot[applied={T|T < 7023417501762564096}]
I20240503 01:58:58.575368 3404 mvcc.cc:205] Tried to move back new op lower bound from 7023417501762564096 to 7023417500479406080. Current Snapshot: MvccSnapshot[applied={T|T < 7023417501762564096}]
I20240503 01:58:58.581988 3380 mvcc.cc:205] Tried to move back new op lower bound from 7023417501762564096 to 7023417500479406080. Current Snapshot: MvccSnapshot[applied={T|T < 7023417501762564096}]
INSERT report
rows total: 2000
time total: 215.623 ms
time per row: 0.107811 ms
I20240503 01:58:58.645155 2256 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53804:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 5
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:58:58.650923 2641 tablet_service.cc:1467] Processing CreateTablet for tablet a6e17a2c4ba0482e96d04eb9d3ba854a (DEFAULT_TABLE table=rebalance_test_table_1 [id=d0846ac337c64bf1ac218add4c796fff]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:58.650923 2511 tablet_service.cc:1467] Processing CreateTablet for tablet a6e17a2c4ba0482e96d04eb9d3ba854a (DEFAULT_TABLE table=rebalance_test_table_1 [id=d0846ac337c64bf1ac218add4c796fff]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:58.650938 2904 tablet_service.cc:1467] Processing CreateTablet for tablet a6e17a2c4ba0482e96d04eb9d3ba854a (DEFAULT_TABLE table=rebalance_test_table_1 [id=d0846ac337c64bf1ac218add4c796fff]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:58.651242 2904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6e17a2c4ba0482e96d04eb9d3ba854a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:58.651237 2641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6e17a2c4ba0482e96d04eb9d3ba854a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:58.651237 2511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6e17a2c4ba0482e96d04eb9d3ba854a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:58.653514 3440 tablet_bootstrap.cc:492] T a6e17a2c4ba0482e96d04eb9d3ba854a P b40a9a7b30d34a6db007ed8febd8a759: Bootstrap starting.
I20240503 01:58:58.653788 2772 tablet_service.cc:1467] Processing CreateTablet for tablet a6e17a2c4ba0482e96d04eb9d3ba854a (DEFAULT_TABLE table=rebalance_test_table_1 [id=d0846ac337c64bf1ac218add4c796fff]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:58.654069 2772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6e17a2c4ba0482e96d04eb9d3ba854a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:58.654103 3034 tablet_service.cc:1467] Processing CreateTablet for tablet a6e17a2c4ba0482e96d04eb9d3ba854a (DEFAULT_TABLE table=rebalance_test_table_1 [id=d0846ac337c64bf1ac218add4c796fff]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:58.654250 2902 tablet_service.cc:1467] Processing CreateTablet for tablet c10343409a2a423eb0b21759bfc9ba61 (DEFAULT_TABLE table=rebalance_test_table_1 [id=d0846ac337c64bf1ac218add4c796fff]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:58.654398 3034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6e17a2c4ba0482e96d04eb9d3ba854a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:58.654486 3440 tablet_bootstrap.cc:654] T a6e17a2c4ba0482e96d04eb9d3ba854a P b40a9a7b30d34a6db007ed8febd8a759: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:58.654490 2902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c10343409a2a423eb0b21759bfc9ba61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:58.655534 3440 tablet_bootstrap.cc:492] T a6e17a2c4ba0482e96d04eb9d3ba854a P b40a9a7b30d34a6db007ed8febd8a759: No bootstrap required, opened a new log
I20240503 01:58:58.655622 3440 ts_tablet_manager.cc:1367] T a6e17a2c4ba0482e96d04eb9d3ba854a P b40a9a7b30d34a6db007ed8febd8a759: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:58:58.656059 3440 raft_consensus.cc:348] T a6e17a2c4ba0482e96d04eb9d3ba854a P b40a9a7b30d34a6db007ed8febd8a759 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } }
I20240503 01:58:58.656180 3440 raft_consensus.cc:374] T a6e17a2c4ba0482e96d04eb9d3ba854a P b40a9a7b30d34a6db007ed8febd8a759 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:58.656234 3440 raft_consensus.cc:729] T a6e17a2c4ba0482e96d04eb9d3ba854a P b40a9a7b30d34a6db007ed8febd8a759 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b40a9a7b30d34a6db007ed8febd8a759, State: Initialized, Role: FOLLOWER
I20240503 01:58:58.656365 3440 consensus_queue.cc:260] T a6e17a2c4ba0482e96d04eb9d3ba854a P b40a9a7b30d34a6db007ed8febd8a759 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } }
I20240503 01:58:58.656718 3440 ts_tablet_manager.cc:1398] T a6e17a2c4ba0482e96d04eb9d3ba854a P b40a9a7b30d34a6db007ed8febd8a759: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:58:58.656847 3035 tablet_service.cc:1467] Processing CreateTablet for tablet 9d84cbdf02c14873b8838be296977aaf (DEFAULT_TABLE table=rebalance_test_table_1 [id=d0846ac337c64bf1ac218add4c796fff]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:58.657086 3035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d84cbdf02c14873b8838be296977aaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:58.657313 2641 tablet_service.cc:1467] Processing CreateTablet for tablet 9d84cbdf02c14873b8838be296977aaf (DEFAULT_TABLE table=rebalance_test_table_1 [id=d0846ac337c64bf1ac218add4c796fff]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:58.657404 2642 tablet_service.cc:1467] Processing CreateTablet for tablet c10343409a2a423eb0b21759bfc9ba61 (DEFAULT_TABLE table=rebalance_test_table_1 [id=d0846ac337c64bf1ac218add4c796fff]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:58.657526 2641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d84cbdf02c14873b8838be296977aaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:58.657652 2642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c10343409a2a423eb0b21759bfc9ba61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:58.657754 2773 tablet_service.cc:1467] Processing CreateTablet for tablet c10343409a2a423eb0b21759bfc9ba61 (DEFAULT_TABLE table=rebalance_test_table_1 [id=d0846ac337c64bf1ac218add4c796fff]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:58.657974 2773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c10343409a2a423eb0b21759bfc9ba61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:58.659168 2378 tablet_service.cc:1467] Processing CreateTablet for tablet c10343409a2a423eb0b21759bfc9ba61 (DEFAULT_TABLE table=rebalance_test_table_1 [id=d0846ac337c64bf1ac218add4c796fff]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:58.659459 2378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c10343409a2a423eb0b21759bfc9ba61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:58.660112 2510 tablet_service.cc:1467] Processing CreateTablet for tablet c10343409a2a423eb0b21759bfc9ba61 (DEFAULT_TABLE table=rebalance_test_table_1 [id=d0846ac337c64bf1ac218add4c796fff]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:58.660193 2772 tablet_service.cc:1467] Processing CreateTablet for tablet 9d84cbdf02c14873b8838be296977aaf (DEFAULT_TABLE table=rebalance_test_table_1 [id=d0846ac337c64bf1ac218add4c796fff]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:58.660359 2510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c10343409a2a423eb0b21759bfc9ba61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:58.660392 2772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d84cbdf02c14873b8838be296977aaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:58.661480 3445 tablet_bootstrap.cc:492] T c10343409a2a423eb0b21759bfc9ba61 P b63928177fa54227a5ff9929265ab9dd: Bootstrap starting.
I20240503 01:58:58.662055 3444 tablet_bootstrap.cc:492] T a6e17a2c4ba0482e96d04eb9d3ba854a P 6568f69a78c44bb985162d2b7d0a9376: Bootstrap starting.
I20240503 01:58:58.662376 3445 tablet_bootstrap.cc:654] T c10343409a2a423eb0b21759bfc9ba61 P b63928177fa54227a5ff9929265ab9dd: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:58.662783 3440 tablet_bootstrap.cc:492] T 9d84cbdf02c14873b8838be296977aaf P b40a9a7b30d34a6db007ed8febd8a759: Bootstrap starting.
I20240503 01:58:58.662927 3441 tablet_bootstrap.cc:492] T a6e17a2c4ba0482e96d04eb9d3ba854a P a1a844ecd71145e4a362f4c42500fbcb: Bootstrap starting.
I20240503 01:58:58.663091 3444 tablet_bootstrap.cc:654] T a6e17a2c4ba0482e96d04eb9d3ba854a P 6568f69a78c44bb985162d2b7d0a9376: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:58.663352 2904 tablet_service.cc:1467] Processing CreateTablet for tablet 9d84cbdf02c14873b8838be296977aaf (DEFAULT_TABLE table=rebalance_test_table_1 [id=d0846ac337c64bf1ac218add4c796fff]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:58.663630 2904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d84cbdf02c14873b8838be296977aaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:58.663712 3440 tablet_bootstrap.cc:654] T 9d84cbdf02c14873b8838be296977aaf P b40a9a7b30d34a6db007ed8febd8a759: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:58.664032 3441 tablet_bootstrap.cc:654] T a6e17a2c4ba0482e96d04eb9d3ba854a P a1a844ecd71145e4a362f4c42500fbcb: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:58.664796 3443 tablet_bootstrap.cc:492] T a6e17a2c4ba0482e96d04eb9d3ba854a P 52c196c15e3348b985c689652bc49918: Bootstrap starting.
I20240503 01:58:58.665439 3440 tablet_bootstrap.cc:492] T 9d84cbdf02c14873b8838be296977aaf P b40a9a7b30d34a6db007ed8febd8a759: No bootstrap required, opened a new log
I20240503 01:58:58.665532 3440 ts_tablet_manager.cc:1367] T 9d84cbdf02c14873b8838be296977aaf P b40a9a7b30d34a6db007ed8febd8a759: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:58:58.665709 3443 tablet_bootstrap.cc:654] T a6e17a2c4ba0482e96d04eb9d3ba854a P 52c196c15e3348b985c689652bc49918: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:58.665999 3440 raft_consensus.cc:348] T 9d84cbdf02c14873b8838be296977aaf P b40a9a7b30d34a6db007ed8febd8a759 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } }
I20240503 01:58:58.666115 3440 raft_consensus.cc:374] T 9d84cbdf02c14873b8838be296977aaf P b40a9a7b30d34a6db007ed8febd8a759 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:58.666146 3440 raft_consensus.cc:729] T 9d84cbdf02c14873b8838be296977aaf P b40a9a7b30d34a6db007ed8febd8a759 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b40a9a7b30d34a6db007ed8febd8a759, State: Initialized, Role: FOLLOWER
I20240503 01:58:58.666262 3440 consensus_queue.cc:260] T 9d84cbdf02c14873b8838be296977aaf P b40a9a7b30d34a6db007ed8febd8a759 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } }
I20240503 01:58:58.666568 3440 ts_tablet_manager.cc:1398] T 9d84cbdf02c14873b8838be296977aaf P b40a9a7b30d34a6db007ed8febd8a759: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:58:58.666733 3440 tablet_bootstrap.cc:492] T c10343409a2a423eb0b21759bfc9ba61 P b40a9a7b30d34a6db007ed8febd8a759: Bootstrap starting.
I20240503 01:58:58.667196 2511 tablet_service.cc:1467] Processing CreateTablet for tablet 9d84cbdf02c14873b8838be296977aaf (DEFAULT_TABLE table=rebalance_test_table_1 [id=d0846ac337c64bf1ac218add4c796fff]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:58.667372 3441 tablet_bootstrap.cc:492] T a6e17a2c4ba0482e96d04eb9d3ba854a P a1a844ecd71145e4a362f4c42500fbcb: No bootstrap required, opened a new log
I20240503 01:58:58.667410 2511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d84cbdf02c14873b8838be296977aaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:58.667446 3441 ts_tablet_manager.cc:1367] T a6e17a2c4ba0482e96d04eb9d3ba854a P a1a844ecd71145e4a362f4c42500fbcb: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 01:58:58.667603 3440 tablet_bootstrap.cc:654] T c10343409a2a423eb0b21759bfc9ba61 P b40a9a7b30d34a6db007ed8febd8a759: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:58.667846 3445 tablet_bootstrap.cc:492] T c10343409a2a423eb0b21759bfc9ba61 P b63928177fa54227a5ff9929265ab9dd: No bootstrap required, opened a new log
I20240503 01:58:58.667845 3441 raft_consensus.cc:348] T a6e17a2c4ba0482e96d04eb9d3ba854a P a1a844ecd71145e4a362f4c42500fbcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } }
I20240503 01:58:58.667932 3445 ts_tablet_manager.cc:1367] T c10343409a2a423eb0b21759bfc9ba61 P b63928177fa54227a5ff9929265ab9dd: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 01:58:58.667940 3441 raft_consensus.cc:374] T a6e17a2c4ba0482e96d04eb9d3ba854a P a1a844ecd71145e4a362f4c42500fbcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:58.667969 3441 raft_consensus.cc:729] T a6e17a2c4ba0482e96d04eb9d3ba854a P a1a844ecd71145e4a362f4c42500fbcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1a844ecd71145e4a362f4c42500fbcb, State: Initialized, Role: FOLLOWER
I20240503 01:58:58.668079 3441 consensus_queue.cc:260] T a6e17a2c4ba0482e96d04eb9d3ba854a P a1a844ecd71145e4a362f4c42500fbcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } }
I20240503 01:58:58.668357 3441 ts_tablet_manager.cc:1398] T a6e17a2c4ba0482e96d04eb9d3ba854a P a1a844ecd71145e4a362f4c42500fbcb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:58:58.668385 3445 raft_consensus.cc:348] T c10343409a2a423eb0b21759bfc9ba61 P b63928177fa54227a5ff9929265ab9dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } }
I20240503 01:58:58.668470 3445 raft_consensus.cc:374] T c10343409a2a423eb0b21759bfc9ba61 P b63928177fa54227a5ff9929265ab9dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:58.668490 3445 raft_consensus.cc:729] T c10343409a2a423eb0b21759bfc9ba61 P b63928177fa54227a5ff9929265ab9dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b63928177fa54227a5ff9929265ab9dd, State: Initialized, Role: FOLLOWER
I20240503 01:58:58.668494 3441 tablet_bootstrap.cc:492] T c10343409a2a423eb0b21759bfc9ba61 P a1a844ecd71145e4a362f4c42500fbcb: Bootstrap starting.
I20240503 01:58:58.668593 3445 consensus_queue.cc:260] T c10343409a2a423eb0b21759bfc9ba61 P b63928177fa54227a5ff9929265ab9dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } }
I20240503 01:58:58.669435 3441 tablet_bootstrap.cc:654] T c10343409a2a423eb0b21759bfc9ba61 P a1a844ecd71145e4a362f4c42500fbcb: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:58.670575 3445 ts_tablet_manager.cc:1398] T c10343409a2a423eb0b21759bfc9ba61 P b63928177fa54227a5ff9929265ab9dd: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 01:58:58.670660 3443 tablet_bootstrap.cc:492] T a6e17a2c4ba0482e96d04eb9d3ba854a P 52c196c15e3348b985c689652bc49918: No bootstrap required, opened a new log
I20240503 01:58:58.670765 3443 ts_tablet_manager.cc:1367] T a6e17a2c4ba0482e96d04eb9d3ba854a P 52c196c15e3348b985c689652bc49918: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 01:58:58.671237 3443 raft_consensus.cc:348] T a6e17a2c4ba0482e96d04eb9d3ba854a P 52c196c15e3348b985c689652bc49918 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } }
I20240503 01:58:58.671373 3443 raft_consensus.cc:374] T a6e17a2c4ba0482e96d04eb9d3ba854a P 52c196c15e3348b985c689652bc49918 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:58.671406 3443 raft_consensus.cc:729] T a6e17a2c4ba0482e96d04eb9d3ba854a P 52c196c15e3348b985c689652bc49918 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52c196c15e3348b985c689652bc49918, State: Initialized, Role: FOLLOWER
I20240503 01:58:58.671517 3443 consensus_queue.cc:260] T a6e17a2c4ba0482e96d04eb9d3ba854a P 52c196c15e3348b985c689652bc49918 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } }
I20240503 01:58:58.671787 3443 ts_tablet_manager.cc:1398] T a6e17a2c4ba0482e96d04eb9d3ba854a P 52c196c15e3348b985c689652bc49918: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:58:58.671969 3443 tablet_bootstrap.cc:492] T c10343409a2a423eb0b21759bfc9ba61 P 52c196c15e3348b985c689652bc49918: Bootstrap starting.
I20240503 01:58:58.672848 3443 tablet_bootstrap.cc:654] T c10343409a2a423eb0b21759bfc9ba61 P 52c196c15e3348b985c689652bc49918: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:58.673262 3440 tablet_bootstrap.cc:492] T c10343409a2a423eb0b21759bfc9ba61 P b40a9a7b30d34a6db007ed8febd8a759: No bootstrap required, opened a new log
I20240503 01:58:58.673345 3440 ts_tablet_manager.cc:1367] T c10343409a2a423eb0b21759bfc9ba61 P b40a9a7b30d34a6db007ed8febd8a759: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20240503 01:58:58.673827 3440 raft_consensus.cc:348] T c10343409a2a423eb0b21759bfc9ba61 P b40a9a7b30d34a6db007ed8febd8a759 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } }
I20240503 01:58:58.673962 3440 raft_consensus.cc:374] T c10343409a2a423eb0b21759bfc9ba61 P b40a9a7b30d34a6db007ed8febd8a759 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:58.673995 3440 raft_consensus.cc:729] T c10343409a2a423eb0b21759bfc9ba61 P b40a9a7b30d34a6db007ed8febd8a759 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b40a9a7b30d34a6db007ed8febd8a759, State: Initialized, Role: FOLLOWER
I20240503 01:58:58.674120 3440 consensus_queue.cc:260] T c10343409a2a423eb0b21759bfc9ba61 P b40a9a7b30d34a6db007ed8febd8a759 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } }
I20240503 01:58:58.674376 3440 ts_tablet_manager.cc:1398] T c10343409a2a423eb0b21759bfc9ba61 P b40a9a7b30d34a6db007ed8febd8a759: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:58:58.674527 3441 tablet_bootstrap.cc:492] T c10343409a2a423eb0b21759bfc9ba61 P a1a844ecd71145e4a362f4c42500fbcb: No bootstrap required, opened a new log
I20240503 01:58:58.674612 3441 ts_tablet_manager.cc:1367] T c10343409a2a423eb0b21759bfc9ba61 P a1a844ecd71145e4a362f4c42500fbcb: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 01:58:58.675045 3441 raft_consensus.cc:348] T c10343409a2a423eb0b21759bfc9ba61 P a1a844ecd71145e4a362f4c42500fbcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } }
I20240503 01:58:58.675153 3441 raft_consensus.cc:374] T c10343409a2a423eb0b21759bfc9ba61 P a1a844ecd71145e4a362f4c42500fbcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:58.675184 3441 raft_consensus.cc:729] T c10343409a2a423eb0b21759bfc9ba61 P a1a844ecd71145e4a362f4c42500fbcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1a844ecd71145e4a362f4c42500fbcb, State: Initialized, Role: FOLLOWER
I20240503 01:58:58.675187 3439 tablet_bootstrap.cc:492] T a6e17a2c4ba0482e96d04eb9d3ba854a P 327aa26a8e434a609ebcb382aa665fce: Bootstrap starting.
I20240503 01:58:58.675320 3441 consensus_queue.cc:260] T c10343409a2a423eb0b21759bfc9ba61 P a1a844ecd71145e4a362f4c42500fbcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } }
I20240503 01:58:58.675571 3441 ts_tablet_manager.cc:1398] T c10343409a2a423eb0b21759bfc9ba61 P a1a844ecd71145e4a362f4c42500fbcb: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:58:58.675720 3441 tablet_bootstrap.cc:492] T 9d84cbdf02c14873b8838be296977aaf P a1a844ecd71145e4a362f4c42500fbcb: Bootstrap starting.
I20240503 01:58:58.676546 3441 tablet_bootstrap.cc:654] T 9d84cbdf02c14873b8838be296977aaf P a1a844ecd71145e4a362f4c42500fbcb: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:58.676910 3444 tablet_bootstrap.cc:492] T a6e17a2c4ba0482e96d04eb9d3ba854a P 6568f69a78c44bb985162d2b7d0a9376: No bootstrap required, opened a new log
I20240503 01:58:58.677002 3444 ts_tablet_manager.cc:1367] T a6e17a2c4ba0482e96d04eb9d3ba854a P 6568f69a78c44bb985162d2b7d0a9376: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20240503 01:58:58.677510 3444 raft_consensus.cc:348] T a6e17a2c4ba0482e96d04eb9d3ba854a P 6568f69a78c44bb985162d2b7d0a9376 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } }
I20240503 01:58:58.677642 3444 raft_consensus.cc:374] T a6e17a2c4ba0482e96d04eb9d3ba854a P 6568f69a78c44bb985162d2b7d0a9376 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:58.677685 3444 raft_consensus.cc:729] T a6e17a2c4ba0482e96d04eb9d3ba854a P 6568f69a78c44bb985162d2b7d0a9376 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6568f69a78c44bb985162d2b7d0a9376, State: Initialized, Role: FOLLOWER
I20240503 01:58:58.677805 3444 consensus_queue.cc:260] T a6e17a2c4ba0482e96d04eb9d3ba854a P 6568f69a78c44bb985162d2b7d0a9376 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } }
I20240503 01:58:58.678102 3444 ts_tablet_manager.cc:1398] T a6e17a2c4ba0482e96d04eb9d3ba854a P 6568f69a78c44bb985162d2b7d0a9376: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:58:58.678118 3443 tablet_bootstrap.cc:492] T c10343409a2a423eb0b21759bfc9ba61 P 52c196c15e3348b985c689652bc49918: No bootstrap required, opened a new log
I20240503 01:58:58.678196 3443 ts_tablet_manager.cc:1367] T c10343409a2a423eb0b21759bfc9ba61 P 52c196c15e3348b985c689652bc49918: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240503 01:58:58.678261 3444 tablet_bootstrap.cc:492] T 9d84cbdf02c14873b8838be296977aaf P 6568f69a78c44bb985162d2b7d0a9376: Bootstrap starting.
I20240503 01:58:58.678699 3443 raft_consensus.cc:348] T c10343409a2a423eb0b21759bfc9ba61 P 52c196c15e3348b985c689652bc49918 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } }
I20240503 01:58:58.678843 3443 raft_consensus.cc:374] T c10343409a2a423eb0b21759bfc9ba61 P 52c196c15e3348b985c689652bc49918 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:58.678879 3443 raft_consensus.cc:729] T c10343409a2a423eb0b21759bfc9ba61 P 52c196c15e3348b985c689652bc49918 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52c196c15e3348b985c689652bc49918, State: Initialized, Role: FOLLOWER
I20240503 01:58:58.678973 3444 tablet_bootstrap.cc:654] T 9d84cbdf02c14873b8838be296977aaf P 6568f69a78c44bb985162d2b7d0a9376: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:58.678990 3443 consensus_queue.cc:260] T c10343409a2a423eb0b21759bfc9ba61 P 52c196c15e3348b985c689652bc49918 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } }
I20240503 01:58:58.679318 3443 ts_tablet_manager.cc:1398] T c10343409a2a423eb0b21759bfc9ba61 P 52c196c15e3348b985c689652bc49918: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:58:58.679355 3439 tablet_bootstrap.cc:654] T a6e17a2c4ba0482e96d04eb9d3ba854a P 327aa26a8e434a609ebcb382aa665fce: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:58.679499 3443 tablet_bootstrap.cc:492] T 9d84cbdf02c14873b8838be296977aaf P 52c196c15e3348b985c689652bc49918: Bootstrap starting.
I20240503 01:58:58.680437 3443 tablet_bootstrap.cc:654] T 9d84cbdf02c14873b8838be296977aaf P 52c196c15e3348b985c689652bc49918: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:58.680495 3439 tablet_bootstrap.cc:492] T a6e17a2c4ba0482e96d04eb9d3ba854a P 327aa26a8e434a609ebcb382aa665fce: No bootstrap required, opened a new log
I20240503 01:58:58.680586 3439 ts_tablet_manager.cc:1367] T a6e17a2c4ba0482e96d04eb9d3ba854a P 327aa26a8e434a609ebcb382aa665fce: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 01:58:58.681075 3439 raft_consensus.cc:348] T a6e17a2c4ba0482e96d04eb9d3ba854a P 327aa26a8e434a609ebcb382aa665fce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } }
I20240503 01:58:58.681206 3439 raft_consensus.cc:374] T a6e17a2c4ba0482e96d04eb9d3ba854a P 327aa26a8e434a609ebcb382aa665fce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:58.681257 3439 raft_consensus.cc:729] T a6e17a2c4ba0482e96d04eb9d3ba854a P 327aa26a8e434a609ebcb382aa665fce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 327aa26a8e434a609ebcb382aa665fce, State: Initialized, Role: FOLLOWER
I20240503 01:58:58.681396 3439 consensus_queue.cc:260] T a6e17a2c4ba0482e96d04eb9d3ba854a P 327aa26a8e434a609ebcb382aa665fce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } }
I20240503 01:58:58.681707 3441 tablet_bootstrap.cc:492] T 9d84cbdf02c14873b8838be296977aaf P a1a844ecd71145e4a362f4c42500fbcb: No bootstrap required, opened a new log
I20240503 01:58:58.681744 3439 ts_tablet_manager.cc:1398] T a6e17a2c4ba0482e96d04eb9d3ba854a P 327aa26a8e434a609ebcb382aa665fce: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:58:58.681785 3441 ts_tablet_manager.cc:1367] T 9d84cbdf02c14873b8838be296977aaf P a1a844ecd71145e4a362f4c42500fbcb: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 01:58:58.681843 3443 tablet_bootstrap.cc:492] T 9d84cbdf02c14873b8838be296977aaf P 52c196c15e3348b985c689652bc49918: No bootstrap required, opened a new log
I20240503 01:58:58.681913 3439 tablet_bootstrap.cc:492] T c10343409a2a423eb0b21759bfc9ba61 P 327aa26a8e434a609ebcb382aa665fce: Bootstrap starting.
I20240503 01:58:58.681910 3443 ts_tablet_manager.cc:1367] T 9d84cbdf02c14873b8838be296977aaf P 52c196c15e3348b985c689652bc49918: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:58:58.682238 3441 raft_consensus.cc:348] T 9d84cbdf02c14873b8838be296977aaf P a1a844ecd71145e4a362f4c42500fbcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } }
I20240503 01:58:58.682351 3441 raft_consensus.cc:374] T 9d84cbdf02c14873b8838be296977aaf P a1a844ecd71145e4a362f4c42500fbcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:58.682308 3443 raft_consensus.cc:348] T 9d84cbdf02c14873b8838be296977aaf P 52c196c15e3348b985c689652bc49918 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } }
I20240503 01:58:58.682390 3441 raft_consensus.cc:729] T 9d84cbdf02c14873b8838be296977aaf P a1a844ecd71145e4a362f4c42500fbcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1a844ecd71145e4a362f4c42500fbcb, State: Initialized, Role: FOLLOWER
I20240503 01:58:58.682411 3443 raft_consensus.cc:374] T 9d84cbdf02c14873b8838be296977aaf P 52c196c15e3348b985c689652bc49918 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:58.682441 3443 raft_consensus.cc:729] T 9d84cbdf02c14873b8838be296977aaf P 52c196c15e3348b985c689652bc49918 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52c196c15e3348b985c689652bc49918, State: Initialized, Role: FOLLOWER
I20240503 01:58:58.682510 3441 consensus_queue.cc:260] T 9d84cbdf02c14873b8838be296977aaf P a1a844ecd71145e4a362f4c42500fbcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } }
I20240503 01:58:58.682547 3443 consensus_queue.cc:260] T 9d84cbdf02c14873b8838be296977aaf P 52c196c15e3348b985c689652bc49918 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } }
I20240503 01:58:58.682742 3439 tablet_bootstrap.cc:654] T c10343409a2a423eb0b21759bfc9ba61 P 327aa26a8e434a609ebcb382aa665fce: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:58.682816 3443 ts_tablet_manager.cc:1398] T 9d84cbdf02c14873b8838be296977aaf P 52c196c15e3348b985c689652bc49918: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:58:58.682811 3441 ts_tablet_manager.cc:1398] T 9d84cbdf02c14873b8838be296977aaf P a1a844ecd71145e4a362f4c42500fbcb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:58:58.683724 3439 tablet_bootstrap.cc:492] T c10343409a2a423eb0b21759bfc9ba61 P 327aa26a8e434a609ebcb382aa665fce: No bootstrap required, opened a new log
I20240503 01:58:58.683789 3439 ts_tablet_manager.cc:1367] T c10343409a2a423eb0b21759bfc9ba61 P 327aa26a8e434a609ebcb382aa665fce: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:58:58.684247 3439 raft_consensus.cc:348] T c10343409a2a423eb0b21759bfc9ba61 P 327aa26a8e434a609ebcb382aa665fce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } }
I20240503 01:58:58.684360 3439 raft_consensus.cc:374] T c10343409a2a423eb0b21759bfc9ba61 P 327aa26a8e434a609ebcb382aa665fce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:58.684393 3439 raft_consensus.cc:729] T c10343409a2a423eb0b21759bfc9ba61 P 327aa26a8e434a609ebcb382aa665fce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 327aa26a8e434a609ebcb382aa665fce, State: Initialized, Role: FOLLOWER
I20240503 01:58:58.684531 3439 consensus_queue.cc:260] T c10343409a2a423eb0b21759bfc9ba61 P 327aa26a8e434a609ebcb382aa665fce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } }
I20240503 01:58:58.684835 3439 ts_tablet_manager.cc:1398] T c10343409a2a423eb0b21759bfc9ba61 P 327aa26a8e434a609ebcb382aa665fce: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:58:58.685002 3439 tablet_bootstrap.cc:492] T 9d84cbdf02c14873b8838be296977aaf P 327aa26a8e434a609ebcb382aa665fce: Bootstrap starting.
I20240503 01:58:58.685871 3439 tablet_bootstrap.cc:654] T 9d84cbdf02c14873b8838be296977aaf P 327aa26a8e434a609ebcb382aa665fce: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:58.686815 3444 tablet_bootstrap.cc:492] T 9d84cbdf02c14873b8838be296977aaf P 6568f69a78c44bb985162d2b7d0a9376: No bootstrap required, opened a new log
I20240503 01:58:58.687014 3444 ts_tablet_manager.cc:1367] T 9d84cbdf02c14873b8838be296977aaf P 6568f69a78c44bb985162d2b7d0a9376: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 01:58:58.687868 3444 raft_consensus.cc:348] T 9d84cbdf02c14873b8838be296977aaf P 6568f69a78c44bb985162d2b7d0a9376 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } }
I20240503 01:58:58.687987 3444 raft_consensus.cc:374] T 9d84cbdf02c14873b8838be296977aaf P 6568f69a78c44bb985162d2b7d0a9376 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:58.688023 3444 raft_consensus.cc:729] T 9d84cbdf02c14873b8838be296977aaf P 6568f69a78c44bb985162d2b7d0a9376 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6568f69a78c44bb985162d2b7d0a9376, State: Initialized, Role: FOLLOWER
I20240503 01:58:58.688136 3439 tablet_bootstrap.cc:492] T 9d84cbdf02c14873b8838be296977aaf P 327aa26a8e434a609ebcb382aa665fce: No bootstrap required, opened a new log
I20240503 01:58:58.688212 3439 ts_tablet_manager.cc:1367] T 9d84cbdf02c14873b8838be296977aaf P 327aa26a8e434a609ebcb382aa665fce: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240503 01:58:58.688164 3444 consensus_queue.cc:260] T 9d84cbdf02c14873b8838be296977aaf P 6568f69a78c44bb985162d2b7d0a9376 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } }
I20240503 01:58:58.688462 3444 ts_tablet_manager.cc:1398] T 9d84cbdf02c14873b8838be296977aaf P 6568f69a78c44bb985162d2b7d0a9376: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:58:58.688709 3439 raft_consensus.cc:348] T 9d84cbdf02c14873b8838be296977aaf P 327aa26a8e434a609ebcb382aa665fce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } }
I20240503 01:58:58.688815 3439 raft_consensus.cc:374] T 9d84cbdf02c14873b8838be296977aaf P 327aa26a8e434a609ebcb382aa665fce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:58.688849 3439 raft_consensus.cc:729] T 9d84cbdf02c14873b8838be296977aaf P 327aa26a8e434a609ebcb382aa665fce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 327aa26a8e434a609ebcb382aa665fce, State: Initialized, Role: FOLLOWER
I20240503 01:58:58.688958 3439 consensus_queue.cc:260] T 9d84cbdf02c14873b8838be296977aaf P 327aa26a8e434a609ebcb382aa665fce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } }
I20240503 01:58:58.689215 3439 ts_tablet_manager.cc:1398] T 9d84cbdf02c14873b8838be296977aaf P 327aa26a8e434a609ebcb382aa665fce: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:58:58.768201 3393 raft_consensus.cc:482] T 9d84cbdf02c14873b8838be296977aaf P 327aa26a8e434a609ebcb382aa665fce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:58:58.768343 3393 raft_consensus.cc:504] T 9d84cbdf02c14873b8838be296977aaf P 327aa26a8e434a609ebcb382aa665fce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } }
I20240503 01:58:58.769600 3393 leader_election.cc:288] T 9d84cbdf02c14873b8838be296977aaf P 327aa26a8e434a609ebcb382aa665fce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b40a9a7b30d34a6db007ed8febd8a759 (127.0.99.131:34079), a1a844ecd71145e4a362f4c42500fbcb (127.0.99.130:46033), 52c196c15e3348b985c689652bc49918 (127.0.99.132:42887), 6568f69a78c44bb985162d2b7d0a9376 (127.0.99.134:43603)
I20240503 01:58:58.774513 2662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d84cbdf02c14873b8838be296977aaf" candidate_uuid: "327aa26a8e434a609ebcb382aa665fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b40a9a7b30d34a6db007ed8febd8a759" is_pre_election: true
I20240503 01:58:58.774682 2662 raft_consensus.cc:2454] T 9d84cbdf02c14873b8838be296977aaf P b40a9a7b30d34a6db007ed8febd8a759 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 327aa26a8e434a609ebcb382aa665fce in term 0.
I20240503 01:58:58.775202 3055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d84cbdf02c14873b8838be296977aaf" candidate_uuid: "327aa26a8e434a609ebcb382aa665fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6568f69a78c44bb985162d2b7d0a9376" is_pre_election: true
I20240503 01:58:58.775365 3055 raft_consensus.cc:2454] T 9d84cbdf02c14873b8838be296977aaf P 6568f69a78c44bb985162d2b7d0a9376 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 327aa26a8e434a609ebcb382aa665fce in term 0.
I20240503 01:58:58.775411 2530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d84cbdf02c14873b8838be296977aaf" candidate_uuid: "327aa26a8e434a609ebcb382aa665fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1a844ecd71145e4a362f4c42500fbcb" is_pre_election: true
I20240503 01:58:58.775553 2530 raft_consensus.cc:2454] T 9d84cbdf02c14873b8838be296977aaf P a1a844ecd71145e4a362f4c42500fbcb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 327aa26a8e434a609ebcb382aa665fce in term 0.
I20240503 01:58:58.775708 2858 leader_election.cc:302] T 9d84cbdf02c14873b8838be296977aaf P 327aa26a8e434a609ebcb382aa665fce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 327aa26a8e434a609ebcb382aa665fce, 6568f69a78c44bb985162d2b7d0a9376, b40a9a7b30d34a6db007ed8febd8a759; no voters:
I20240503 01:58:58.775836 2793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d84cbdf02c14873b8838be296977aaf" candidate_uuid: "327aa26a8e434a609ebcb382aa665fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52c196c15e3348b985c689652bc49918" is_pre_election: true
I20240503 01:58:58.775949 3393 raft_consensus.cc:2781] T 9d84cbdf02c14873b8838be296977aaf P 327aa26a8e434a609ebcb382aa665fce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:58:58.776012 3393 raft_consensus.cc:482] T 9d84cbdf02c14873b8838be296977aaf P 327aa26a8e434a609ebcb382aa665fce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:58:58.775995 2793 raft_consensus.cc:2454] T 9d84cbdf02c14873b8838be296977aaf P 52c196c15e3348b985c689652bc49918 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 327aa26a8e434a609ebcb382aa665fce in term 0.
I20240503 01:58:58.776062 3393 raft_consensus.cc:3037] T 9d84cbdf02c14873b8838be296977aaf P 327aa26a8e434a609ebcb382aa665fce [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:58.776993 3393 raft_consensus.cc:504] T 9d84cbdf02c14873b8838be296977aaf P 327aa26a8e434a609ebcb382aa665fce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } }
I20240503 01:58:58.777482 3393 leader_election.cc:288] T 9d84cbdf02c14873b8838be296977aaf P 327aa26a8e434a609ebcb382aa665fce [CANDIDATE]: Term 1 election: Requested vote from peers b40a9a7b30d34a6db007ed8febd8a759 (127.0.99.131:34079), a1a844ecd71145e4a362f4c42500fbcb (127.0.99.130:46033), 52c196c15e3348b985c689652bc49918 (127.0.99.132:42887), 6568f69a78c44bb985162d2b7d0a9376 (127.0.99.134:43603)
I20240503 01:58:58.777691 2662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d84cbdf02c14873b8838be296977aaf" candidate_uuid: "327aa26a8e434a609ebcb382aa665fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b40a9a7b30d34a6db007ed8febd8a759"
I20240503 01:58:58.777815 2662 raft_consensus.cc:3037] T 9d84cbdf02c14873b8838be296977aaf P b40a9a7b30d34a6db007ed8febd8a759 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:58.777864 3055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d84cbdf02c14873b8838be296977aaf" candidate_uuid: "327aa26a8e434a609ebcb382aa665fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6568f69a78c44bb985162d2b7d0a9376"
I20240503 01:58:58.777864 2530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d84cbdf02c14873b8838be296977aaf" candidate_uuid: "327aa26a8e434a609ebcb382aa665fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1a844ecd71145e4a362f4c42500fbcb"
I20240503 01:58:58.777967 3055 raft_consensus.cc:3037] T 9d84cbdf02c14873b8838be296977aaf P 6568f69a78c44bb985162d2b7d0a9376 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:58.777967 2530 raft_consensus.cc:3037] T 9d84cbdf02c14873b8838be296977aaf P a1a844ecd71145e4a362f4c42500fbcb [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:58.777958 2793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d84cbdf02c14873b8838be296977aaf" candidate_uuid: "327aa26a8e434a609ebcb382aa665fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52c196c15e3348b985c689652bc49918"
I20240503 01:58:58.778062 2793 raft_consensus.cc:3037] T 9d84cbdf02c14873b8838be296977aaf P 52c196c15e3348b985c689652bc49918 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:58.778823 2662 raft_consensus.cc:2454] T 9d84cbdf02c14873b8838be296977aaf P b40a9a7b30d34a6db007ed8febd8a759 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 327aa26a8e434a609ebcb382aa665fce in term 1.
I20240503 01:58:58.778867 2530 raft_consensus.cc:2454] T 9d84cbdf02c14873b8838be296977aaf P a1a844ecd71145e4a362f4c42500fbcb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 327aa26a8e434a609ebcb382aa665fce in term 1.
I20240503 01:58:58.778874 3055 raft_consensus.cc:2454] T 9d84cbdf02c14873b8838be296977aaf P 6568f69a78c44bb985162d2b7d0a9376 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 327aa26a8e434a609ebcb382aa665fce in term 1.
I20240503 01:58:58.779053 2793 raft_consensus.cc:2454] T 9d84cbdf02c14873b8838be296977aaf P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 327aa26a8e434a609ebcb382aa665fce in term 1.
I20240503 01:58:58.779192 2857 leader_election.cc:302] T 9d84cbdf02c14873b8838be296977aaf P 327aa26a8e434a609ebcb382aa665fce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 327aa26a8e434a609ebcb382aa665fce, 6568f69a78c44bb985162d2b7d0a9376, b40a9a7b30d34a6db007ed8febd8a759; no voters:
I20240503 01:58:58.779366 3393 raft_consensus.cc:2781] T 9d84cbdf02c14873b8838be296977aaf P 327aa26a8e434a609ebcb382aa665fce [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:58:58.779448 3393 raft_consensus.cc:686] T 9d84cbdf02c14873b8838be296977aaf P 327aa26a8e434a609ebcb382aa665fce [term 1 LEADER]: Becoming Leader. State: Replica: 327aa26a8e434a609ebcb382aa665fce, State: Running, Role: LEADER
I20240503 01:58:58.779749 3393 consensus_queue.cc:237] T 9d84cbdf02c14873b8838be296977aaf P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } }
I20240503 01:58:58.782145 2256 catalog_manager.cc:5484] T 9d84cbdf02c14873b8838be296977aaf P 327aa26a8e434a609ebcb382aa665fce reported cstate change: term changed from 0 to 1, leader changed from <none> to 327aa26a8e434a609ebcb382aa665fce (127.0.99.133). New cstate: current_term: 1 leader_uuid: "327aa26a8e434a609ebcb382aa665fce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } health_report { overall_health: UNKNOWN } } }
I20240503 01:58:58.857815 3393 raft_consensus.cc:482] T a6e17a2c4ba0482e96d04eb9d3ba854a P 327aa26a8e434a609ebcb382aa665fce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:58:58.857955 3393 raft_consensus.cc:504] T a6e17a2c4ba0482e96d04eb9d3ba854a P 327aa26a8e434a609ebcb382aa665fce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } }
I20240503 01:58:58.858570 3393 leader_election.cc:288] T a6e17a2c4ba0482e96d04eb9d3ba854a P 327aa26a8e434a609ebcb382aa665fce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b40a9a7b30d34a6db007ed8febd8a759 (127.0.99.131:34079), 52c196c15e3348b985c689652bc49918 (127.0.99.132:42887), a1a844ecd71145e4a362f4c42500fbcb (127.0.99.130:46033), 6568f69a78c44bb985162d2b7d0a9376 (127.0.99.134:43603)
I20240503 01:58:58.858880 2662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6e17a2c4ba0482e96d04eb9d3ba854a" candidate_uuid: "327aa26a8e434a609ebcb382aa665fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b40a9a7b30d34a6db007ed8febd8a759" is_pre_election: true
I20240503 01:58:58.858938 3055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6e17a2c4ba0482e96d04eb9d3ba854a" candidate_uuid: "327aa26a8e434a609ebcb382aa665fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6568f69a78c44bb985162d2b7d0a9376" is_pre_election: true
I20240503 01:58:58.858989 2530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6e17a2c4ba0482e96d04eb9d3ba854a" candidate_uuid: "327aa26a8e434a609ebcb382aa665fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1a844ecd71145e4a362f4c42500fbcb" is_pre_election: true
I20240503 01:58:58.859030 2662 raft_consensus.cc:2454] T a6e17a2c4ba0482e96d04eb9d3ba854a P b40a9a7b30d34a6db007ed8febd8a759 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 327aa26a8e434a609ebcb382aa665fce in term 0.
I20240503 01:58:58.859006 2793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6e17a2c4ba0482e96d04eb9d3ba854a" candidate_uuid: "327aa26a8e434a609ebcb382aa665fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52c196c15e3348b985c689652bc49918" is_pre_election: true
I20240503 01:58:58.859097 3055 raft_consensus.cc:2454] T a6e17a2c4ba0482e96d04eb9d3ba854a P 6568f69a78c44bb985162d2b7d0a9376 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 327aa26a8e434a609ebcb382aa665fce in term 0.
I20240503 01:58:58.859113 2530 raft_consensus.cc:2454] T a6e17a2c4ba0482e96d04eb9d3ba854a P a1a844ecd71145e4a362f4c42500fbcb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 327aa26a8e434a609ebcb382aa665fce in term 0.
I20240503 01:58:58.859144 2793 raft_consensus.cc:2454] T a6e17a2c4ba0482e96d04eb9d3ba854a P 52c196c15e3348b985c689652bc49918 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 327aa26a8e434a609ebcb382aa665fce in term 0.
I20240503 01:58:58.859434 2857 leader_election.cc:302] T a6e17a2c4ba0482e96d04eb9d3ba854a P 327aa26a8e434a609ebcb382aa665fce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 327aa26a8e434a609ebcb382aa665fce, 6568f69a78c44bb985162d2b7d0a9376, b40a9a7b30d34a6db007ed8febd8a759; no voters:
I20240503 01:58:58.859620 3393 raft_consensus.cc:2781] T a6e17a2c4ba0482e96d04eb9d3ba854a P 327aa26a8e434a609ebcb382aa665fce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:58:58.859670 3393 raft_consensus.cc:482] T a6e17a2c4ba0482e96d04eb9d3ba854a P 327aa26a8e434a609ebcb382aa665fce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:58:58.859699 3393 raft_consensus.cc:3037] T a6e17a2c4ba0482e96d04eb9d3ba854a P 327aa26a8e434a609ebcb382aa665fce [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:58.860538 3393 raft_consensus.cc:504] T a6e17a2c4ba0482e96d04eb9d3ba854a P 327aa26a8e434a609ebcb382aa665fce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } }
I20240503 01:58:58.860949 3393 leader_election.cc:288] T a6e17a2c4ba0482e96d04eb9d3ba854a P 327aa26a8e434a609ebcb382aa665fce [CANDIDATE]: Term 1 election: Requested vote from peers b40a9a7b30d34a6db007ed8febd8a759 (127.0.99.131:34079), 52c196c15e3348b985c689652bc49918 (127.0.99.132:42887), a1a844ecd71145e4a362f4c42500fbcb (127.0.99.130:46033), 6568f69a78c44bb985162d2b7d0a9376 (127.0.99.134:43603)
I20240503 01:58:58.861296 2530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6e17a2c4ba0482e96d04eb9d3ba854a" candidate_uuid: "327aa26a8e434a609ebcb382aa665fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1a844ecd71145e4a362f4c42500fbcb"
I20240503 01:58:58.861307 2662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6e17a2c4ba0482e96d04eb9d3ba854a" candidate_uuid: "327aa26a8e434a609ebcb382aa665fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b40a9a7b30d34a6db007ed8febd8a759"
I20240503 01:58:58.861387 2793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6e17a2c4ba0482e96d04eb9d3ba854a" candidate_uuid: "327aa26a8e434a609ebcb382aa665fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52c196c15e3348b985c689652bc49918"
I20240503 01:58:58.861415 2662 raft_consensus.cc:3037] T a6e17a2c4ba0482e96d04eb9d3ba854a P b40a9a7b30d34a6db007ed8febd8a759 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:58.861418 2530 raft_consensus.cc:3037] T a6e17a2c4ba0482e96d04eb9d3ba854a P a1a844ecd71145e4a362f4c42500fbcb [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:58.861407 3055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6e17a2c4ba0482e96d04eb9d3ba854a" candidate_uuid: "327aa26a8e434a609ebcb382aa665fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6568f69a78c44bb985162d2b7d0a9376"
I20240503 01:58:58.861483 2793 raft_consensus.cc:3037] T a6e17a2c4ba0482e96d04eb9d3ba854a P 52c196c15e3348b985c689652bc49918 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:58.861497 3055 raft_consensus.cc:3037] T a6e17a2c4ba0482e96d04eb9d3ba854a P 6568f69a78c44bb985162d2b7d0a9376 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:58.862380 2793 raft_consensus.cc:2454] T a6e17a2c4ba0482e96d04eb9d3ba854a P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 327aa26a8e434a609ebcb382aa665fce in term 1.
I20240503 01:58:58.862382 2662 raft_consensus.cc:2454] T a6e17a2c4ba0482e96d04eb9d3ba854a P b40a9a7b30d34a6db007ed8febd8a759 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 327aa26a8e434a609ebcb382aa665fce in term 1.
I20240503 01:58:58.862380 2530 raft_consensus.cc:2454] T a6e17a2c4ba0482e96d04eb9d3ba854a P a1a844ecd71145e4a362f4c42500fbcb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 327aa26a8e434a609ebcb382aa665fce in term 1.
I20240503 01:58:58.862421 3055 raft_consensus.cc:2454] T a6e17a2c4ba0482e96d04eb9d3ba854a P 6568f69a78c44bb985162d2b7d0a9376 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 327aa26a8e434a609ebcb382aa665fce in term 1.
I20240503 01:58:58.862746 2860 leader_election.cc:302] T a6e17a2c4ba0482e96d04eb9d3ba854a P 327aa26a8e434a609ebcb382aa665fce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 327aa26a8e434a609ebcb382aa665fce, 52c196c15e3348b985c689652bc49918, a1a844ecd71145e4a362f4c42500fbcb; no voters:
I20240503 01:58:58.862898 3393 raft_consensus.cc:2781] T a6e17a2c4ba0482e96d04eb9d3ba854a P 327aa26a8e434a609ebcb382aa665fce [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:58:58.862977 3393 raft_consensus.cc:686] T a6e17a2c4ba0482e96d04eb9d3ba854a P 327aa26a8e434a609ebcb382aa665fce [term 1 LEADER]: Becoming Leader. State: Replica: 327aa26a8e434a609ebcb382aa665fce, State: Running, Role: LEADER
I20240503 01:58:58.863119 3393 consensus_queue.cc:237] T a6e17a2c4ba0482e96d04eb9d3ba854a P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } }
I20240503 01:58:58.864929 2256 catalog_manager.cc:5484] T a6e17a2c4ba0482e96d04eb9d3ba854a P 327aa26a8e434a609ebcb382aa665fce reported cstate change: term changed from 0 to 1, leader changed from <none> to 327aa26a8e434a609ebcb382aa665fce (127.0.99.133). New cstate: current_term: 1 leader_uuid: "327aa26a8e434a609ebcb382aa665fce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } health_report { overall_health: UNKNOWN } } }
I20240503 01:58:58.961592 3314 raft_consensus.cc:482] T c10343409a2a423eb0b21759bfc9ba61 P b63928177fa54227a5ff9929265ab9dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:58:58.961710 3314 raft_consensus.cc:504] T c10343409a2a423eb0b21759bfc9ba61 P b63928177fa54227a5ff9929265ab9dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } }
I20240503 01:58:58.962919 3314 leader_election.cc:288] T c10343409a2a423eb0b21759bfc9ba61 P b63928177fa54227a5ff9929265ab9dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b40a9a7b30d34a6db007ed8febd8a759 (127.0.99.131:34079), a1a844ecd71145e4a362f4c42500fbcb (127.0.99.130:46033), 52c196c15e3348b985c689652bc49918 (127.0.99.132:42887), 327aa26a8e434a609ebcb382aa665fce (127.0.99.133:38853)
I20240503 01:58:58.968120 2530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c10343409a2a423eb0b21759bfc9ba61" candidate_uuid: "b63928177fa54227a5ff9929265ab9dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1a844ecd71145e4a362f4c42500fbcb" is_pre_election: true
I20240503 01:58:58.968263 2530 raft_consensus.cc:2454] T c10343409a2a423eb0b21759bfc9ba61 P a1a844ecd71145e4a362f4c42500fbcb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b63928177fa54227a5ff9929265ab9dd in term 0.
I20240503 01:58:58.968819 2793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c10343409a2a423eb0b21759bfc9ba61" candidate_uuid: "b63928177fa54227a5ff9929265ab9dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52c196c15e3348b985c689652bc49918" is_pre_election: true
I20240503 01:58:58.968966 2793 raft_consensus.cc:2454] T c10343409a2a423eb0b21759bfc9ba61 P 52c196c15e3348b985c689652bc49918 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b63928177fa54227a5ff9929265ab9dd in term 0.
I20240503 01:58:58.969287 2336 leader_election.cc:302] T c10343409a2a423eb0b21759bfc9ba61 P b63928177fa54227a5ff9929265ab9dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 52c196c15e3348b985c689652bc49918, a1a844ecd71145e4a362f4c42500fbcb, b63928177fa54227a5ff9929265ab9dd; no voters:
I20240503 01:58:58.969637 3314 raft_consensus.cc:2781] T c10343409a2a423eb0b21759bfc9ba61 P b63928177fa54227a5ff9929265ab9dd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:58:58.969693 3314 raft_consensus.cc:482] T c10343409a2a423eb0b21759bfc9ba61 P b63928177fa54227a5ff9929265ab9dd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:58:58.969729 3314 raft_consensus.cc:3037] T c10343409a2a423eb0b21759bfc9ba61 P b63928177fa54227a5ff9929265ab9dd [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:58.969961 2662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c10343409a2a423eb0b21759bfc9ba61" candidate_uuid: "b63928177fa54227a5ff9929265ab9dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b40a9a7b30d34a6db007ed8febd8a759" is_pre_election: true
I20240503 01:58:58.970077 2662 raft_consensus.cc:2454] T c10343409a2a423eb0b21759bfc9ba61 P b40a9a7b30d34a6db007ed8febd8a759 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b63928177fa54227a5ff9929265ab9dd in term 0.
I20240503 01:58:58.970816 3314 raft_consensus.cc:504] T c10343409a2a423eb0b21759bfc9ba61 P b63928177fa54227a5ff9929265ab9dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } }
I20240503 01:58:58.971316 3314 leader_election.cc:288] T c10343409a2a423eb0b21759bfc9ba61 P b63928177fa54227a5ff9929265ab9dd [CANDIDATE]: Term 1 election: Requested vote from peers b40a9a7b30d34a6db007ed8febd8a759 (127.0.99.131:34079), a1a844ecd71145e4a362f4c42500fbcb (127.0.99.130:46033), 52c196c15e3348b985c689652bc49918 (127.0.99.132:42887), 327aa26a8e434a609ebcb382aa665fce (127.0.99.133:38853)
I20240503 01:58:58.971482 2662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c10343409a2a423eb0b21759bfc9ba61" candidate_uuid: "b63928177fa54227a5ff9929265ab9dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b40a9a7b30d34a6db007ed8febd8a759"
I20240503 01:58:58.971596 2662 raft_consensus.cc:3037] T c10343409a2a423eb0b21759bfc9ba61 P b40a9a7b30d34a6db007ed8febd8a759 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:58.971974 2793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c10343409a2a423eb0b21759bfc9ba61" candidate_uuid: "b63928177fa54227a5ff9929265ab9dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52c196c15e3348b985c689652bc49918"
I20240503 01:58:58.971985 2530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c10343409a2a423eb0b21759bfc9ba61" candidate_uuid: "b63928177fa54227a5ff9929265ab9dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1a844ecd71145e4a362f4c42500fbcb"
I20240503 01:58:58.972079 2793 raft_consensus.cc:3037] T c10343409a2a423eb0b21759bfc9ba61 P 52c196c15e3348b985c689652bc49918 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:58.972085 2530 raft_consensus.cc:3037] T c10343409a2a423eb0b21759bfc9ba61 P a1a844ecd71145e4a362f4c42500fbcb [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:58.972601 2662 raft_consensus.cc:2454] T c10343409a2a423eb0b21759bfc9ba61 P b40a9a7b30d34a6db007ed8febd8a759 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b63928177fa54227a5ff9929265ab9dd in term 1.
I20240503 01:58:58.973003 2530 raft_consensus.cc:2454] T c10343409a2a423eb0b21759bfc9ba61 P a1a844ecd71145e4a362f4c42500fbcb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b63928177fa54227a5ff9929265ab9dd in term 1.
I20240503 01:58:58.973206 2793 raft_consensus.cc:2454] T c10343409a2a423eb0b21759bfc9ba61 P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b63928177fa54227a5ff9929265ab9dd in term 1.
I20240503 01:58:58.973578 2336 leader_election.cc:302] T c10343409a2a423eb0b21759bfc9ba61 P b63928177fa54227a5ff9929265ab9dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: a1a844ecd71145e4a362f4c42500fbcb, b40a9a7b30d34a6db007ed8febd8a759, b63928177fa54227a5ff9929265ab9dd; no voters:
I20240503 01:58:58.973726 2924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c10343409a2a423eb0b21759bfc9ba61" candidate_uuid: "b63928177fa54227a5ff9929265ab9dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "327aa26a8e434a609ebcb382aa665fce" is_pre_election: true
I20240503 01:58:58.973791 3314 raft_consensus.cc:2781] T c10343409a2a423eb0b21759bfc9ba61 P b63928177fa54227a5ff9929265ab9dd [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:58:58.973856 2924 raft_consensus.cc:2454] T c10343409a2a423eb0b21759bfc9ba61 P 327aa26a8e434a609ebcb382aa665fce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b63928177fa54227a5ff9929265ab9dd in term 0.
I20240503 01:58:58.973914 2923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c10343409a2a423eb0b21759bfc9ba61" candidate_uuid: "b63928177fa54227a5ff9929265ab9dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "327aa26a8e434a609ebcb382aa665fce"
I20240503 01:58:58.973986 3314 raft_consensus.cc:686] T c10343409a2a423eb0b21759bfc9ba61 P b63928177fa54227a5ff9929265ab9dd [term 1 LEADER]: Becoming Leader. State: Replica: b63928177fa54227a5ff9929265ab9dd, State: Running, Role: LEADER
I20240503 01:58:58.974027 2923 raft_consensus.cc:3037] T c10343409a2a423eb0b21759bfc9ba61 P 327aa26a8e434a609ebcb382aa665fce [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:58.974318 3314 consensus_queue.cc:237] T c10343409a2a423eb0b21759bfc9ba61 P b63928177fa54227a5ff9929265ab9dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } }
I20240503 01:58:58.974947 2923 raft_consensus.cc:2454] T c10343409a2a423eb0b21759bfc9ba61 P 327aa26a8e434a609ebcb382aa665fce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b63928177fa54227a5ff9929265ab9dd in term 1.
I20240503 01:58:58.977085 2256 catalog_manager.cc:5484] T c10343409a2a423eb0b21759bfc9ba61 P b63928177fa54227a5ff9929265ab9dd reported cstate change: term changed from 0 to 1, leader changed from <none> to b63928177fa54227a5ff9929265ab9dd (127.0.99.129). New cstate: current_term: 1 leader_uuid: "b63928177fa54227a5ff9929265ab9dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } health_report { overall_health: UNKNOWN } } }
W20240503 01:58:59.093080 2598 consensus_peers.cc:485] T 240749b0904f4bb2a4c79b7713473411 P b40a9a7b30d34a6db007ed8febd8a759 -> Peer 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135:39501): Couldn't send request to peer 4125b1a70e864e129bd54bdc2720d7fa. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:39501: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 01:58:59.154007 3476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:58:59.154335 3476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:58:59.157895 3476 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240503 01:58:59.237144 2923 raft_consensus.cc:1261] T c10343409a2a423eb0b21759bfc9ba61 P 327aa26a8e434a609ebcb382aa665fce [term 1 FOLLOWER]: Refusing update from remote peer b63928177fa54227a5ff9929265ab9dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:58:59.237551 3055 raft_consensus.cc:1261] T 9d84cbdf02c14873b8838be296977aaf P 6568f69a78c44bb985162d2b7d0a9376 [term 1 FOLLOWER]: Refusing update from remote peer 327aa26a8e434a609ebcb382aa665fce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:58:59.237937 2793 raft_consensus.cc:1261] T 9d84cbdf02c14873b8838be296977aaf P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Refusing update from remote peer 327aa26a8e434a609ebcb382aa665fce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:58:59.238201 3314 consensus_queue.cc:1035] T c10343409a2a423eb0b21759bfc9ba61 P b63928177fa54227a5ff9929265ab9dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:59.238411 3393 consensus_queue.cc:1035] T 9d84cbdf02c14873b8838be296977aaf P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:59.239567 3055 raft_consensus.cc:1261] T a6e17a2c4ba0482e96d04eb9d3ba854a P 6568f69a78c44bb985162d2b7d0a9376 [term 1 FOLLOWER]: Refusing update from remote peer 327aa26a8e434a609ebcb382aa665fce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:58:59.240077 3393 consensus_queue.cc:1035] T 9d84cbdf02c14873b8838be296977aaf P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:59.240494 3393 consensus_queue.cc:1035] T a6e17a2c4ba0482e96d04eb9d3ba854a P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:59.241169 2662 raft_consensus.cc:1261] T 9d84cbdf02c14873b8838be296977aaf P b40a9a7b30d34a6db007ed8febd8a759 [term 1 FOLLOWER]: Refusing update from remote peer 327aa26a8e434a609ebcb382aa665fce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:58:59.241566 3393 consensus_queue.cc:1035] T 9d84cbdf02c14873b8838be296977aaf P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:59.241653 2530 raft_consensus.cc:1261] T 9d84cbdf02c14873b8838be296977aaf P a1a844ecd71145e4a362f4c42500fbcb [term 1 FOLLOWER]: Refusing update from remote peer 327aa26a8e434a609ebcb382aa665fce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:58:59.242038 3393 consensus_queue.cc:1035] T 9d84cbdf02c14873b8838be296977aaf P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:59.244755 2661 raft_consensus.cc:1261] T c10343409a2a423eb0b21759bfc9ba61 P b40a9a7b30d34a6db007ed8febd8a759 [term 1 FOLLOWER]: Refusing update from remote peer b63928177fa54227a5ff9929265ab9dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:58:59.245147 3314 consensus_queue.cc:1035] T c10343409a2a423eb0b21759bfc9ba61 P b63928177fa54227a5ff9929265ab9dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:59.245431 2531 raft_consensus.cc:1261] T c10343409a2a423eb0b21759bfc9ba61 P a1a844ecd71145e4a362f4c42500fbcb [term 1 FOLLOWER]: Refusing update from remote peer b63928177fa54227a5ff9929265ab9dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:58:59.245755 2792 raft_consensus.cc:1261] T c10343409a2a423eb0b21759bfc9ba61 P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Refusing update from remote peer b63928177fa54227a5ff9929265ab9dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:58:59.246318 3314 consensus_queue.cc:1035] T c10343409a2a423eb0b21759bfc9ba61 P b63928177fa54227a5ff9929265ab9dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:59.246973 3314 consensus_queue.cc:1035] T c10343409a2a423eb0b21759bfc9ba61 P b63928177fa54227a5ff9929265ab9dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:59.252851 2529 raft_consensus.cc:1261] T a6e17a2c4ba0482e96d04eb9d3ba854a P a1a844ecd71145e4a362f4c42500fbcb [term 1 FOLLOWER]: Refusing update from remote peer 327aa26a8e434a609ebcb382aa665fce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:58:59.253312 3327 consensus_queue.cc:1035] T a6e17a2c4ba0482e96d04eb9d3ba854a P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:59.253373 2660 raft_consensus.cc:1261] T a6e17a2c4ba0482e96d04eb9d3ba854a P b40a9a7b30d34a6db007ed8febd8a759 [term 1 FOLLOWER]: Refusing update from remote peer 327aa26a8e434a609ebcb382aa665fce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:58:59.253851 2791 raft_consensus.cc:1261] T a6e17a2c4ba0482e96d04eb9d3ba854a P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Refusing update from remote peer 327aa26a8e434a609ebcb382aa665fce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:58:59.254297 3393 consensus_queue.cc:1035] T a6e17a2c4ba0482e96d04eb9d3ba854a P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:59.259994 3393 consensus_queue.cc:1035] T a6e17a2c4ba0482e96d04eb9d3ba854a P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:59.286677 3528 mvcc.cc:205] Tried to move back new op lower bound from 7023417504709120000 to 7023417503642841088. Current Snapshot: MvccSnapshot[applied={T|T < 7023417504709120000}]
INSERT report
rows total: 2000
time total: 144.53 ms
time per row: 0.0722648 ms
I20240503 01:58:59.351086 2257 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53804:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 5
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:58:59.359874 2641 tablet_service.cc:1467] Processing CreateTablet for tablet 19b3a90e13194997bde2b9a806f8f2b5 (DEFAULT_TABLE table=rebalance_test_table_2 [id=18b5c04f80684606bc736d3ca6780b16]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:59.359908 3035 tablet_service.cc:1467] Processing CreateTablet for tablet 19b3a90e13194997bde2b9a806f8f2b5 (DEFAULT_TABLE table=rebalance_test_table_2 [id=18b5c04f80684606bc736d3ca6780b16]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:59.360205 3035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19b3a90e13194997bde2b9a806f8f2b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:59.360208 2641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19b3a90e13194997bde2b9a806f8f2b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:59.359912 3033 tablet_service.cc:1467] Processing CreateTablet for tablet 8f4c3f70f2714cc299bc0d595f838c85 (DEFAULT_TABLE table=rebalance_test_table_2 [id=18b5c04f80684606bc736d3ca6780b16]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:59.360394 3033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f4c3f70f2714cc299bc0d595f838c85. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:59.360620 3034 tablet_service.cc:1467] Processing CreateTablet for tablet c2911cf48f384bf09074f7bea2a4c173 (DEFAULT_TABLE table=rebalance_test_table_2 [id=18b5c04f80684606bc736d3ca6780b16]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:59.360853 3034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2911cf48f384bf09074f7bea2a4c173. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:59.361913 2902 tablet_service.cc:1467] Processing CreateTablet for tablet 8f4c3f70f2714cc299bc0d595f838c85 (DEFAULT_TABLE table=rebalance_test_table_2 [id=18b5c04f80684606bc736d3ca6780b16]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:59.362203 2902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f4c3f70f2714cc299bc0d595f838c85. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:59.362911 3562 tablet_bootstrap.cc:492] T 19b3a90e13194997bde2b9a806f8f2b5 P b40a9a7b30d34a6db007ed8febd8a759: Bootstrap starting.
I20240503 01:58:59.363137 3563 tablet_bootstrap.cc:492] T 8f4c3f70f2714cc299bc0d595f838c85 P 6568f69a78c44bb985162d2b7d0a9376: Bootstrap starting.
I20240503 01:58:59.363752 2772 tablet_service.cc:1467] Processing CreateTablet for tablet 19b3a90e13194997bde2b9a806f8f2b5 (DEFAULT_TABLE table=rebalance_test_table_2 [id=18b5c04f80684606bc736d3ca6780b16]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:59.364014 2772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19b3a90e13194997bde2b9a806f8f2b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:59.363973 2378 tablet_service.cc:1467] Processing CreateTablet for tablet 19b3a90e13194997bde2b9a806f8f2b5 (DEFAULT_TABLE table=rebalance_test_table_2 [id=18b5c04f80684606bc736d3ca6780b16]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:59.364089 2511 tablet_service.cc:1467] Processing CreateTablet for tablet 19b3a90e13194997bde2b9a806f8f2b5 (DEFAULT_TABLE table=rebalance_test_table_2 [id=18b5c04f80684606bc736d3ca6780b16]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:59.364236 2378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19b3a90e13194997bde2b9a806f8f2b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:59.364324 2511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19b3a90e13194997bde2b9a806f8f2b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:59.364881 2379 tablet_service.cc:1467] Processing CreateTablet for tablet c2911cf48f384bf09074f7bea2a4c173 (DEFAULT_TABLE table=rebalance_test_table_2 [id=18b5c04f80684606bc736d3ca6780b16]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:59.364907 2642 tablet_service.cc:1467] Processing CreateTablet for tablet c2911cf48f384bf09074f7bea2a4c173 (DEFAULT_TABLE table=rebalance_test_table_2 [id=18b5c04f80684606bc736d3ca6780b16]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:59.365093 2379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2911cf48f384bf09074f7bea2a4c173. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:59.365140 2642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2911cf48f384bf09074f7bea2a4c173. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:59.366535 2510 tablet_service.cc:1467] Processing CreateTablet for tablet c2911cf48f384bf09074f7bea2a4c173 (DEFAULT_TABLE table=rebalance_test_table_2 [id=18b5c04f80684606bc736d3ca6780b16]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:59.366746 2510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2911cf48f384bf09074f7bea2a4c173. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:59.367049 3562 tablet_bootstrap.cc:654] T 19b3a90e13194997bde2b9a806f8f2b5 P b40a9a7b30d34a6db007ed8febd8a759: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:59.367717 3565 tablet_bootstrap.cc:492] T 19b3a90e13194997bde2b9a806f8f2b5 P 52c196c15e3348b985c689652bc49918: Bootstrap starting.
I20240503 01:58:59.367760 2904 tablet_service.cc:1467] Processing CreateTablet for tablet c2911cf48f384bf09074f7bea2a4c173 (DEFAULT_TABLE table=rebalance_test_table_2 [id=18b5c04f80684606bc736d3ca6780b16]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:59.368009 2904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2911cf48f384bf09074f7bea2a4c173. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:59.368847 3565 tablet_bootstrap.cc:654] T 19b3a90e13194997bde2b9a806f8f2b5 P 52c196c15e3348b985c689652bc49918: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:59.368840 2509 tablet_service.cc:1467] Processing CreateTablet for tablet 8f4c3f70f2714cc299bc0d595f838c85 (DEFAULT_TABLE table=rebalance_test_table_2 [id=18b5c04f80684606bc736d3ca6780b16]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:59.369064 2509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f4c3f70f2714cc299bc0d595f838c85. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:59.369537 3563 tablet_bootstrap.cc:654] T 8f4c3f70f2714cc299bc0d595f838c85 P 6568f69a78c44bb985162d2b7d0a9376: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:59.370007 3562 tablet_bootstrap.cc:492] T 19b3a90e13194997bde2b9a806f8f2b5 P b40a9a7b30d34a6db007ed8febd8a759: No bootstrap required, opened a new log
I20240503 01:58:59.370234 3567 tablet_bootstrap.cc:492] T c2911cf48f384bf09074f7bea2a4c173 P b63928177fa54227a5ff9929265ab9dd: Bootstrap starting.
I20240503 01:58:59.370083 3562 ts_tablet_manager.cc:1367] T 19b3a90e13194997bde2b9a806f8f2b5 P b40a9a7b30d34a6db007ed8febd8a759: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 01:58:59.370407 3564 tablet_bootstrap.cc:492] T 8f4c3f70f2714cc299bc0d595f838c85 P 327aa26a8e434a609ebcb382aa665fce: Bootstrap starting.
I20240503 01:58:59.370779 3562 raft_consensus.cc:348] T 19b3a90e13194997bde2b9a806f8f2b5 P b40a9a7b30d34a6db007ed8febd8a759 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } }
I20240503 01:58:59.370889 3562 raft_consensus.cc:374] T 19b3a90e13194997bde2b9a806f8f2b5 P b40a9a7b30d34a6db007ed8febd8a759 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:59.370918 3562 raft_consensus.cc:729] T 19b3a90e13194997bde2b9a806f8f2b5 P b40a9a7b30d34a6db007ed8febd8a759 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b40a9a7b30d34a6db007ed8febd8a759, State: Initialized, Role: FOLLOWER
I20240503 01:58:59.371114 2773 tablet_service.cc:1467] Processing CreateTablet for tablet 8f4c3f70f2714cc299bc0d595f838c85 (DEFAULT_TABLE table=rebalance_test_table_2 [id=18b5c04f80684606bc736d3ca6780b16]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:59.371222 3564 tablet_bootstrap.cc:654] T 8f4c3f70f2714cc299bc0d595f838c85 P 327aa26a8e434a609ebcb382aa665fce: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:59.371130 3562 consensus_queue.cc:260] T 19b3a90e13194997bde2b9a806f8f2b5 P b40a9a7b30d34a6db007ed8febd8a759 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } }
I20240503 01:58:59.371245 3567 tablet_bootstrap.cc:654] T c2911cf48f384bf09074f7bea2a4c173 P b63928177fa54227a5ff9929265ab9dd: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:59.371433 2773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f4c3f70f2714cc299bc0d595f838c85. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:59.372313 3564 tablet_bootstrap.cc:492] T 8f4c3f70f2714cc299bc0d595f838c85 P 327aa26a8e434a609ebcb382aa665fce: No bootstrap required, opened a new log
I20240503 01:58:59.372399 3564 ts_tablet_manager.cc:1367] T 8f4c3f70f2714cc299bc0d595f838c85 P 327aa26a8e434a609ebcb382aa665fce: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:58:59.372534 3565 tablet_bootstrap.cc:492] T 19b3a90e13194997bde2b9a806f8f2b5 P 52c196c15e3348b985c689652bc49918: No bootstrap required, opened a new log
I20240503 01:58:59.372619 3565 ts_tablet_manager.cc:1367] T 19b3a90e13194997bde2b9a806f8f2b5 P 52c196c15e3348b985c689652bc49918: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20240503 01:58:59.372860 3564 raft_consensus.cc:348] T 8f4c3f70f2714cc299bc0d595f838c85 P 327aa26a8e434a609ebcb382aa665fce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } }
I20240503 01:58:59.372969 3564 raft_consensus.cc:374] T 8f4c3f70f2714cc299bc0d595f838c85 P 327aa26a8e434a609ebcb382aa665fce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:59.373005 3564 raft_consensus.cc:729] T 8f4c3f70f2714cc299bc0d595f838c85 P 327aa26a8e434a609ebcb382aa665fce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 327aa26a8e434a609ebcb382aa665fce, State: Initialized, Role: FOLLOWER
I20240503 01:58:59.373152 3564 consensus_queue.cc:260] T 8f4c3f70f2714cc299bc0d595f838c85 P 327aa26a8e434a609ebcb382aa665fce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } }
I20240503 01:58:59.373080 3565 raft_consensus.cc:348] T 19b3a90e13194997bde2b9a806f8f2b5 P 52c196c15e3348b985c689652bc49918 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } }
I20240503 01:58:59.373347 3565 raft_consensus.cc:374] T 19b3a90e13194997bde2b9a806f8f2b5 P 52c196c15e3348b985c689652bc49918 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:59.373380 3565 raft_consensus.cc:729] T 19b3a90e13194997bde2b9a806f8f2b5 P 52c196c15e3348b985c689652bc49918 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52c196c15e3348b985c689652bc49918, State: Initialized, Role: FOLLOWER
I20240503 01:58:59.373442 3564 ts_tablet_manager.cc:1398] T 8f4c3f70f2714cc299bc0d595f838c85 P 327aa26a8e434a609ebcb382aa665fce: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:58:59.373490 3565 consensus_queue.cc:260] T 19b3a90e13194997bde2b9a806f8f2b5 P 52c196c15e3348b985c689652bc49918 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } }
I20240503 01:58:59.373605 3564 tablet_bootstrap.cc:492] T c2911cf48f384bf09074f7bea2a4c173 P 327aa26a8e434a609ebcb382aa665fce: Bootstrap starting.
I20240503 01:58:59.373684 3562 ts_tablet_manager.cc:1398] T 19b3a90e13194997bde2b9a806f8f2b5 P b40a9a7b30d34a6db007ed8febd8a759: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 01:58:59.373682 2380 tablet_service.cc:1467] Processing CreateTablet for tablet 8f4c3f70f2714cc299bc0d595f838c85 (DEFAULT_TABLE table=rebalance_test_table_2 [id=18b5c04f80684606bc736d3ca6780b16]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:59.373883 2380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f4c3f70f2714cc299bc0d595f838c85. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:59.374220 3565 ts_tablet_manager.cc:1398] T 19b3a90e13194997bde2b9a806f8f2b5 P 52c196c15e3348b985c689652bc49918: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:58:59.374368 3565 tablet_bootstrap.cc:492] T 8f4c3f70f2714cc299bc0d595f838c85 P 52c196c15e3348b985c689652bc49918: Bootstrap starting.
I20240503 01:58:59.374437 3564 tablet_bootstrap.cc:654] T c2911cf48f384bf09074f7bea2a4c173 P 327aa26a8e434a609ebcb382aa665fce: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:59.375221 3565 tablet_bootstrap.cc:654] T 8f4c3f70f2714cc299bc0d595f838c85 P 52c196c15e3348b985c689652bc49918: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:59.376425 3565 tablet_bootstrap.cc:492] T 8f4c3f70f2714cc299bc0d595f838c85 P 52c196c15e3348b985c689652bc49918: No bootstrap required, opened a new log
I20240503 01:58:59.376507 3565 ts_tablet_manager.cc:1367] T 8f4c3f70f2714cc299bc0d595f838c85 P 52c196c15e3348b985c689652bc49918: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240503 01:58:59.376940 3565 raft_consensus.cc:348] T 8f4c3f70f2714cc299bc0d595f838c85 P 52c196c15e3348b985c689652bc49918 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } }
I20240503 01:58:59.377035 3565 raft_consensus.cc:374] T 8f4c3f70f2714cc299bc0d595f838c85 P 52c196c15e3348b985c689652bc49918 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:59.377060 3565 raft_consensus.cc:729] T 8f4c3f70f2714cc299bc0d595f838c85 P 52c196c15e3348b985c689652bc49918 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52c196c15e3348b985c689652bc49918, State: Initialized, Role: FOLLOWER
I20240503 01:58:59.377159 3565 consensus_queue.cc:260] T 8f4c3f70f2714cc299bc0d595f838c85 P 52c196c15e3348b985c689652bc49918 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } }
I20240503 01:58:59.377420 3565 ts_tablet_manager.cc:1398] T 8f4c3f70f2714cc299bc0d595f838c85 P 52c196c15e3348b985c689652bc49918: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:58:59.377467 3562 tablet_bootstrap.cc:492] T c2911cf48f384bf09074f7bea2a4c173 P b40a9a7b30d34a6db007ed8febd8a759: Bootstrap starting.
I20240503 01:58:59.377983 3566 tablet_bootstrap.cc:492] T 19b3a90e13194997bde2b9a806f8f2b5 P a1a844ecd71145e4a362f4c42500fbcb: Bootstrap starting.
I20240503 01:58:59.378298 3562 tablet_bootstrap.cc:654] T c2911cf48f384bf09074f7bea2a4c173 P b40a9a7b30d34a6db007ed8febd8a759: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:59.379142 3563 tablet_bootstrap.cc:492] T 8f4c3f70f2714cc299bc0d595f838c85 P 6568f69a78c44bb985162d2b7d0a9376: No bootstrap required, opened a new log
I20240503 01:58:59.379182 3566 tablet_bootstrap.cc:654] T 19b3a90e13194997bde2b9a806f8f2b5 P a1a844ecd71145e4a362f4c42500fbcb: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:59.379236 3563 ts_tablet_manager.cc:1367] T 8f4c3f70f2714cc299bc0d595f838c85 P 6568f69a78c44bb985162d2b7d0a9376: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20240503 01:58:59.379367 3564 tablet_bootstrap.cc:492] T c2911cf48f384bf09074f7bea2a4c173 P 327aa26a8e434a609ebcb382aa665fce: No bootstrap required, opened a new log
I20240503 01:58:59.379423 3564 ts_tablet_manager.cc:1367] T c2911cf48f384bf09074f7bea2a4c173 P 327aa26a8e434a609ebcb382aa665fce: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240503 01:58:59.379673 3563 raft_consensus.cc:348] T 8f4c3f70f2714cc299bc0d595f838c85 P 6568f69a78c44bb985162d2b7d0a9376 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } }
I20240503 01:58:59.379801 3563 raft_consensus.cc:374] T 8f4c3f70f2714cc299bc0d595f838c85 P 6568f69a78c44bb985162d2b7d0a9376 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:59.379726 3564 raft_consensus.cc:348] T c2911cf48f384bf09074f7bea2a4c173 P 327aa26a8e434a609ebcb382aa665fce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } }
I20240503 01:58:59.379835 3563 raft_consensus.cc:729] T 8f4c3f70f2714cc299bc0d595f838c85 P 6568f69a78c44bb985162d2b7d0a9376 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6568f69a78c44bb985162d2b7d0a9376, State: Initialized, Role: FOLLOWER
I20240503 01:58:59.379843 3564 raft_consensus.cc:374] T c2911cf48f384bf09074f7bea2a4c173 P 327aa26a8e434a609ebcb382aa665fce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:59.379884 3564 raft_consensus.cc:729] T c2911cf48f384bf09074f7bea2a4c173 P 327aa26a8e434a609ebcb382aa665fce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 327aa26a8e434a609ebcb382aa665fce, State: Initialized, Role: FOLLOWER
I20240503 01:58:59.379967 3563 consensus_queue.cc:260] T 8f4c3f70f2714cc299bc0d595f838c85 P 6568f69a78c44bb985162d2b7d0a9376 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } }
I20240503 01:58:59.380045 3564 consensus_queue.cc:260] T c2911cf48f384bf09074f7bea2a4c173 P 327aa26a8e434a609ebcb382aa665fce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } }
I20240503 01:58:59.380259 3563 ts_tablet_manager.cc:1398] T 8f4c3f70f2714cc299bc0d595f838c85 P 6568f69a78c44bb985162d2b7d0a9376: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:58:59.380335 3566 tablet_bootstrap.cc:492] T 19b3a90e13194997bde2b9a806f8f2b5 P a1a844ecd71145e4a362f4c42500fbcb: No bootstrap required, opened a new log
I20240503 01:58:59.380321 3564 ts_tablet_manager.cc:1398] T c2911cf48f384bf09074f7bea2a4c173 P 327aa26a8e434a609ebcb382aa665fce: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:58:59.380424 3563 tablet_bootstrap.cc:492] T c2911cf48f384bf09074f7bea2a4c173 P 6568f69a78c44bb985162d2b7d0a9376: Bootstrap starting.
I20240503 01:58:59.380424 3567 tablet_bootstrap.cc:492] T c2911cf48f384bf09074f7bea2a4c173 P b63928177fa54227a5ff9929265ab9dd: No bootstrap required, opened a new log
I20240503 01:58:59.380496 3567 ts_tablet_manager.cc:1367] T c2911cf48f384bf09074f7bea2a4c173 P b63928177fa54227a5ff9929265ab9dd: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240503 01:58:59.381000 3566 ts_tablet_manager.cc:1367] T 19b3a90e13194997bde2b9a806f8f2b5 P a1a844ecd71145e4a362f4c42500fbcb: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:58:59.380960 3567 raft_consensus.cc:348] T c2911cf48f384bf09074f7bea2a4c173 P b63928177fa54227a5ff9929265ab9dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } }
I20240503 01:58:59.381075 3567 raft_consensus.cc:374] T c2911cf48f384bf09074f7bea2a4c173 P b63928177fa54227a5ff9929265ab9dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:59.381124 3567 raft_consensus.cc:729] T c2911cf48f384bf09074f7bea2a4c173 P b63928177fa54227a5ff9929265ab9dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b63928177fa54227a5ff9929265ab9dd, State: Initialized, Role: FOLLOWER
I20240503 01:58:59.381265 3563 tablet_bootstrap.cc:654] T c2911cf48f384bf09074f7bea2a4c173 P 6568f69a78c44bb985162d2b7d0a9376: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:59.381250 3567 consensus_queue.cc:260] T c2911cf48f384bf09074f7bea2a4c173 P b63928177fa54227a5ff9929265ab9dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } }
I20240503 01:58:59.381525 3567 ts_tablet_manager.cc:1398] T c2911cf48f384bf09074f7bea2a4c173 P b63928177fa54227a5ff9929265ab9dd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:58:59.381476 3566 raft_consensus.cc:348] T 19b3a90e13194997bde2b9a806f8f2b5 P a1a844ecd71145e4a362f4c42500fbcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } }
I20240503 01:58:59.381589 3566 raft_consensus.cc:374] T 19b3a90e13194997bde2b9a806f8f2b5 P a1a844ecd71145e4a362f4c42500fbcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:59.381644 3566 raft_consensus.cc:729] T 19b3a90e13194997bde2b9a806f8f2b5 P a1a844ecd71145e4a362f4c42500fbcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1a844ecd71145e4a362f4c42500fbcb, State: Initialized, Role: FOLLOWER
I20240503 01:58:59.381722 3567 tablet_bootstrap.cc:492] T 19b3a90e13194997bde2b9a806f8f2b5 P b63928177fa54227a5ff9929265ab9dd: Bootstrap starting.
I20240503 01:58:59.381785 3566 consensus_queue.cc:260] T 19b3a90e13194997bde2b9a806f8f2b5 P a1a844ecd71145e4a362f4c42500fbcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } }
I20240503 01:58:59.382087 3566 ts_tablet_manager.cc:1398] T 19b3a90e13194997bde2b9a806f8f2b5 P a1a844ecd71145e4a362f4c42500fbcb: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:58:59.382247 3566 tablet_bootstrap.cc:492] T c2911cf48f384bf09074f7bea2a4c173 P a1a844ecd71145e4a362f4c42500fbcb: Bootstrap starting.
I20240503 01:58:59.382680 3567 tablet_bootstrap.cc:654] T 19b3a90e13194997bde2b9a806f8f2b5 P b63928177fa54227a5ff9929265ab9dd: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:59.383215 3566 tablet_bootstrap.cc:654] T c2911cf48f384bf09074f7bea2a4c173 P a1a844ecd71145e4a362f4c42500fbcb: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:59.384284 3567 tablet_bootstrap.cc:492] T 19b3a90e13194997bde2b9a806f8f2b5 P b63928177fa54227a5ff9929265ab9dd: No bootstrap required, opened a new log
I20240503 01:58:59.384366 3567 ts_tablet_manager.cc:1367] T 19b3a90e13194997bde2b9a806f8f2b5 P b63928177fa54227a5ff9929265ab9dd: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:58:59.384604 3562 tablet_bootstrap.cc:492] T c2911cf48f384bf09074f7bea2a4c173 P b40a9a7b30d34a6db007ed8febd8a759: No bootstrap required, opened a new log
I20240503 01:58:59.384702 3562 ts_tablet_manager.cc:1367] T c2911cf48f384bf09074f7bea2a4c173 P b40a9a7b30d34a6db007ed8febd8a759: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 01:58:59.384755 3563 tablet_bootstrap.cc:492] T c2911cf48f384bf09074f7bea2a4c173 P 6568f69a78c44bb985162d2b7d0a9376: No bootstrap required, opened a new log
I20240503 01:58:59.384837 3563 ts_tablet_manager.cc:1367] T c2911cf48f384bf09074f7bea2a4c173 P 6568f69a78c44bb985162d2b7d0a9376: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 01:58:59.384855 3567 raft_consensus.cc:348] T 19b3a90e13194997bde2b9a806f8f2b5 P b63928177fa54227a5ff9929265ab9dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } }
I20240503 01:58:59.384940 3567 raft_consensus.cc:374] T 19b3a90e13194997bde2b9a806f8f2b5 P b63928177fa54227a5ff9929265ab9dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:59.384972 3567 raft_consensus.cc:729] T 19b3a90e13194997bde2b9a806f8f2b5 P b63928177fa54227a5ff9929265ab9dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b63928177fa54227a5ff9929265ab9dd, State: Initialized, Role: FOLLOWER
I20240503 01:58:59.385073 3567 consensus_queue.cc:260] T 19b3a90e13194997bde2b9a806f8f2b5 P b63928177fa54227a5ff9929265ab9dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } }
I20240503 01:58:59.385124 3562 raft_consensus.cc:348] T c2911cf48f384bf09074f7bea2a4c173 P b40a9a7b30d34a6db007ed8febd8a759 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } }
I20240503 01:58:59.385229 3562 raft_consensus.cc:374] T c2911cf48f384bf09074f7bea2a4c173 P b40a9a7b30d34a6db007ed8febd8a759 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:59.385260 3562 raft_consensus.cc:729] T c2911cf48f384bf09074f7bea2a4c173 P b40a9a7b30d34a6db007ed8febd8a759 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b40a9a7b30d34a6db007ed8febd8a759, State: Initialized, Role: FOLLOWER
I20240503 01:58:59.385324 3567 ts_tablet_manager.cc:1398] T 19b3a90e13194997bde2b9a806f8f2b5 P b63928177fa54227a5ff9929265ab9dd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:58:59.385286 3563 raft_consensus.cc:348] T c2911cf48f384bf09074f7bea2a4c173 P 6568f69a78c44bb985162d2b7d0a9376 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } }
I20240503 01:58:59.385401 3563 raft_consensus.cc:374] T c2911cf48f384bf09074f7bea2a4c173 P 6568f69a78c44bb985162d2b7d0a9376 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:59.385433 3563 raft_consensus.cc:729] T c2911cf48f384bf09074f7bea2a4c173 P 6568f69a78c44bb985162d2b7d0a9376 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6568f69a78c44bb985162d2b7d0a9376, State: Initialized, Role: FOLLOWER
I20240503 01:58:59.385385 3562 consensus_queue.cc:260] T c2911cf48f384bf09074f7bea2a4c173 P b40a9a7b30d34a6db007ed8febd8a759 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } }
I20240503 01:58:59.385462 3567 tablet_bootstrap.cc:492] T 8f4c3f70f2714cc299bc0d595f838c85 P b63928177fa54227a5ff9929265ab9dd: Bootstrap starting.
I20240503 01:58:59.385538 3563 consensus_queue.cc:260] T c2911cf48f384bf09074f7bea2a4c173 P 6568f69a78c44bb985162d2b7d0a9376 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } }
I20240503 01:58:59.385658 3562 ts_tablet_manager.cc:1398] T c2911cf48f384bf09074f7bea2a4c173 P b40a9a7b30d34a6db007ed8febd8a759: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:58:59.386142 3563 ts_tablet_manager.cc:1398] T c2911cf48f384bf09074f7bea2a4c173 P 6568f69a78c44bb985162d2b7d0a9376: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:58:59.386322 3567 tablet_bootstrap.cc:654] T 8f4c3f70f2714cc299bc0d595f838c85 P b63928177fa54227a5ff9929265ab9dd: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:59.386319 3563 tablet_bootstrap.cc:492] T 19b3a90e13194997bde2b9a806f8f2b5 P 6568f69a78c44bb985162d2b7d0a9376: Bootstrap starting.
I20240503 01:58:59.387017 3566 tablet_bootstrap.cc:492] T c2911cf48f384bf09074f7bea2a4c173 P a1a844ecd71145e4a362f4c42500fbcb: No bootstrap required, opened a new log
I20240503 01:58:59.387162 3566 ts_tablet_manager.cc:1367] T c2911cf48f384bf09074f7bea2a4c173 P a1a844ecd71145e4a362f4c42500fbcb: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 01:58:59.387518 3563 tablet_bootstrap.cc:654] T 19b3a90e13194997bde2b9a806f8f2b5 P 6568f69a78c44bb985162d2b7d0a9376: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:59.387645 3567 tablet_bootstrap.cc:492] T 8f4c3f70f2714cc299bc0d595f838c85 P b63928177fa54227a5ff9929265ab9dd: No bootstrap required, opened a new log
I20240503 01:58:59.387733 3567 ts_tablet_manager.cc:1367] T 8f4c3f70f2714cc299bc0d595f838c85 P b63928177fa54227a5ff9929265ab9dd: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:58:59.387961 3566 raft_consensus.cc:348] T c2911cf48f384bf09074f7bea2a4c173 P a1a844ecd71145e4a362f4c42500fbcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } }
I20240503 01:58:59.388098 3566 raft_consensus.cc:374] T c2911cf48f384bf09074f7bea2a4c173 P a1a844ecd71145e4a362f4c42500fbcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:59.388149 3566 raft_consensus.cc:729] T c2911cf48f384bf09074f7bea2a4c173 P a1a844ecd71145e4a362f4c42500fbcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1a844ecd71145e4a362f4c42500fbcb, State: Initialized, Role: FOLLOWER
I20240503 01:58:59.388202 3567 raft_consensus.cc:348] T 8f4c3f70f2714cc299bc0d595f838c85 P b63928177fa54227a5ff9929265ab9dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } }
I20240503 01:58:59.388301 3567 raft_consensus.cc:374] T 8f4c3f70f2714cc299bc0d595f838c85 P b63928177fa54227a5ff9929265ab9dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:59.388342 3567 raft_consensus.cc:729] T 8f4c3f70f2714cc299bc0d595f838c85 P b63928177fa54227a5ff9929265ab9dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b63928177fa54227a5ff9929265ab9dd, State: Initialized, Role: FOLLOWER
I20240503 01:58:59.388327 3566 consensus_queue.cc:260] T c2911cf48f384bf09074f7bea2a4c173 P a1a844ecd71145e4a362f4c42500fbcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } }
I20240503 01:58:59.388458 3567 consensus_queue.cc:260] T 8f4c3f70f2714cc299bc0d595f838c85 P b63928177fa54227a5ff9929265ab9dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } }
I20240503 01:58:59.388747 3567 ts_tablet_manager.cc:1398] T 8f4c3f70f2714cc299bc0d595f838c85 P b63928177fa54227a5ff9929265ab9dd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:58:59.388811 3566 ts_tablet_manager.cc:1398] T c2911cf48f384bf09074f7bea2a4c173 P a1a844ecd71145e4a362f4c42500fbcb: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:58:59.388976 3566 tablet_bootstrap.cc:492] T 8f4c3f70f2714cc299bc0d595f838c85 P a1a844ecd71145e4a362f4c42500fbcb: Bootstrap starting.
I20240503 01:58:59.389524 3563 tablet_bootstrap.cc:492] T 19b3a90e13194997bde2b9a806f8f2b5 P 6568f69a78c44bb985162d2b7d0a9376: No bootstrap required, opened a new log
I20240503 01:58:59.389612 3563 ts_tablet_manager.cc:1367] T 19b3a90e13194997bde2b9a806f8f2b5 P 6568f69a78c44bb985162d2b7d0a9376: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:58:59.390031 3563 raft_consensus.cc:348] T 19b3a90e13194997bde2b9a806f8f2b5 P 6568f69a78c44bb985162d2b7d0a9376 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } }
I20240503 01:58:59.390149 3563 raft_consensus.cc:374] T 19b3a90e13194997bde2b9a806f8f2b5 P 6568f69a78c44bb985162d2b7d0a9376 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:59.390182 3563 raft_consensus.cc:729] T 19b3a90e13194997bde2b9a806f8f2b5 P 6568f69a78c44bb985162d2b7d0a9376 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6568f69a78c44bb985162d2b7d0a9376, State: Initialized, Role: FOLLOWER
I20240503 01:58:59.390283 3563 consensus_queue.cc:260] T 19b3a90e13194997bde2b9a806f8f2b5 P 6568f69a78c44bb985162d2b7d0a9376 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } }
I20240503 01:58:59.390568 3566 tablet_bootstrap.cc:654] T 8f4c3f70f2714cc299bc0d595f838c85 P a1a844ecd71145e4a362f4c42500fbcb: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:59.390580 3563 ts_tablet_manager.cc:1398] T 19b3a90e13194997bde2b9a806f8f2b5 P 6568f69a78c44bb985162d2b7d0a9376: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:58:59.391566 3566 tablet_bootstrap.cc:492] T 8f4c3f70f2714cc299bc0d595f838c85 P a1a844ecd71145e4a362f4c42500fbcb: No bootstrap required, opened a new log
I20240503 01:58:59.391652 3566 ts_tablet_manager.cc:1367] T 8f4c3f70f2714cc299bc0d595f838c85 P a1a844ecd71145e4a362f4c42500fbcb: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 01:58:59.392096 3566 raft_consensus.cc:348] T 8f4c3f70f2714cc299bc0d595f838c85 P a1a844ecd71145e4a362f4c42500fbcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } }
I20240503 01:58:59.392190 3566 raft_consensus.cc:374] T 8f4c3f70f2714cc299bc0d595f838c85 P a1a844ecd71145e4a362f4c42500fbcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:59.392225 3566 raft_consensus.cc:729] T 8f4c3f70f2714cc299bc0d595f838c85 P a1a844ecd71145e4a362f4c42500fbcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1a844ecd71145e4a362f4c42500fbcb, State: Initialized, Role: FOLLOWER
I20240503 01:58:59.392346 3566 consensus_queue.cc:260] T 8f4c3f70f2714cc299bc0d595f838c85 P a1a844ecd71145e4a362f4c42500fbcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } }
I20240503 01:58:59.392603 3566 ts_tablet_manager.cc:1398] T 8f4c3f70f2714cc299bc0d595f838c85 P a1a844ecd71145e4a362f4c42500fbcb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:58:59.429131 3408 raft_consensus.cc:482] T 8f4c3f70f2714cc299bc0d595f838c85 P 6568f69a78c44bb985162d2b7d0a9376 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:58:59.429263 3408 raft_consensus.cc:504] T 8f4c3f70f2714cc299bc0d595f838c85 P 6568f69a78c44bb985162d2b7d0a9376 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } }
I20240503 01:58:59.429801 3408 leader_election.cc:288] T 8f4c3f70f2714cc299bc0d595f838c85 P 6568f69a78c44bb985162d2b7d0a9376 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52c196c15e3348b985c689652bc49918 (127.0.99.132:42887), b63928177fa54227a5ff9929265ab9dd (127.0.99.129:36425), 327aa26a8e434a609ebcb382aa665fce (127.0.99.133:38853), a1a844ecd71145e4a362f4c42500fbcb (127.0.99.130:46033)
I20240503 01:58:59.430109 2923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f4c3f70f2714cc299bc0d595f838c85" candidate_uuid: "6568f69a78c44bb985162d2b7d0a9376" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "327aa26a8e434a609ebcb382aa665fce" is_pre_election: true
I20240503 01:58:59.430163 2531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f4c3f70f2714cc299bc0d595f838c85" candidate_uuid: "6568f69a78c44bb985162d2b7d0a9376" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1a844ecd71145e4a362f4c42500fbcb" is_pre_election: true
I20240503 01:58:59.430233 2923 raft_consensus.cc:2454] T 8f4c3f70f2714cc299bc0d595f838c85 P 327aa26a8e434a609ebcb382aa665fce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6568f69a78c44bb985162d2b7d0a9376 in term 0.
I20240503 01:58:59.430307 2531 raft_consensus.cc:2454] T 8f4c3f70f2714cc299bc0d595f838c85 P a1a844ecd71145e4a362f4c42500fbcb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6568f69a78c44bb985162d2b7d0a9376 in term 0.
I20240503 01:58:59.430351 2398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f4c3f70f2714cc299bc0d595f838c85" candidate_uuid: "6568f69a78c44bb985162d2b7d0a9376" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b63928177fa54227a5ff9929265ab9dd" is_pre_election: true
I20240503 01:58:59.430498 2398 raft_consensus.cc:2454] T 8f4c3f70f2714cc299bc0d595f838c85 P b63928177fa54227a5ff9929265ab9dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6568f69a78c44bb985162d2b7d0a9376 in term 0.
I20240503 01:58:59.430648 2991 leader_election.cc:302] T 8f4c3f70f2714cc299bc0d595f838c85 P 6568f69a78c44bb985162d2b7d0a9376 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 327aa26a8e434a609ebcb382aa665fce, 6568f69a78c44bb985162d2b7d0a9376, a1a844ecd71145e4a362f4c42500fbcb; no voters:
I20240503 01:58:59.430675 2791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f4c3f70f2714cc299bc0d595f838c85" candidate_uuid: "6568f69a78c44bb985162d2b7d0a9376" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52c196c15e3348b985c689652bc49918" is_pre_election: true
I20240503 01:58:59.430795 2791 raft_consensus.cc:2454] T 8f4c3f70f2714cc299bc0d595f838c85 P 52c196c15e3348b985c689652bc49918 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6568f69a78c44bb985162d2b7d0a9376 in term 0.
I20240503 01:58:59.430907 3408 raft_consensus.cc:2781] T 8f4c3f70f2714cc299bc0d595f838c85 P 6568f69a78c44bb985162d2b7d0a9376 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:58:59.430955 3408 raft_consensus.cc:482] T 8f4c3f70f2714cc299bc0d595f838c85 P 6568f69a78c44bb985162d2b7d0a9376 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:58:59.430986 3408 raft_consensus.cc:3037] T 8f4c3f70f2714cc299bc0d595f838c85 P 6568f69a78c44bb985162d2b7d0a9376 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:59.431983 3408 raft_consensus.cc:504] T 8f4c3f70f2714cc299bc0d595f838c85 P 6568f69a78c44bb985162d2b7d0a9376 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } }
I20240503 01:58:59.432413 3408 leader_election.cc:288] T 8f4c3f70f2714cc299bc0d595f838c85 P 6568f69a78c44bb985162d2b7d0a9376 [CANDIDATE]: Term 1 election: Requested vote from peers 52c196c15e3348b985c689652bc49918 (127.0.99.132:42887), b63928177fa54227a5ff9929265ab9dd (127.0.99.129:36425), 327aa26a8e434a609ebcb382aa665fce (127.0.99.133:38853), a1a844ecd71145e4a362f4c42500fbcb (127.0.99.130:46033)
I20240503 01:58:59.432710 2923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f4c3f70f2714cc299bc0d595f838c85" candidate_uuid: "6568f69a78c44bb985162d2b7d0a9376" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "327aa26a8e434a609ebcb382aa665fce"
I20240503 01:58:59.432806 2923 raft_consensus.cc:3037] T 8f4c3f70f2714cc299bc0d595f838c85 P 327aa26a8e434a609ebcb382aa665fce [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:59.432826 2531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f4c3f70f2714cc299bc0d595f838c85" candidate_uuid: "6568f69a78c44bb985162d2b7d0a9376" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1a844ecd71145e4a362f4c42500fbcb"
I20240503 01:58:59.432854 2398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f4c3f70f2714cc299bc0d595f838c85" candidate_uuid: "6568f69a78c44bb985162d2b7d0a9376" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b63928177fa54227a5ff9929265ab9dd"
I20240503 01:58:59.432940 2398 raft_consensus.cc:3037] T 8f4c3f70f2714cc299bc0d595f838c85 P b63928177fa54227a5ff9929265ab9dd [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:59.432937 2531 raft_consensus.cc:3037] T 8f4c3f70f2714cc299bc0d595f838c85 P a1a844ecd71145e4a362f4c42500fbcb [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:59.433857 2923 raft_consensus.cc:2454] T 8f4c3f70f2714cc299bc0d595f838c85 P 327aa26a8e434a609ebcb382aa665fce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6568f69a78c44bb985162d2b7d0a9376 in term 1.
I20240503 01:58:59.433975 2398 raft_consensus.cc:2454] T 8f4c3f70f2714cc299bc0d595f838c85 P b63928177fa54227a5ff9929265ab9dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6568f69a78c44bb985162d2b7d0a9376 in term 1.
I20240503 01:58:59.434041 2531 raft_consensus.cc:2454] T 8f4c3f70f2714cc299bc0d595f838c85 P a1a844ecd71145e4a362f4c42500fbcb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6568f69a78c44bb985162d2b7d0a9376 in term 1.
I20240503 01:58:59.434206 2791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f4c3f70f2714cc299bc0d595f838c85" candidate_uuid: "6568f69a78c44bb985162d2b7d0a9376" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52c196c15e3348b985c689652bc49918"
I20240503 01:58:59.434309 2791 raft_consensus.cc:3037] T 8f4c3f70f2714cc299bc0d595f838c85 P 52c196c15e3348b985c689652bc49918 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:59.435333 2791 raft_consensus.cc:2454] T 8f4c3f70f2714cc299bc0d595f838c85 P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6568f69a78c44bb985162d2b7d0a9376 in term 1.
I20240503 01:58:59.435748 2991 leader_election.cc:302] T 8f4c3f70f2714cc299bc0d595f838c85 P 6568f69a78c44bb985162d2b7d0a9376 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 327aa26a8e434a609ebcb382aa665fce, 6568f69a78c44bb985162d2b7d0a9376, a1a844ecd71145e4a362f4c42500fbcb; no voters:
I20240503 01:58:59.435971 3408 raft_consensus.cc:2781] T 8f4c3f70f2714cc299bc0d595f838c85 P 6568f69a78c44bb985162d2b7d0a9376 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:58:59.436216 3408 raft_consensus.cc:686] T 8f4c3f70f2714cc299bc0d595f838c85 P 6568f69a78c44bb985162d2b7d0a9376 [term 1 LEADER]: Becoming Leader. State: Replica: 6568f69a78c44bb985162d2b7d0a9376, State: Running, Role: LEADER
I20240503 01:58:59.436395 3408 consensus_queue.cc:237] T 8f4c3f70f2714cc299bc0d595f838c85 P 6568f69a78c44bb985162d2b7d0a9376 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } }
I20240503 01:58:59.438604 2251 catalog_manager.cc:5484] T 8f4c3f70f2714cc299bc0d595f838c85 P 6568f69a78c44bb985162d2b7d0a9376 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6568f69a78c44bb985162d2b7d0a9376 (127.0.99.134). New cstate: current_term: 1 leader_uuid: "6568f69a78c44bb985162d2b7d0a9376" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } health_report { overall_health: UNKNOWN } } }
I20240503 01:58:59.485466 3539 raft_consensus.cc:482] T 19b3a90e13194997bde2b9a806f8f2b5 P b63928177fa54227a5ff9929265ab9dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:58:59.485611 3539 raft_consensus.cc:504] T 19b3a90e13194997bde2b9a806f8f2b5 P b63928177fa54227a5ff9929265ab9dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } }
I20240503 01:58:59.486186 3539 leader_election.cc:288] T 19b3a90e13194997bde2b9a806f8f2b5 P b63928177fa54227a5ff9929265ab9dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b40a9a7b30d34a6db007ed8febd8a759 (127.0.99.131:34079), a1a844ecd71145e4a362f4c42500fbcb (127.0.99.130:46033), 52c196c15e3348b985c689652bc49918 (127.0.99.132:42887), 6568f69a78c44bb985162d2b7d0a9376 (127.0.99.134:43603)
I20240503 01:58:59.486485 2661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19b3a90e13194997bde2b9a806f8f2b5" candidate_uuid: "b63928177fa54227a5ff9929265ab9dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b40a9a7b30d34a6db007ed8febd8a759" is_pre_election: true
I20240503 01:58:59.486541 2791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19b3a90e13194997bde2b9a806f8f2b5" candidate_uuid: "b63928177fa54227a5ff9929265ab9dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52c196c15e3348b985c689652bc49918" is_pre_election: true
I20240503 01:58:59.486641 2661 raft_consensus.cc:2454] T 19b3a90e13194997bde2b9a806f8f2b5 P b40a9a7b30d34a6db007ed8febd8a759 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b63928177fa54227a5ff9929265ab9dd in term 0.
I20240503 01:58:59.486690 2791 raft_consensus.cc:2454] T 19b3a90e13194997bde2b9a806f8f2b5 P 52c196c15e3348b985c689652bc49918 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b63928177fa54227a5ff9929265ab9dd in term 0.
I20240503 01:58:59.487264 2531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19b3a90e13194997bde2b9a806f8f2b5" candidate_uuid: "b63928177fa54227a5ff9929265ab9dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1a844ecd71145e4a362f4c42500fbcb" is_pre_election: true
I20240503 01:58:59.487421 2531 raft_consensus.cc:2454] T 19b3a90e13194997bde2b9a806f8f2b5 P a1a844ecd71145e4a362f4c42500fbcb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b63928177fa54227a5ff9929265ab9dd in term 0.
I20240503 01:58:59.487430 2336 leader_election.cc:302] T 19b3a90e13194997bde2b9a806f8f2b5 P b63928177fa54227a5ff9929265ab9dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 52c196c15e3348b985c689652bc49918, b40a9a7b30d34a6db007ed8febd8a759, b63928177fa54227a5ff9929265ab9dd; no voters:
I20240503 01:58:59.487591 3539 raft_consensus.cc:2781] T 19b3a90e13194997bde2b9a806f8f2b5 P b63928177fa54227a5ff9929265ab9dd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:58:59.487661 3539 raft_consensus.cc:482] T 19b3a90e13194997bde2b9a806f8f2b5 P b63928177fa54227a5ff9929265ab9dd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:58:59.487710 3539 raft_consensus.cc:3037] T 19b3a90e13194997bde2b9a806f8f2b5 P b63928177fa54227a5ff9929265ab9dd [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:59.488627 3539 raft_consensus.cc:504] T 19b3a90e13194997bde2b9a806f8f2b5 P b63928177fa54227a5ff9929265ab9dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } }
I20240503 01:58:59.489090 3539 leader_election.cc:288] T 19b3a90e13194997bde2b9a806f8f2b5 P b63928177fa54227a5ff9929265ab9dd [CANDIDATE]: Term 1 election: Requested vote from peers b40a9a7b30d34a6db007ed8febd8a759 (127.0.99.131:34079), a1a844ecd71145e4a362f4c42500fbcb (127.0.99.130:46033), 52c196c15e3348b985c689652bc49918 (127.0.99.132:42887), 6568f69a78c44bb985162d2b7d0a9376 (127.0.99.134:43603)
I20240503 01:58:59.489239 2661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19b3a90e13194997bde2b9a806f8f2b5" candidate_uuid: "b63928177fa54227a5ff9929265ab9dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b40a9a7b30d34a6db007ed8febd8a759"
I20240503 01:58:59.489334 2661 raft_consensus.cc:3037] T 19b3a90e13194997bde2b9a806f8f2b5 P b40a9a7b30d34a6db007ed8febd8a759 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:59.489894 2791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19b3a90e13194997bde2b9a806f8f2b5" candidate_uuid: "b63928177fa54227a5ff9929265ab9dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52c196c15e3348b985c689652bc49918"
I20240503 01:58:59.490015 2791 raft_consensus.cc:3037] T 19b3a90e13194997bde2b9a806f8f2b5 P 52c196c15e3348b985c689652bc49918 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:59.490406 2661 raft_consensus.cc:2454] T 19b3a90e13194997bde2b9a806f8f2b5 P b40a9a7b30d34a6db007ed8febd8a759 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b63928177fa54227a5ff9929265ab9dd in term 1.
I20240503 01:58:59.491001 2531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19b3a90e13194997bde2b9a806f8f2b5" candidate_uuid: "b63928177fa54227a5ff9929265ab9dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1a844ecd71145e4a362f4c42500fbcb"
I20240503 01:58:59.491089 2791 raft_consensus.cc:2454] T 19b3a90e13194997bde2b9a806f8f2b5 P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b63928177fa54227a5ff9929265ab9dd in term 1.
I20240503 01:58:59.491137 2531 raft_consensus.cc:3037] T 19b3a90e13194997bde2b9a806f8f2b5 P a1a844ecd71145e4a362f4c42500fbcb [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:59.491389 2336 leader_election.cc:302] T 19b3a90e13194997bde2b9a806f8f2b5 P b63928177fa54227a5ff9929265ab9dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 52c196c15e3348b985c689652bc49918, b40a9a7b30d34a6db007ed8febd8a759, b63928177fa54227a5ff9929265ab9dd; no voters:
I20240503 01:58:59.491653 3055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19b3a90e13194997bde2b9a806f8f2b5" candidate_uuid: "b63928177fa54227a5ff9929265ab9dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6568f69a78c44bb985162d2b7d0a9376" is_pre_election: true
I20240503 01:58:59.491680 3054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19b3a90e13194997bde2b9a806f8f2b5" candidate_uuid: "b63928177fa54227a5ff9929265ab9dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6568f69a78c44bb985162d2b7d0a9376"
I20240503 01:58:59.491773 3055 raft_consensus.cc:2454] T 19b3a90e13194997bde2b9a806f8f2b5 P 6568f69a78c44bb985162d2b7d0a9376 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b63928177fa54227a5ff9929265ab9dd in term 0.
I20240503 01:58:59.492098 2531 raft_consensus.cc:2454] T 19b3a90e13194997bde2b9a806f8f2b5 P a1a844ecd71145e4a362f4c42500fbcb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b63928177fa54227a5ff9929265ab9dd in term 1.
I20240503 01:58:59.492182 3539 raft_consensus.cc:2781] T 19b3a90e13194997bde2b9a806f8f2b5 P b63928177fa54227a5ff9929265ab9dd [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:58:59.492262 3539 raft_consensus.cc:686] T 19b3a90e13194997bde2b9a806f8f2b5 P b63928177fa54227a5ff9929265ab9dd [term 1 LEADER]: Becoming Leader. State: Replica: b63928177fa54227a5ff9929265ab9dd, State: Running, Role: LEADER
I20240503 01:58:59.492385 3539 consensus_queue.cc:237] T 19b3a90e13194997bde2b9a806f8f2b5 P b63928177fa54227a5ff9929265ab9dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } }
I20240503 01:58:59.494330 2251 catalog_manager.cc:5484] T 19b3a90e13194997bde2b9a806f8f2b5 P b63928177fa54227a5ff9929265ab9dd reported cstate change: term changed from 0 to 1, leader changed from <none> to b63928177fa54227a5ff9929265ab9dd (127.0.99.129). New cstate: current_term: 1 leader_uuid: "b63928177fa54227a5ff9929265ab9dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } health_report { overall_health: UNKNOWN } } }
I20240503 01:58:59.646782 3408 raft_consensus.cc:482] T c2911cf48f384bf09074f7bea2a4c173 P 6568f69a78c44bb985162d2b7d0a9376 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:58:59.646927 3408 raft_consensus.cc:504] T c2911cf48f384bf09074f7bea2a4c173 P 6568f69a78c44bb985162d2b7d0a9376 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } }
I20240503 01:58:59.647632 3408 leader_election.cc:288] T c2911cf48f384bf09074f7bea2a4c173 P 6568f69a78c44bb985162d2b7d0a9376 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b63928177fa54227a5ff9929265ab9dd (127.0.99.129:36425), a1a844ecd71145e4a362f4c42500fbcb (127.0.99.130:46033), 327aa26a8e434a609ebcb382aa665fce (127.0.99.133:38853), b40a9a7b30d34a6db007ed8febd8a759 (127.0.99.131:34079)
I20240503 01:58:59.648108 2531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2911cf48f384bf09074f7bea2a4c173" candidate_uuid: "6568f69a78c44bb985162d2b7d0a9376" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1a844ecd71145e4a362f4c42500fbcb" is_pre_election: true
I20240503 01:58:59.648155 2398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2911cf48f384bf09074f7bea2a4c173" candidate_uuid: "6568f69a78c44bb985162d2b7d0a9376" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b63928177fa54227a5ff9929265ab9dd" is_pre_election: true
I20240503 01:58:59.648265 2531 raft_consensus.cc:2454] T c2911cf48f384bf09074f7bea2a4c173 P a1a844ecd71145e4a362f4c42500fbcb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6568f69a78c44bb985162d2b7d0a9376 in term 0.
I20240503 01:58:59.648314 2398 raft_consensus.cc:2454] T c2911cf48f384bf09074f7bea2a4c173 P b63928177fa54227a5ff9929265ab9dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6568f69a78c44bb985162d2b7d0a9376 in term 0.
I20240503 01:58:59.648761 2991 leader_election.cc:302] T c2911cf48f384bf09074f7bea2a4c173 P 6568f69a78c44bb985162d2b7d0a9376 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 6568f69a78c44bb985162d2b7d0a9376, a1a844ecd71145e4a362f4c42500fbcb, b63928177fa54227a5ff9929265ab9dd; no voters:
I20240503 01:58:59.648952 3408 raft_consensus.cc:2781] T c2911cf48f384bf09074f7bea2a4c173 P 6568f69a78c44bb985162d2b7d0a9376 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:58:59.649013 3408 raft_consensus.cc:482] T c2911cf48f384bf09074f7bea2a4c173 P 6568f69a78c44bb985162d2b7d0a9376 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:58:59.649051 3408 raft_consensus.cc:3037] T c2911cf48f384bf09074f7bea2a4c173 P 6568f69a78c44bb985162d2b7d0a9376 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:59.649003 2923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2911cf48f384bf09074f7bea2a4c173" candidate_uuid: "6568f69a78c44bb985162d2b7d0a9376" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "327aa26a8e434a609ebcb382aa665fce" is_pre_election: true
I20240503 01:58:59.649159 2923 raft_consensus.cc:2454] T c2911cf48f384bf09074f7bea2a4c173 P 327aa26a8e434a609ebcb382aa665fce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6568f69a78c44bb985162d2b7d0a9376 in term 0.
I20240503 01:58:59.650051 3408 raft_consensus.cc:504] T c2911cf48f384bf09074f7bea2a4c173 P 6568f69a78c44bb985162d2b7d0a9376 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } }
I20240503 01:58:59.650506 3408 leader_election.cc:288] T c2911cf48f384bf09074f7bea2a4c173 P 6568f69a78c44bb985162d2b7d0a9376 [CANDIDATE]: Term 1 election: Requested vote from peers b63928177fa54227a5ff9929265ab9dd (127.0.99.129:36425), a1a844ecd71145e4a362f4c42500fbcb (127.0.99.130:46033), 327aa26a8e434a609ebcb382aa665fce (127.0.99.133:38853), b40a9a7b30d34a6db007ed8febd8a759 (127.0.99.131:34079)
I20240503 01:58:59.650877 2531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2911cf48f384bf09074f7bea2a4c173" candidate_uuid: "6568f69a78c44bb985162d2b7d0a9376" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1a844ecd71145e4a362f4c42500fbcb"
I20240503 01:58:59.650884 2398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2911cf48f384bf09074f7bea2a4c173" candidate_uuid: "6568f69a78c44bb985162d2b7d0a9376" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b63928177fa54227a5ff9929265ab9dd"
I20240503 01:58:59.650883 2923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2911cf48f384bf09074f7bea2a4c173" candidate_uuid: "6568f69a78c44bb985162d2b7d0a9376" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "327aa26a8e434a609ebcb382aa665fce"
I20240503 01:58:59.650990 2531 raft_consensus.cc:3037] T c2911cf48f384bf09074f7bea2a4c173 P a1a844ecd71145e4a362f4c42500fbcb [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:59.651067 2923 raft_consensus.cc:3037] T c2911cf48f384bf09074f7bea2a4c173 P 327aa26a8e434a609ebcb382aa665fce [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:59.651067 2398 raft_consensus.cc:3037] T c2911cf48f384bf09074f7bea2a4c173 P b63928177fa54227a5ff9929265ab9dd [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:59.652127 2531 raft_consensus.cc:2454] T c2911cf48f384bf09074f7bea2a4c173 P a1a844ecd71145e4a362f4c42500fbcb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6568f69a78c44bb985162d2b7d0a9376 in term 1.
I20240503 01:58:59.652299 2398 raft_consensus.cc:2454] T c2911cf48f384bf09074f7bea2a4c173 P b63928177fa54227a5ff9929265ab9dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6568f69a78c44bb985162d2b7d0a9376 in term 1.
I20240503 01:58:59.652300 2923 raft_consensus.cc:2454] T c2911cf48f384bf09074f7bea2a4c173 P 327aa26a8e434a609ebcb382aa665fce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6568f69a78c44bb985162d2b7d0a9376 in term 1.
I20240503 01:58:59.652705 2991 leader_election.cc:302] T c2911cf48f384bf09074f7bea2a4c173 P 6568f69a78c44bb985162d2b7d0a9376 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 6568f69a78c44bb985162d2b7d0a9376, a1a844ecd71145e4a362f4c42500fbcb, b63928177fa54227a5ff9929265ab9dd; no voters:
I20240503 01:58:59.652941 3408 raft_consensus.cc:2781] T c2911cf48f384bf09074f7bea2a4c173 P 6568f69a78c44bb985162d2b7d0a9376 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:58:59.653060 3408 raft_consensus.cc:686] T c2911cf48f384bf09074f7bea2a4c173 P 6568f69a78c44bb985162d2b7d0a9376 [term 1 LEADER]: Becoming Leader. State: Replica: 6568f69a78c44bb985162d2b7d0a9376, State: Running, Role: LEADER
I20240503 01:58:59.653012 2662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2911cf48f384bf09074f7bea2a4c173" candidate_uuid: "6568f69a78c44bb985162d2b7d0a9376" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b40a9a7b30d34a6db007ed8febd8a759"
I20240503 01:58:59.653198 2662 raft_consensus.cc:3037] T c2911cf48f384bf09074f7bea2a4c173 P b40a9a7b30d34a6db007ed8febd8a759 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:59.653272 3408 consensus_queue.cc:237] T c2911cf48f384bf09074f7bea2a4c173 P 6568f69a78c44bb985162d2b7d0a9376 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } }
I20240503 01:58:59.653005 2661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2911cf48f384bf09074f7bea2a4c173" candidate_uuid: "6568f69a78c44bb985162d2b7d0a9376" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b40a9a7b30d34a6db007ed8febd8a759" is_pre_election: true
I20240503 01:58:59.654135 2662 raft_consensus.cc:2454] T c2911cf48f384bf09074f7bea2a4c173 P b40a9a7b30d34a6db007ed8febd8a759 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6568f69a78c44bb985162d2b7d0a9376 in term 1.
I20240503 01:58:59.655537 2251 catalog_manager.cc:5484] T c2911cf48f384bf09074f7bea2a4c173 P 6568f69a78c44bb985162d2b7d0a9376 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6568f69a78c44bb985162d2b7d0a9376 (127.0.99.134). New cstate: current_term: 1 leader_uuid: "6568f69a78c44bb985162d2b7d0a9376" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } health_report { overall_health: UNKNOWN } } }
W20240503 01:58:59.789847 3591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:58:59.790088 3591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:58:59.793567 3591 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240503 01:58:59.833484 3408 consensus_queue.cc:1035] T 8f4c3f70f2714cc299bc0d595f838c85 P 6568f69a78c44bb985162d2b7d0a9376 [LEADER]: Connected to new peer: Peer: permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:59.875975 3408 consensus_queue.cc:1035] T 8f4c3f70f2714cc299bc0d595f838c85 P 6568f69a78c44bb985162d2b7d0a9376 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:59.885315 3616 consensus_queue.cc:1035] T 8f4c3f70f2714cc299bc0d595f838c85 P 6568f69a78c44bb985162d2b7d0a9376 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:59.885551 3583 consensus_queue.cc:1035] T 8f4c3f70f2714cc299bc0d595f838c85 P 6568f69a78c44bb985162d2b7d0a9376 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:59.897265 3474 consensus_queue.cc:1035] T 19b3a90e13194997bde2b9a806f8f2b5 P b63928177fa54227a5ff9929265ab9dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:59.905405 2662 raft_consensus.cc:1261] T 19b3a90e13194997bde2b9a806f8f2b5 P b40a9a7b30d34a6db007ed8febd8a759 [term 1 FOLLOWER]: Refusing update from remote peer b63928177fa54227a5ff9929265ab9dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:58:59.906405 3474 consensus_queue.cc:1035] T 19b3a90e13194997bde2b9a806f8f2b5 P b63928177fa54227a5ff9929265ab9dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:59.907810 3054 raft_consensus.cc:3037] T 19b3a90e13194997bde2b9a806f8f2b5 P 6568f69a78c44bb985162d2b7d0a9376 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:59.907927 2791 raft_consensus.cc:1261] T 19b3a90e13194997bde2b9a806f8f2b5 P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Refusing update from remote peer b63928177fa54227a5ff9929265ab9dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:58:59.909111 3054 raft_consensus.cc:1261] T 19b3a90e13194997bde2b9a806f8f2b5 P 6568f69a78c44bb985162d2b7d0a9376 [term 1 FOLLOWER]: Refusing update from remote peer b63928177fa54227a5ff9929265ab9dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:58:59.909647 3474 consensus_queue.cc:1035] T 19b3a90e13194997bde2b9a806f8f2b5 P b63928177fa54227a5ff9929265ab9dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:59.913076 3539 consensus_queue.cc:1035] T 19b3a90e13194997bde2b9a806f8f2b5 P b63928177fa54227a5ff9929265ab9dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:59.934718 2923 raft_consensus.cc:1261] T c2911cf48f384bf09074f7bea2a4c173 P 327aa26a8e434a609ebcb382aa665fce [term 1 FOLLOWER]: Refusing update from remote peer 6568f69a78c44bb985162d2b7d0a9376: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:58:59.934852 2398 raft_consensus.cc:1261] T c2911cf48f384bf09074f7bea2a4c173 P b63928177fa54227a5ff9929265ab9dd [term 1 FOLLOWER]: Refusing update from remote peer 6568f69a78c44bb985162d2b7d0a9376: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:58:59.935360 2661 raft_consensus.cc:1261] T c2911cf48f384bf09074f7bea2a4c173 P b40a9a7b30d34a6db007ed8febd8a759 [term 1 FOLLOWER]: Refusing update from remote peer 6568f69a78c44bb985162d2b7d0a9376: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:58:59.936038 3616 consensus_queue.cc:1035] T c2911cf48f384bf09074f7bea2a4c173 P 6568f69a78c44bb985162d2b7d0a9376 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:59.936465 2531 raft_consensus.cc:1261] T c2911cf48f384bf09074f7bea2a4c173 P a1a844ecd71145e4a362f4c42500fbcb [term 1 FOLLOWER]: Refusing update from remote peer 6568f69a78c44bb985162d2b7d0a9376: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:58:59.936960 3583 consensus_queue.cc:1035] T c2911cf48f384bf09074f7bea2a4c173 P 6568f69a78c44bb985162d2b7d0a9376 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:59.937335 3583 consensus_queue.cc:1035] T c2911cf48f384bf09074f7bea2a4c173 P 6568f69a78c44bb985162d2b7d0a9376 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 01:58:59.942926 2598 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:39501: connect: Connection refused (error 111) [suppressed 11 similar messages]
I20240503 01:58:59.944703 3408 consensus_queue.cc:1035] T c2911cf48f384bf09074f7bea2a4c173 P 6568f69a78c44bb985162d2b7d0a9376 [LEADER]: Connected to new peer: Peer: permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 187.576 ms
time per row: 0.0937881 ms
I20240503 01:59:00.034583 2251 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53804:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 5
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:59:00.040896 2904 tablet_service.cc:1467] Processing CreateTablet for tablet 5fd1010447524b6695abb98192dc54f1 (DEFAULT_TABLE table=rebalance_test_table_3 [id=f7b0d84dbe0541aaa2606d6e366719c8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:00.040896 2642 tablet_service.cc:1467] Processing CreateTablet for tablet 5fd1010447524b6695abb98192dc54f1 (DEFAULT_TABLE table=rebalance_test_table_3 [id=f7b0d84dbe0541aaa2606d6e366719c8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:00.041217 2904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fd1010447524b6695abb98192dc54f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:00.041229 2642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fd1010447524b6695abb98192dc54f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:00.043879 2378 tablet_service.cc:1467] Processing CreateTablet for tablet 8682bff56e604cc3b7e9b07c8ba3fca5 (DEFAULT_TABLE table=rebalance_test_table_3 [id=f7b0d84dbe0541aaa2606d6e366719c8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:00.043882 2771 tablet_service.cc:1467] Processing CreateTablet for tablet 10ecacdfb9a44d068b0df24e991612ee (DEFAULT_TABLE table=rebalance_test_table_3 [id=f7b0d84dbe0541aaa2606d6e366719c8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:00.043987 2773 tablet_service.cc:1467] Processing CreateTablet for tablet 5fd1010447524b6695abb98192dc54f1 (DEFAULT_TABLE table=rebalance_test_table_3 [id=f7b0d84dbe0541aaa2606d6e366719c8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:00.044183 2378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8682bff56e604cc3b7e9b07c8ba3fca5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:00.044330 2771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10ecacdfb9a44d068b0df24e991612ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:00.044502 2380 tablet_service.cc:1467] Processing CreateTablet for tablet 5fd1010447524b6695abb98192dc54f1 (DEFAULT_TABLE table=rebalance_test_table_3 [id=f7b0d84dbe0541aaa2606d6e366719c8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:00.044737 2380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fd1010447524b6695abb98192dc54f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:00.045202 2772 tablet_service.cc:1467] Processing CreateTablet for tablet 8682bff56e604cc3b7e9b07c8ba3fca5 (DEFAULT_TABLE table=rebalance_test_table_3 [id=f7b0d84dbe0541aaa2606d6e366719c8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:00.045492 2772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8682bff56e604cc3b7e9b07c8ba3fca5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:00.046715 2773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fd1010447524b6695abb98192dc54f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:00.046763 2379 tablet_service.cc:1467] Processing CreateTablet for tablet 10ecacdfb9a44d068b0df24e991612ee (DEFAULT_TABLE table=rebalance_test_table_3 [id=f7b0d84dbe0541aaa2606d6e366719c8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:00.046984 2379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10ecacdfb9a44d068b0df24e991612ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:00.049897 2902 tablet_service.cc:1467] Processing CreateTablet for tablet 10ecacdfb9a44d068b0df24e991612ee (DEFAULT_TABLE table=rebalance_test_table_3 [id=f7b0d84dbe0541aaa2606d6e366719c8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:00.049939 2509 tablet_service.cc:1467] Processing CreateTablet for tablet 10ecacdfb9a44d068b0df24e991612ee (DEFAULT_TABLE table=rebalance_test_table_3 [id=f7b0d84dbe0541aaa2606d6e366719c8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:00.050154 2902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10ecacdfb9a44d068b0df24e991612ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:00.050201 2509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10ecacdfb9a44d068b0df24e991612ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:00.050698 2641 tablet_service.cc:1467] Processing CreateTablet for tablet 10ecacdfb9a44d068b0df24e991612ee (DEFAULT_TABLE table=rebalance_test_table_3 [id=f7b0d84dbe0541aaa2606d6e366719c8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:00.051013 2641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10ecacdfb9a44d068b0df24e991612ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:00.051898 3669 tablet_bootstrap.cc:492] T 8682bff56e604cc3b7e9b07c8ba3fca5 P b63928177fa54227a5ff9929265ab9dd: Bootstrap starting.
I20240503 01:59:00.053251 2642 tablet_service.cc:1467] Processing CreateTablet for tablet 8682bff56e604cc3b7e9b07c8ba3fca5 (DEFAULT_TABLE table=rebalance_test_table_3 [id=f7b0d84dbe0541aaa2606d6e366719c8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:00.053498 3669 tablet_bootstrap.cc:654] T 8682bff56e604cc3b7e9b07c8ba3fca5 P b63928177fa54227a5ff9929265ab9dd: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:00.053568 2642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8682bff56e604cc3b7e9b07c8ba3fca5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:00.054112 3035 tablet_service.cc:1467] Processing CreateTablet for tablet 5fd1010447524b6695abb98192dc54f1 (DEFAULT_TABLE table=rebalance_test_table_3 [id=f7b0d84dbe0541aaa2606d6e366719c8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:00.054383 3035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fd1010447524b6695abb98192dc54f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:00.056185 3668 tablet_bootstrap.cc:492] T 5fd1010447524b6695abb98192dc54f1 P b40a9a7b30d34a6db007ed8febd8a759: Bootstrap starting.
I20240503 01:59:00.056485 3670 tablet_bootstrap.cc:492] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 52c196c15e3348b985c689652bc49918: Bootstrap starting.
I20240503 01:59:00.056828 3671 tablet_bootstrap.cc:492] T 10ecacdfb9a44d068b0df24e991612ee P a1a844ecd71145e4a362f4c42500fbcb: Bootstrap starting.
I20240503 01:59:00.056880 3034 tablet_service.cc:1467] Processing CreateTablet for tablet 8682bff56e604cc3b7e9b07c8ba3fca5 (DEFAULT_TABLE table=rebalance_test_table_3 [id=f7b0d84dbe0541aaa2606d6e366719c8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:00.057013 3673 tablet_bootstrap.cc:492] T 5fd1010447524b6695abb98192dc54f1 P 6568f69a78c44bb985162d2b7d0a9376: Bootstrap starting.
I20240503 01:59:00.057130 3034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8682bff56e604cc3b7e9b07c8ba3fca5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:00.057283 3668 tablet_bootstrap.cc:654] T 5fd1010447524b6695abb98192dc54f1 P b40a9a7b30d34a6db007ed8febd8a759: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:00.057835 3671 tablet_bootstrap.cc:654] T 10ecacdfb9a44d068b0df24e991612ee P a1a844ecd71145e4a362f4c42500fbcb: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:00.059187 3670 tablet_bootstrap.cc:654] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 52c196c15e3348b985c689652bc49918: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:00.058476 3673 tablet_bootstrap.cc:654] T 5fd1010447524b6695abb98192dc54f1 P 6568f69a78c44bb985162d2b7d0a9376: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:00.059192 2904 tablet_service.cc:1467] Processing CreateTablet for tablet 8682bff56e604cc3b7e9b07c8ba3fca5 (DEFAULT_TABLE table=rebalance_test_table_3 [id=f7b0d84dbe0541aaa2606d6e366719c8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:00.059775 2904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8682bff56e604cc3b7e9b07c8ba3fca5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:00.060739 3670 tablet_bootstrap.cc:492] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 52c196c15e3348b985c689652bc49918: No bootstrap required, opened a new log
I20240503 01:59:00.060839 3670 ts_tablet_manager.cc:1367] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 52c196c15e3348b985c689652bc49918: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 01:59:00.061337 3670 raft_consensus.cc:348] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 52c196c15e3348b985c689652bc49918 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } }
I20240503 01:59:00.061425 3673 tablet_bootstrap.cc:492] T 5fd1010447524b6695abb98192dc54f1 P 6568f69a78c44bb985162d2b7d0a9376: No bootstrap required, opened a new log
I20240503 01:59:00.061473 3670 raft_consensus.cc:374] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 52c196c15e3348b985c689652bc49918 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:00.061523 3670 raft_consensus.cc:729] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 52c196c15e3348b985c689652bc49918 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52c196c15e3348b985c689652bc49918, State: Initialized, Role: FOLLOWER
I20240503 01:59:00.061738 3667 tablet_bootstrap.cc:492] T 5fd1010447524b6695abb98192dc54f1 P 327aa26a8e434a609ebcb382aa665fce: Bootstrap starting.
I20240503 01:59:00.061760 3673 ts_tablet_manager.cc:1367] T 5fd1010447524b6695abb98192dc54f1 P 6568f69a78c44bb985162d2b7d0a9376: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 01:59:00.062036 3671 tablet_bootstrap.cc:492] T 10ecacdfb9a44d068b0df24e991612ee P a1a844ecd71145e4a362f4c42500fbcb: No bootstrap required, opened a new log
I20240503 01:59:00.062140 3671 ts_tablet_manager.cc:1367] T 10ecacdfb9a44d068b0df24e991612ee P a1a844ecd71145e4a362f4c42500fbcb: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 01:59:00.062450 3673 raft_consensus.cc:348] T 5fd1010447524b6695abb98192dc54f1 P 6568f69a78c44bb985162d2b7d0a9376 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } }
I20240503 01:59:00.062638 3673 raft_consensus.cc:374] T 5fd1010447524b6695abb98192dc54f1 P 6568f69a78c44bb985162d2b7d0a9376 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:00.062568 3671 raft_consensus.cc:348] T 10ecacdfb9a44d068b0df24e991612ee P a1a844ecd71145e4a362f4c42500fbcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } }
I20240503 01:59:00.062675 3667 tablet_bootstrap.cc:654] T 5fd1010447524b6695abb98192dc54f1 P 327aa26a8e434a609ebcb382aa665fce: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:00.062687 3671 raft_consensus.cc:374] T 10ecacdfb9a44d068b0df24e991612ee P a1a844ecd71145e4a362f4c42500fbcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:00.062723 3671 raft_consensus.cc:729] T 10ecacdfb9a44d068b0df24e991612ee P a1a844ecd71145e4a362f4c42500fbcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1a844ecd71145e4a362f4c42500fbcb, State: Initialized, Role: FOLLOWER
I20240503 01:59:00.062897 3671 consensus_queue.cc:260] T 10ecacdfb9a44d068b0df24e991612ee P a1a844ecd71145e4a362f4c42500fbcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } }
I20240503 01:59:00.063081 3673 raft_consensus.cc:729] T 5fd1010447524b6695abb98192dc54f1 P 6568f69a78c44bb985162d2b7d0a9376 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6568f69a78c44bb985162d2b7d0a9376, State: Initialized, Role: FOLLOWER
I20240503 01:59:00.063334 3671 ts_tablet_manager.cc:1398] T 10ecacdfb9a44d068b0df24e991612ee P a1a844ecd71145e4a362f4c42500fbcb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:00.063503 3673 consensus_queue.cc:260] T 5fd1010447524b6695abb98192dc54f1 P 6568f69a78c44bb985162d2b7d0a9376 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } }
I20240503 01:59:00.063474 3670 consensus_queue.cc:260] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 52c196c15e3348b985c689652bc49918 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } }
I20240503 01:59:00.063987 3670 ts_tablet_manager.cc:1398] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 52c196c15e3348b985c689652bc49918: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 01:59:00.064174 3670 tablet_bootstrap.cc:492] T 5fd1010447524b6695abb98192dc54f1 P 52c196c15e3348b985c689652bc49918: Bootstrap starting.
I20240503 01:59:00.064388 3673 ts_tablet_manager.cc:1398] T 5fd1010447524b6695abb98192dc54f1 P 6568f69a78c44bb985162d2b7d0a9376: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:00.064572 3673 tablet_bootstrap.cc:492] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 6568f69a78c44bb985162d2b7d0a9376: Bootstrap starting.
I20240503 01:59:00.064617 3668 tablet_bootstrap.cc:492] T 5fd1010447524b6695abb98192dc54f1 P b40a9a7b30d34a6db007ed8febd8a759: No bootstrap required, opened a new log
I20240503 01:59:00.064821 3668 ts_tablet_manager.cc:1367] T 5fd1010447524b6695abb98192dc54f1 P b40a9a7b30d34a6db007ed8febd8a759: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.003s
I20240503 01:59:00.065106 3670 tablet_bootstrap.cc:654] T 5fd1010447524b6695abb98192dc54f1 P 52c196c15e3348b985c689652bc49918: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:00.065387 3668 raft_consensus.cc:348] T 5fd1010447524b6695abb98192dc54f1 P b40a9a7b30d34a6db007ed8febd8a759 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } }
I20240503 01:59:00.065526 3673 tablet_bootstrap.cc:654] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 6568f69a78c44bb985162d2b7d0a9376: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:00.065531 3668 raft_consensus.cc:374] T 5fd1010447524b6695abb98192dc54f1 P b40a9a7b30d34a6db007ed8febd8a759 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:00.065577 3668 raft_consensus.cc:729] T 5fd1010447524b6695abb98192dc54f1 P b40a9a7b30d34a6db007ed8febd8a759 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b40a9a7b30d34a6db007ed8febd8a759, State: Initialized, Role: FOLLOWER
I20240503 01:59:00.065748 3668 consensus_queue.cc:260] T 5fd1010447524b6695abb98192dc54f1 P b40a9a7b30d34a6db007ed8febd8a759 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } }
I20240503 01:59:00.066187 3668 ts_tablet_manager.cc:1398] T 5fd1010447524b6695abb98192dc54f1 P b40a9a7b30d34a6db007ed8febd8a759: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:00.066238 3670 tablet_bootstrap.cc:492] T 5fd1010447524b6695abb98192dc54f1 P 52c196c15e3348b985c689652bc49918: No bootstrap required, opened a new log
I20240503 01:59:00.066299 3670 ts_tablet_manager.cc:1367] T 5fd1010447524b6695abb98192dc54f1 P 52c196c15e3348b985c689652bc49918: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:00.066468 3668 tablet_bootstrap.cc:492] T 10ecacdfb9a44d068b0df24e991612ee P b40a9a7b30d34a6db007ed8febd8a759: Bootstrap starting.
I20240503 01:59:00.066606 3670 raft_consensus.cc:348] T 5fd1010447524b6695abb98192dc54f1 P 52c196c15e3348b985c689652bc49918 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } }
I20240503 01:59:00.066692 3670 raft_consensus.cc:374] T 5fd1010447524b6695abb98192dc54f1 P 52c196c15e3348b985c689652bc49918 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:00.066735 3670 raft_consensus.cc:729] T 5fd1010447524b6695abb98192dc54f1 P 52c196c15e3348b985c689652bc49918 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52c196c15e3348b985c689652bc49918, State: Initialized, Role: FOLLOWER
I20240503 01:59:00.066874 3670 consensus_queue.cc:260] T 5fd1010447524b6695abb98192dc54f1 P 52c196c15e3348b985c689652bc49918 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } }
I20240503 01:59:00.067142 3670 ts_tablet_manager.cc:1398] T 5fd1010447524b6695abb98192dc54f1 P 52c196c15e3348b985c689652bc49918: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:00.067293 3670 tablet_bootstrap.cc:492] T 10ecacdfb9a44d068b0df24e991612ee P 52c196c15e3348b985c689652bc49918: Bootstrap starting.
I20240503 01:59:00.067767 3668 tablet_bootstrap.cc:654] T 10ecacdfb9a44d068b0df24e991612ee P b40a9a7b30d34a6db007ed8febd8a759: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:00.068135 3670 tablet_bootstrap.cc:654] T 10ecacdfb9a44d068b0df24e991612ee P 52c196c15e3348b985c689652bc49918: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:00.069535 3670 tablet_bootstrap.cc:492] T 10ecacdfb9a44d068b0df24e991612ee P 52c196c15e3348b985c689652bc49918: No bootstrap required, opened a new log
I20240503 01:59:00.069633 3670 ts_tablet_manager.cc:1367] T 10ecacdfb9a44d068b0df24e991612ee P 52c196c15e3348b985c689652bc49918: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:00.069870 3669 tablet_bootstrap.cc:492] T 8682bff56e604cc3b7e9b07c8ba3fca5 P b63928177fa54227a5ff9929265ab9dd: No bootstrap required, opened a new log
I20240503 01:59:00.070006 3669 ts_tablet_manager.cc:1367] T 8682bff56e604cc3b7e9b07c8ba3fca5 P b63928177fa54227a5ff9929265ab9dd: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.000s
I20240503 01:59:00.070048 3670 raft_consensus.cc:348] T 10ecacdfb9a44d068b0df24e991612ee P 52c196c15e3348b985c689652bc49918 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } }
I20240503 01:59:00.070159 3670 raft_consensus.cc:374] T 10ecacdfb9a44d068b0df24e991612ee P 52c196c15e3348b985c689652bc49918 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:00.070199 3670 raft_consensus.cc:729] T 10ecacdfb9a44d068b0df24e991612ee P 52c196c15e3348b985c689652bc49918 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52c196c15e3348b985c689652bc49918, State: Initialized, Role: FOLLOWER
I20240503 01:59:00.070334 3670 consensus_queue.cc:260] T 10ecacdfb9a44d068b0df24e991612ee P 52c196c15e3348b985c689652bc49918 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } }
I20240503 01:59:00.070596 3670 ts_tablet_manager.cc:1398] T 10ecacdfb9a44d068b0df24e991612ee P 52c196c15e3348b985c689652bc49918: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:00.070972 3669 raft_consensus.cc:348] T 8682bff56e604cc3b7e9b07c8ba3fca5 P b63928177fa54227a5ff9929265ab9dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } }
I20240503 01:59:00.071168 3669 raft_consensus.cc:374] T 8682bff56e604cc3b7e9b07c8ba3fca5 P b63928177fa54227a5ff9929265ab9dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:00.071233 3669 raft_consensus.cc:729] T 8682bff56e604cc3b7e9b07c8ba3fca5 P b63928177fa54227a5ff9929265ab9dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b63928177fa54227a5ff9929265ab9dd, State: Initialized, Role: FOLLOWER
I20240503 01:59:00.071408 3669 consensus_queue.cc:260] T 8682bff56e604cc3b7e9b07c8ba3fca5 P b63928177fa54227a5ff9929265ab9dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } }
I20240503 01:59:00.071619 3673 tablet_bootstrap.cc:492] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 6568f69a78c44bb985162d2b7d0a9376: No bootstrap required, opened a new log
I20240503 01:59:00.071722 3673 ts_tablet_manager.cc:1367] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 6568f69a78c44bb985162d2b7d0a9376: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 01:59:00.071782 3668 tablet_bootstrap.cc:492] T 10ecacdfb9a44d068b0df24e991612ee P b40a9a7b30d34a6db007ed8febd8a759: No bootstrap required, opened a new log
I20240503 01:59:00.071846 3669 ts_tablet_manager.cc:1398] T 8682bff56e604cc3b7e9b07c8ba3fca5 P b63928177fa54227a5ff9929265ab9dd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:00.071923 3668 ts_tablet_manager.cc:1367] T 10ecacdfb9a44d068b0df24e991612ee P b40a9a7b30d34a6db007ed8febd8a759: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20240503 01:59:00.072229 3673 raft_consensus.cc:348] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 6568f69a78c44bb985162d2b7d0a9376 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } }
I20240503 01:59:00.072367 3673 raft_consensus.cc:374] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 6568f69a78c44bb985162d2b7d0a9376 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:00.072407 3673 raft_consensus.cc:729] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 6568f69a78c44bb985162d2b7d0a9376 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6568f69a78c44bb985162d2b7d0a9376, State: Initialized, Role: FOLLOWER
I20240503 01:59:00.072559 3673 consensus_queue.cc:260] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 6568f69a78c44bb985162d2b7d0a9376 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } }
I20240503 01:59:00.072902 3673 ts_tablet_manager.cc:1398] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 6568f69a78c44bb985162d2b7d0a9376: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:00.073742 3669 tablet_bootstrap.cc:492] T 5fd1010447524b6695abb98192dc54f1 P b63928177fa54227a5ff9929265ab9dd: Bootstrap starting.
I20240503 01:59:00.073697 3668 raft_consensus.cc:348] T 10ecacdfb9a44d068b0df24e991612ee P b40a9a7b30d34a6db007ed8febd8a759 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } }
I20240503 01:59:00.073881 3668 raft_consensus.cc:374] T 10ecacdfb9a44d068b0df24e991612ee P b40a9a7b30d34a6db007ed8febd8a759 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:00.073957 3668 raft_consensus.cc:729] T 10ecacdfb9a44d068b0df24e991612ee P b40a9a7b30d34a6db007ed8febd8a759 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b40a9a7b30d34a6db007ed8febd8a759, State: Initialized, Role: FOLLOWER
I20240503 01:59:00.075186 3669 tablet_bootstrap.cc:654] T 5fd1010447524b6695abb98192dc54f1 P b63928177fa54227a5ff9929265ab9dd: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:00.075203 3668 consensus_queue.cc:260] T 10ecacdfb9a44d068b0df24e991612ee P b40a9a7b30d34a6db007ed8febd8a759 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } }
I20240503 01:59:00.076772 3667 tablet_bootstrap.cc:492] T 5fd1010447524b6695abb98192dc54f1 P 327aa26a8e434a609ebcb382aa665fce: No bootstrap required, opened a new log
I20240503 01:59:00.076915 3667 ts_tablet_manager.cc:1367] T 5fd1010447524b6695abb98192dc54f1 P 327aa26a8e434a609ebcb382aa665fce: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20240503 01:59:00.077469 3669 tablet_bootstrap.cc:492] T 5fd1010447524b6695abb98192dc54f1 P b63928177fa54227a5ff9929265ab9dd: No bootstrap required, opened a new log
I20240503 01:59:00.077581 3669 ts_tablet_manager.cc:1367] T 5fd1010447524b6695abb98192dc54f1 P b63928177fa54227a5ff9929265ab9dd: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 01:59:00.077479 3667 raft_consensus.cc:348] T 5fd1010447524b6695abb98192dc54f1 P 327aa26a8e434a609ebcb382aa665fce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } }
I20240503 01:59:00.077634 3667 raft_consensus.cc:374] T 5fd1010447524b6695abb98192dc54f1 P 327aa26a8e434a609ebcb382aa665fce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:00.077675 3667 raft_consensus.cc:729] T 5fd1010447524b6695abb98192dc54f1 P 327aa26a8e434a609ebcb382aa665fce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 327aa26a8e434a609ebcb382aa665fce, State: Initialized, Role: FOLLOWER
I20240503 01:59:00.078053 3669 raft_consensus.cc:348] T 5fd1010447524b6695abb98192dc54f1 P b63928177fa54227a5ff9929265ab9dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } }
I20240503 01:59:00.078065 3667 consensus_queue.cc:260] T 5fd1010447524b6695abb98192dc54f1 P 327aa26a8e434a609ebcb382aa665fce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } }
I20240503 01:59:00.078161 3669 raft_consensus.cc:374] T 5fd1010447524b6695abb98192dc54f1 P b63928177fa54227a5ff9929265ab9dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:00.078190 3669 raft_consensus.cc:729] T 5fd1010447524b6695abb98192dc54f1 P b63928177fa54227a5ff9929265ab9dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b63928177fa54227a5ff9929265ab9dd, State: Initialized, Role: FOLLOWER
I20240503 01:59:00.078336 3669 consensus_queue.cc:260] T 5fd1010447524b6695abb98192dc54f1 P b63928177fa54227a5ff9929265ab9dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } }
I20240503 01:59:00.078445 3667 ts_tablet_manager.cc:1398] T 5fd1010447524b6695abb98192dc54f1 P 327aa26a8e434a609ebcb382aa665fce: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:00.078612 3669 ts_tablet_manager.cc:1398] T 5fd1010447524b6695abb98192dc54f1 P b63928177fa54227a5ff9929265ab9dd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:00.078781 3669 tablet_bootstrap.cc:492] T 10ecacdfb9a44d068b0df24e991612ee P b63928177fa54227a5ff9929265ab9dd: Bootstrap starting.
I20240503 01:59:00.079309 3667 tablet_bootstrap.cc:492] T 10ecacdfb9a44d068b0df24e991612ee P 327aa26a8e434a609ebcb382aa665fce: Bootstrap starting.
I20240503 01:59:00.079648 3669 tablet_bootstrap.cc:654] T 10ecacdfb9a44d068b0df24e991612ee P b63928177fa54227a5ff9929265ab9dd: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:00.080439 3667 tablet_bootstrap.cc:654] T 10ecacdfb9a44d068b0df24e991612ee P 327aa26a8e434a609ebcb382aa665fce: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:00.082669 3667 tablet_bootstrap.cc:492] T 10ecacdfb9a44d068b0df24e991612ee P 327aa26a8e434a609ebcb382aa665fce: No bootstrap required, opened a new log
I20240503 01:59:00.082773 3667 ts_tablet_manager.cc:1367] T 10ecacdfb9a44d068b0df24e991612ee P 327aa26a8e434a609ebcb382aa665fce: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240503 01:59:00.083374 3667 raft_consensus.cc:348] T 10ecacdfb9a44d068b0df24e991612ee P 327aa26a8e434a609ebcb382aa665fce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } }
I20240503 01:59:00.083527 3667 raft_consensus.cc:374] T 10ecacdfb9a44d068b0df24e991612ee P 327aa26a8e434a609ebcb382aa665fce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:00.083582 3667 raft_consensus.cc:729] T 10ecacdfb9a44d068b0df24e991612ee P 327aa26a8e434a609ebcb382aa665fce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 327aa26a8e434a609ebcb382aa665fce, State: Initialized, Role: FOLLOWER
I20240503 01:59:00.083741 3667 consensus_queue.cc:260] T 10ecacdfb9a44d068b0df24e991612ee P 327aa26a8e434a609ebcb382aa665fce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } }
I20240503 01:59:00.084126 3667 ts_tablet_manager.cc:1398] T 10ecacdfb9a44d068b0df24e991612ee P 327aa26a8e434a609ebcb382aa665fce: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:00.084326 3667 tablet_bootstrap.cc:492] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 327aa26a8e434a609ebcb382aa665fce: Bootstrap starting.
I20240503 01:59:00.084875 3668 ts_tablet_manager.cc:1398] T 10ecacdfb9a44d068b0df24e991612ee P b40a9a7b30d34a6db007ed8febd8a759: Time spent starting tablet: real 0.013s user 0.000s sys 0.001s
I20240503 01:59:00.085470 3668 tablet_bootstrap.cc:492] T 8682bff56e604cc3b7e9b07c8ba3fca5 P b40a9a7b30d34a6db007ed8febd8a759: Bootstrap starting.
I20240503 01:59:00.085646 3667 tablet_bootstrap.cc:654] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 327aa26a8e434a609ebcb382aa665fce: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:00.087190 3668 tablet_bootstrap.cc:654] T 8682bff56e604cc3b7e9b07c8ba3fca5 P b40a9a7b30d34a6db007ed8febd8a759: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:00.088196 3667 tablet_bootstrap.cc:492] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 327aa26a8e434a609ebcb382aa665fce: No bootstrap required, opened a new log
I20240503 01:59:00.088297 3667 ts_tablet_manager.cc:1367] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 327aa26a8e434a609ebcb382aa665fce: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 01:59:00.088809 3667 raft_consensus.cc:348] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 327aa26a8e434a609ebcb382aa665fce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } }
I20240503 01:59:00.088936 3667 raft_consensus.cc:374] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 327aa26a8e434a609ebcb382aa665fce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:00.088979 3667 raft_consensus.cc:729] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 327aa26a8e434a609ebcb382aa665fce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 327aa26a8e434a609ebcb382aa665fce, State: Initialized, Role: FOLLOWER
I20240503 01:59:00.089344 3667 consensus_queue.cc:260] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 327aa26a8e434a609ebcb382aa665fce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } }
I20240503 01:59:00.089668 3667 ts_tablet_manager.cc:1398] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 327aa26a8e434a609ebcb382aa665fce: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:00.090672 3668 tablet_bootstrap.cc:492] T 8682bff56e604cc3b7e9b07c8ba3fca5 P b40a9a7b30d34a6db007ed8febd8a759: No bootstrap required, opened a new log
I20240503 01:59:00.090785 3668 ts_tablet_manager.cc:1367] T 8682bff56e604cc3b7e9b07c8ba3fca5 P b40a9a7b30d34a6db007ed8febd8a759: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240503 01:59:00.090607 3669 tablet_bootstrap.cc:492] T 10ecacdfb9a44d068b0df24e991612ee P b63928177fa54227a5ff9929265ab9dd: No bootstrap required, opened a new log
I20240503 01:59:00.091347 3668 raft_consensus.cc:348] T 8682bff56e604cc3b7e9b07c8ba3fca5 P b40a9a7b30d34a6db007ed8febd8a759 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } }
I20240503 01:59:00.091463 3668 raft_consensus.cc:374] T 8682bff56e604cc3b7e9b07c8ba3fca5 P b40a9a7b30d34a6db007ed8febd8a759 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:00.091495 3668 raft_consensus.cc:729] T 8682bff56e604cc3b7e9b07c8ba3fca5 P b40a9a7b30d34a6db007ed8febd8a759 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b40a9a7b30d34a6db007ed8febd8a759, State: Initialized, Role: FOLLOWER
I20240503 01:59:00.091528 3669 ts_tablet_manager.cc:1367] T 10ecacdfb9a44d068b0df24e991612ee P b63928177fa54227a5ff9929265ab9dd: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240503 01:59:00.091639 3668 consensus_queue.cc:260] T 8682bff56e604cc3b7e9b07c8ba3fca5 P b40a9a7b30d34a6db007ed8febd8a759 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } }
I20240503 01:59:00.091969 3668 ts_tablet_manager.cc:1398] T 8682bff56e604cc3b7e9b07c8ba3fca5 P b40a9a7b30d34a6db007ed8febd8a759: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:00.092159 3669 raft_consensus.cc:348] T 10ecacdfb9a44d068b0df24e991612ee P b63928177fa54227a5ff9929265ab9dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } }
I20240503 01:59:00.092283 3669 raft_consensus.cc:374] T 10ecacdfb9a44d068b0df24e991612ee P b63928177fa54227a5ff9929265ab9dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:00.092324 3669 raft_consensus.cc:729] T 10ecacdfb9a44d068b0df24e991612ee P b63928177fa54227a5ff9929265ab9dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b63928177fa54227a5ff9929265ab9dd, State: Initialized, Role: FOLLOWER
I20240503 01:59:00.092459 3669 consensus_queue.cc:260] T 10ecacdfb9a44d068b0df24e991612ee P b63928177fa54227a5ff9929265ab9dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } }
I20240503 01:59:00.093354 3669 ts_tablet_manager.cc:1398] T 10ecacdfb9a44d068b0df24e991612ee P b63928177fa54227a5ff9929265ab9dd: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:00.109851 3551 raft_consensus.cc:482] T 5fd1010447524b6695abb98192dc54f1 P 327aa26a8e434a609ebcb382aa665fce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:00.109968 3551 raft_consensus.cc:504] T 5fd1010447524b6695abb98192dc54f1 P 327aa26a8e434a609ebcb382aa665fce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } }
I20240503 01:59:00.110599 3551 leader_election.cc:288] T 5fd1010447524b6695abb98192dc54f1 P 327aa26a8e434a609ebcb382aa665fce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b40a9a7b30d34a6db007ed8febd8a759 (127.0.99.131:34079), 52c196c15e3348b985c689652bc49918 (127.0.99.132:42887), 6568f69a78c44bb985162d2b7d0a9376 (127.0.99.134:43603), b63928177fa54227a5ff9929265ab9dd (127.0.99.129:36425)
I20240503 01:59:00.111024 3054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fd1010447524b6695abb98192dc54f1" candidate_uuid: "327aa26a8e434a609ebcb382aa665fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6568f69a78c44bb985162d2b7d0a9376" is_pre_election: true
I20240503 01:59:00.111169 3054 raft_consensus.cc:2454] T 5fd1010447524b6695abb98192dc54f1 P 6568f69a78c44bb985162d2b7d0a9376 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 327aa26a8e434a609ebcb382aa665fce in term 0.
I20240503 01:59:00.111413 2791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fd1010447524b6695abb98192dc54f1" candidate_uuid: "327aa26a8e434a609ebcb382aa665fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52c196c15e3348b985c689652bc49918" is_pre_election: true
I20240503 01:59:00.111373 2661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fd1010447524b6695abb98192dc54f1" candidate_uuid: "327aa26a8e434a609ebcb382aa665fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b40a9a7b30d34a6db007ed8febd8a759" is_pre_election: true
I20240503 01:59:00.111555 2791 raft_consensus.cc:2454] T 5fd1010447524b6695abb98192dc54f1 P 52c196c15e3348b985c689652bc49918 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 327aa26a8e434a609ebcb382aa665fce in term 0.
I20240503 01:59:00.111651 2661 raft_consensus.cc:2454] T 5fd1010447524b6695abb98192dc54f1 P b40a9a7b30d34a6db007ed8febd8a759 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 327aa26a8e434a609ebcb382aa665fce in term 0.
I20240503 01:59:00.111982 2857 leader_election.cc:302] T 5fd1010447524b6695abb98192dc54f1 P 327aa26a8e434a609ebcb382aa665fce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 327aa26a8e434a609ebcb382aa665fce, 6568f69a78c44bb985162d2b7d0a9376, b40a9a7b30d34a6db007ed8febd8a759; no voters:
I20240503 01:59:00.112216 3551 raft_consensus.cc:2781] T 5fd1010447524b6695abb98192dc54f1 P 327aa26a8e434a609ebcb382aa665fce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:00.112265 3551 raft_consensus.cc:482] T 5fd1010447524b6695abb98192dc54f1 P 327aa26a8e434a609ebcb382aa665fce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:00.112310 3551 raft_consensus.cc:3037] T 5fd1010447524b6695abb98192dc54f1 P 327aa26a8e434a609ebcb382aa665fce [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:00.115655 2398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fd1010447524b6695abb98192dc54f1" candidate_uuid: "327aa26a8e434a609ebcb382aa665fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b63928177fa54227a5ff9929265ab9dd" is_pre_election: true
I20240503 01:59:00.115824 2398 raft_consensus.cc:2454] T 5fd1010447524b6695abb98192dc54f1 P b63928177fa54227a5ff9929265ab9dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 327aa26a8e434a609ebcb382aa665fce in term 0.
I20240503 01:59:00.147332 3541 raft_consensus.cc:482] T 10ecacdfb9a44d068b0df24e991612ee P 327aa26a8e434a609ebcb382aa665fce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:00.147567 3541 raft_consensus.cc:504] T 10ecacdfb9a44d068b0df24e991612ee P 327aa26a8e434a609ebcb382aa665fce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } }
I20240503 01:59:00.147701 3551 raft_consensus.cc:504] T 5fd1010447524b6695abb98192dc54f1 P 327aa26a8e434a609ebcb382aa665fce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } }
I20240503 01:59:00.148321 3551 leader_election.cc:288] T 5fd1010447524b6695abb98192dc54f1 P 327aa26a8e434a609ebcb382aa665fce [CANDIDATE]: Term 1 election: Requested vote from peers b40a9a7b30d34a6db007ed8febd8a759 (127.0.99.131:34079), 52c196c15e3348b985c689652bc49918 (127.0.99.132:42887), 6568f69a78c44bb985162d2b7d0a9376 (127.0.99.134:43603), b63928177fa54227a5ff9929265ab9dd (127.0.99.129:36425)
I20240503 01:59:00.148386 3541 leader_election.cc:288] T 10ecacdfb9a44d068b0df24e991612ee P 327aa26a8e434a609ebcb382aa665fce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b40a9a7b30d34a6db007ed8febd8a759 (127.0.99.131:34079), a1a844ecd71145e4a362f4c42500fbcb (127.0.99.130:46033), 52c196c15e3348b985c689652bc49918 (127.0.99.132:42887), b63928177fa54227a5ff9929265ab9dd (127.0.99.129:36425)
I20240503 01:59:00.148895 2661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10ecacdfb9a44d068b0df24e991612ee" candidate_uuid: "327aa26a8e434a609ebcb382aa665fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b40a9a7b30d34a6db007ed8febd8a759" is_pre_election: true
I20240503 01:59:00.148960 3054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fd1010447524b6695abb98192dc54f1" candidate_uuid: "327aa26a8e434a609ebcb382aa665fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6568f69a78c44bb985162d2b7d0a9376"
I20240503 01:59:00.149092 3054 raft_consensus.cc:3037] T 5fd1010447524b6695abb98192dc54f1 P 6568f69a78c44bb985162d2b7d0a9376 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:00.149077 2661 raft_consensus.cc:2454] T 10ecacdfb9a44d068b0df24e991612ee P b40a9a7b30d34a6db007ed8febd8a759 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 327aa26a8e434a609ebcb382aa665fce in term 0.
I20240503 01:59:00.149269 2400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10ecacdfb9a44d068b0df24e991612ee" candidate_uuid: "327aa26a8e434a609ebcb382aa665fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b63928177fa54227a5ff9929265ab9dd" is_pre_election: true
I20240503 01:59:00.149406 2400 raft_consensus.cc:2454] T 10ecacdfb9a44d068b0df24e991612ee P b63928177fa54227a5ff9929265ab9dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 327aa26a8e434a609ebcb382aa665fce in term 0.
I20240503 01:59:00.149597 2398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fd1010447524b6695abb98192dc54f1" candidate_uuid: "327aa26a8e434a609ebcb382aa665fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b63928177fa54227a5ff9929265ab9dd"
I20240503 01:59:00.149575 2531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10ecacdfb9a44d068b0df24e991612ee" candidate_uuid: "327aa26a8e434a609ebcb382aa665fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1a844ecd71145e4a362f4c42500fbcb" is_pre_election: true
I20240503 01:59:00.149679 2398 raft_consensus.cc:3037] T 5fd1010447524b6695abb98192dc54f1 P b63928177fa54227a5ff9929265ab9dd [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:00.149746 2531 raft_consensus.cc:2454] T 10ecacdfb9a44d068b0df24e991612ee P a1a844ecd71145e4a362f4c42500fbcb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 327aa26a8e434a609ebcb382aa665fce in term 0.
I20240503 01:59:00.150247 3054 raft_consensus.cc:2454] T 5fd1010447524b6695abb98192dc54f1 P 6568f69a78c44bb985162d2b7d0a9376 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 327aa26a8e434a609ebcb382aa665fce in term 1.
I20240503 01:59:00.150761 2398 raft_consensus.cc:2454] T 5fd1010447524b6695abb98192dc54f1 P b63928177fa54227a5ff9929265ab9dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 327aa26a8e434a609ebcb382aa665fce in term 1.
I20240503 01:59:00.150874 2860 leader_election.cc:302] T 10ecacdfb9a44d068b0df24e991612ee P 327aa26a8e434a609ebcb382aa665fce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 327aa26a8e434a609ebcb382aa665fce, a1a844ecd71145e4a362f4c42500fbcb, b63928177fa54227a5ff9929265ab9dd; no voters:
I20240503 01:59:00.151049 2791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fd1010447524b6695abb98192dc54f1" candidate_uuid: "327aa26a8e434a609ebcb382aa665fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52c196c15e3348b985c689652bc49918"
I20240503 01:59:00.151198 2791 raft_consensus.cc:3037] T 5fd1010447524b6695abb98192dc54f1 P 52c196c15e3348b985c689652bc49918 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:00.151234 2860 leader_election.cc:302] T 5fd1010447524b6695abb98192dc54f1 P 327aa26a8e434a609ebcb382aa665fce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 327aa26a8e434a609ebcb382aa665fce, 6568f69a78c44bb985162d2b7d0a9376, b63928177fa54227a5ff9929265ab9dd; no voters:
I20240503 01:59:00.151859 3551 raft_consensus.cc:2781] T 10ecacdfb9a44d068b0df24e991612ee P 327aa26a8e434a609ebcb382aa665fce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:00.151841 2661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fd1010447524b6695abb98192dc54f1" candidate_uuid: "327aa26a8e434a609ebcb382aa665fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b40a9a7b30d34a6db007ed8febd8a759"
I20240503 01:59:00.151917 3551 raft_consensus.cc:482] T 10ecacdfb9a44d068b0df24e991612ee P 327aa26a8e434a609ebcb382aa665fce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:00.151953 3551 raft_consensus.cc:3037] T 10ecacdfb9a44d068b0df24e991612ee P 327aa26a8e434a609ebcb382aa665fce [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:00.151988 2661 raft_consensus.cc:3037] T 5fd1010447524b6695abb98192dc54f1 P b40a9a7b30d34a6db007ed8febd8a759 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:00.152305 2791 raft_consensus.cc:2454] T 5fd1010447524b6695abb98192dc54f1 P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 327aa26a8e434a609ebcb382aa665fce in term 1.
I20240503 01:59:00.153259 3551 raft_consensus.cc:504] T 10ecacdfb9a44d068b0df24e991612ee P 327aa26a8e434a609ebcb382aa665fce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } }
I20240503 01:59:00.153676 3541 raft_consensus.cc:2781] T 5fd1010447524b6695abb98192dc54f1 P 327aa26a8e434a609ebcb382aa665fce [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:00.153854 2661 raft_consensus.cc:2454] T 5fd1010447524b6695abb98192dc54f1 P b40a9a7b30d34a6db007ed8febd8a759 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 327aa26a8e434a609ebcb382aa665fce in term 1.
I20240503 01:59:00.153921 3541 raft_consensus.cc:686] T 5fd1010447524b6695abb98192dc54f1 P 327aa26a8e434a609ebcb382aa665fce [term 1 LEADER]: Becoming Leader. State: Replica: 327aa26a8e434a609ebcb382aa665fce, State: Running, Role: LEADER
I20240503 01:59:00.153980 2662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10ecacdfb9a44d068b0df24e991612ee" candidate_uuid: "327aa26a8e434a609ebcb382aa665fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b40a9a7b30d34a6db007ed8febd8a759"
I20240503 01:59:00.154101 2662 raft_consensus.cc:3037] T 10ecacdfb9a44d068b0df24e991612ee P b40a9a7b30d34a6db007ed8febd8a759 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:00.154119 3541 consensus_queue.cc:237] T 5fd1010447524b6695abb98192dc54f1 P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } }
I20240503 01:59:00.154219 2792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10ecacdfb9a44d068b0df24e991612ee" candidate_uuid: "327aa26a8e434a609ebcb382aa665fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52c196c15e3348b985c689652bc49918" is_pre_election: true
I20240503 01:59:00.154382 2792 raft_consensus.cc:2454] T 10ecacdfb9a44d068b0df24e991612ee P 52c196c15e3348b985c689652bc49918 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 327aa26a8e434a609ebcb382aa665fce in term 0.
I20240503 01:59:00.155138 2662 raft_consensus.cc:2454] T 10ecacdfb9a44d068b0df24e991612ee P b40a9a7b30d34a6db007ed8febd8a759 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 327aa26a8e434a609ebcb382aa665fce in term 1.
I20240503 01:59:00.155639 3551 leader_election.cc:288] T 10ecacdfb9a44d068b0df24e991612ee P 327aa26a8e434a609ebcb382aa665fce [CANDIDATE]: Term 1 election: Requested vote from peers b40a9a7b30d34a6db007ed8febd8a759 (127.0.99.131:34079), a1a844ecd71145e4a362f4c42500fbcb (127.0.99.130:46033), 52c196c15e3348b985c689652bc49918 (127.0.99.132:42887), b63928177fa54227a5ff9929265ab9dd (127.0.99.129:36425)
I20240503 01:59:00.156018 2531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10ecacdfb9a44d068b0df24e991612ee" candidate_uuid: "327aa26a8e434a609ebcb382aa665fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1a844ecd71145e4a362f4c42500fbcb"
I20240503 01:59:00.156092 2398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10ecacdfb9a44d068b0df24e991612ee" candidate_uuid: "327aa26a8e434a609ebcb382aa665fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b63928177fa54227a5ff9929265ab9dd"
I20240503 01:59:00.156168 2531 raft_consensus.cc:3037] T 10ecacdfb9a44d068b0df24e991612ee P a1a844ecd71145e4a362f4c42500fbcb [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:00.156188 2398 raft_consensus.cc:3037] T 10ecacdfb9a44d068b0df24e991612ee P b63928177fa54227a5ff9929265ab9dd [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:00.156272 2792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10ecacdfb9a44d068b0df24e991612ee" candidate_uuid: "327aa26a8e434a609ebcb382aa665fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52c196c15e3348b985c689652bc49918"
I20240503 01:59:00.156376 2792 raft_consensus.cc:3037] T 10ecacdfb9a44d068b0df24e991612ee P 52c196c15e3348b985c689652bc49918 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:00.156978 2251 catalog_manager.cc:5484] T 5fd1010447524b6695abb98192dc54f1 P 327aa26a8e434a609ebcb382aa665fce reported cstate change: term changed from 0 to 1, leader changed from <none> to 327aa26a8e434a609ebcb382aa665fce (127.0.99.133). New cstate: current_term: 1 leader_uuid: "327aa26a8e434a609ebcb382aa665fce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:00.221300 2792 raft_consensus.cc:2454] T 10ecacdfb9a44d068b0df24e991612ee P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 327aa26a8e434a609ebcb382aa665fce in term 1.
I20240503 01:59:00.221300 2398 raft_consensus.cc:2454] T 10ecacdfb9a44d068b0df24e991612ee P b63928177fa54227a5ff9929265ab9dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 327aa26a8e434a609ebcb382aa665fce in term 1.
I20240503 01:59:00.221329 2531 raft_consensus.cc:2454] T 10ecacdfb9a44d068b0df24e991612ee P a1a844ecd71145e4a362f4c42500fbcb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 327aa26a8e434a609ebcb382aa665fce in term 1.
I20240503 01:59:00.221742 2860 leader_election.cc:302] T 10ecacdfb9a44d068b0df24e991612ee P 327aa26a8e434a609ebcb382aa665fce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 327aa26a8e434a609ebcb382aa665fce, b40a9a7b30d34a6db007ed8febd8a759, b63928177fa54227a5ff9929265ab9dd; no voters:
I20240503 01:59:00.222190 3551 raft_consensus.cc:2781] T 10ecacdfb9a44d068b0df24e991612ee P 327aa26a8e434a609ebcb382aa665fce [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:00.222311 3551 raft_consensus.cc:686] T 10ecacdfb9a44d068b0df24e991612ee P 327aa26a8e434a609ebcb382aa665fce [term 1 LEADER]: Becoming Leader. State: Replica: 327aa26a8e434a609ebcb382aa665fce, State: Running, Role: LEADER
I20240503 01:59:00.222477 3551 consensus_queue.cc:237] T 10ecacdfb9a44d068b0df24e991612ee P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } }
I20240503 01:59:00.224398 2251 catalog_manager.cc:5484] T 10ecacdfb9a44d068b0df24e991612ee P 327aa26a8e434a609ebcb382aa665fce reported cstate change: term changed from 0 to 1, leader changed from <none> to 327aa26a8e434a609ebcb382aa665fce (127.0.99.133). New cstate: current_term: 1 leader_uuid: "327aa26a8e434a609ebcb382aa665fce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:00.318181 3661 raft_consensus.cc:482] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 6568f69a78c44bb985162d2b7d0a9376 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:00.318348 3661 raft_consensus.cc:504] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 6568f69a78c44bb985162d2b7d0a9376 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } }
I20240503 01:59:00.319048 3661 leader_election.cc:288] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 6568f69a78c44bb985162d2b7d0a9376 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b63928177fa54227a5ff9929265ab9dd (127.0.99.129:36425), 52c196c15e3348b985c689652bc49918 (127.0.99.132:42887), 327aa26a8e434a609ebcb382aa665fce (127.0.99.133:38853), b40a9a7b30d34a6db007ed8febd8a759 (127.0.99.131:34079)
I20240503 01:59:00.319553 2792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8682bff56e604cc3b7e9b07c8ba3fca5" candidate_uuid: "6568f69a78c44bb985162d2b7d0a9376" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52c196c15e3348b985c689652bc49918" is_pre_election: true
I20240503 01:59:00.319697 2792 raft_consensus.cc:2454] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 52c196c15e3348b985c689652bc49918 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6568f69a78c44bb985162d2b7d0a9376 in term 0.
I20240503 01:59:00.319882 2398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8682bff56e604cc3b7e9b07c8ba3fca5" candidate_uuid: "6568f69a78c44bb985162d2b7d0a9376" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b63928177fa54227a5ff9929265ab9dd" is_pre_election: true
I20240503 01:59:00.320080 2923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8682bff56e604cc3b7e9b07c8ba3fca5" candidate_uuid: "6568f69a78c44bb985162d2b7d0a9376" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "327aa26a8e434a609ebcb382aa665fce" is_pre_election: true
I20240503 01:59:00.320118 2398 raft_consensus.cc:2454] T 8682bff56e604cc3b7e9b07c8ba3fca5 P b63928177fa54227a5ff9929265ab9dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6568f69a78c44bb985162d2b7d0a9376 in term 0.
I20240503 01:59:00.320228 2923 raft_consensus.cc:2454] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 327aa26a8e434a609ebcb382aa665fce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6568f69a78c44bb985162d2b7d0a9376 in term 0.
I20240503 01:59:00.320533 2989 leader_election.cc:302] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 6568f69a78c44bb985162d2b7d0a9376 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 327aa26a8e434a609ebcb382aa665fce, 52c196c15e3348b985c689652bc49918, 6568f69a78c44bb985162d2b7d0a9376; no voters:
I20240503 01:59:00.320622 2662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8682bff56e604cc3b7e9b07c8ba3fca5" candidate_uuid: "6568f69a78c44bb985162d2b7d0a9376" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b40a9a7b30d34a6db007ed8febd8a759" is_pre_election: true
I20240503 01:59:00.320696 3661 raft_consensus.cc:2781] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 6568f69a78c44bb985162d2b7d0a9376 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:00.320776 3661 raft_consensus.cc:482] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 6568f69a78c44bb985162d2b7d0a9376 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:00.320827 3661 raft_consensus.cc:3037] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 6568f69a78c44bb985162d2b7d0a9376 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:00.320820 2662 raft_consensus.cc:2454] T 8682bff56e604cc3b7e9b07c8ba3fca5 P b40a9a7b30d34a6db007ed8febd8a759 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6568f69a78c44bb985162d2b7d0a9376 in term 0.
I20240503 01:59:00.321795 3661 raft_consensus.cc:504] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 6568f69a78c44bb985162d2b7d0a9376 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } }
I20240503 01:59:00.322331 3661 leader_election.cc:288] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 6568f69a78c44bb985162d2b7d0a9376 [CANDIDATE]: Term 1 election: Requested vote from peers b63928177fa54227a5ff9929265ab9dd (127.0.99.129:36425), 52c196c15e3348b985c689652bc49918 (127.0.99.132:42887), 327aa26a8e434a609ebcb382aa665fce (127.0.99.133:38853), b40a9a7b30d34a6db007ed8febd8a759 (127.0.99.131:34079)
I20240503 01:59:00.322752 2792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8682bff56e604cc3b7e9b07c8ba3fca5" candidate_uuid: "6568f69a78c44bb985162d2b7d0a9376" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52c196c15e3348b985c689652bc49918"
I20240503 01:59:00.322762 2662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8682bff56e604cc3b7e9b07c8ba3fca5" candidate_uuid: "6568f69a78c44bb985162d2b7d0a9376" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b40a9a7b30d34a6db007ed8febd8a759"
I20240503 01:59:00.322853 2792 raft_consensus.cc:3037] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 52c196c15e3348b985c689652bc49918 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:00.322880 2662 raft_consensus.cc:3037] T 8682bff56e604cc3b7e9b07c8ba3fca5 P b40a9a7b30d34a6db007ed8febd8a759 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:00.322855 2923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8682bff56e604cc3b7e9b07c8ba3fca5" candidate_uuid: "6568f69a78c44bb985162d2b7d0a9376" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "327aa26a8e434a609ebcb382aa665fce"
I20240503 01:59:00.323014 2923 raft_consensus.cc:3037] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 327aa26a8e434a609ebcb382aa665fce [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:00.323164 2398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8682bff56e604cc3b7e9b07c8ba3fca5" candidate_uuid: "6568f69a78c44bb985162d2b7d0a9376" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b63928177fa54227a5ff9929265ab9dd"
I20240503 01:59:00.323289 2398 raft_consensus.cc:3037] T 8682bff56e604cc3b7e9b07c8ba3fca5 P b63928177fa54227a5ff9929265ab9dd [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:00.415946 2792 raft_consensus.cc:2454] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6568f69a78c44bb985162d2b7d0a9376 in term 1.
I20240503 01:59:00.415946 2662 raft_consensus.cc:2454] T 8682bff56e604cc3b7e9b07c8ba3fca5 P b40a9a7b30d34a6db007ed8febd8a759 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6568f69a78c44bb985162d2b7d0a9376 in term 1.
I20240503 01:59:00.415975 2923 raft_consensus.cc:2454] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 327aa26a8e434a609ebcb382aa665fce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6568f69a78c44bb985162d2b7d0a9376 in term 1.
I20240503 01:59:00.416415 2991 leader_election.cc:302] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 6568f69a78c44bb985162d2b7d0a9376 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 52c196c15e3348b985c689652bc49918, 6568f69a78c44bb985162d2b7d0a9376, b40a9a7b30d34a6db007ed8febd8a759; no voters:
I20240503 01:59:00.416682 2398 raft_consensus.cc:2454] T 8682bff56e604cc3b7e9b07c8ba3fca5 P b63928177fa54227a5ff9929265ab9dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6568f69a78c44bb985162d2b7d0a9376 in term 1.
I20240503 01:59:00.416739 3661 raft_consensus.cc:2781] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 6568f69a78c44bb985162d2b7d0a9376 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:00.416997 3661 raft_consensus.cc:686] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 6568f69a78c44bb985162d2b7d0a9376 [term 1 LEADER]: Becoming Leader. State: Replica: 6568f69a78c44bb985162d2b7d0a9376, State: Running, Role: LEADER
I20240503 01:59:00.417200 3661 consensus_queue.cc:237] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 6568f69a78c44bb985162d2b7d0a9376 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } }
I20240503 01:59:00.420117 2251 catalog_manager.cc:5484] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 6568f69a78c44bb985162d2b7d0a9376 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6568f69a78c44bb985162d2b7d0a9376 (127.0.99.134). New cstate: current_term: 1 leader_uuid: "6568f69a78c44bb985162d2b7d0a9376" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } health_report { overall_health: UNKNOWN } } }
W20240503 01:59:00.568610 3693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:00.568993 3693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:00.572463 3693 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240503 01:59:00.647043 3551 consensus_queue.cc:1035] T 5fd1010447524b6695abb98192dc54f1 P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:00.658584 2662 raft_consensus.cc:1261] T 10ecacdfb9a44d068b0df24e991612ee P b40a9a7b30d34a6db007ed8febd8a759 [term 1 FOLLOWER]: Refusing update from remote peer 327aa26a8e434a609ebcb382aa665fce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:00.659049 3551 consensus_queue.cc:1035] T 10ecacdfb9a44d068b0df24e991612ee P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:00.659557 2398 raft_consensus.cc:1261] T 10ecacdfb9a44d068b0df24e991612ee P b63928177fa54227a5ff9929265ab9dd [term 1 FOLLOWER]: Refusing update from remote peer 327aa26a8e434a609ebcb382aa665fce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:00.659678 2792 raft_consensus.cc:1261] T 10ecacdfb9a44d068b0df24e991612ee P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Refusing update from remote peer 327aa26a8e434a609ebcb382aa665fce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:00.660509 2531 raft_consensus.cc:1261] T 10ecacdfb9a44d068b0df24e991612ee P a1a844ecd71145e4a362f4c42500fbcb [term 1 FOLLOWER]: Refusing update from remote peer 327aa26a8e434a609ebcb382aa665fce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:00.660989 3689 consensus_queue.cc:1035] T 10ecacdfb9a44d068b0df24e991612ee P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:00.661274 2661 raft_consensus.cc:1261] T 8682bff56e604cc3b7e9b07c8ba3fca5 P b40a9a7b30d34a6db007ed8febd8a759 [term 1 FOLLOWER]: Refusing update from remote peer 6568f69a78c44bb985162d2b7d0a9376: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:00.661296 3689 consensus_queue.cc:1035] T 10ecacdfb9a44d068b0df24e991612ee P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:00.661659 3661 consensus_queue.cc:1035] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 6568f69a78c44bb985162d2b7d0a9376 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:00.662004 2792 raft_consensus.cc:1261] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Refusing update from remote peer 6568f69a78c44bb985162d2b7d0a9376: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:00.661317 2923 raft_consensus.cc:1261] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 327aa26a8e434a609ebcb382aa665fce [term 1 FOLLOWER]: Refusing update from remote peer 6568f69a78c44bb985162d2b7d0a9376: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:00.662325 2398 raft_consensus.cc:1261] T 8682bff56e604cc3b7e9b07c8ba3fca5 P b63928177fa54227a5ff9929265ab9dd [term 1 FOLLOWER]: Refusing update from remote peer 6568f69a78c44bb985162d2b7d0a9376: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:00.662894 3661 consensus_queue.cc:1035] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 6568f69a78c44bb985162d2b7d0a9376 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:00.663192 3661 consensus_queue.cc:1035] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 6568f69a78c44bb985162d2b7d0a9376 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:00.664095 3551 consensus_queue.cc:1035] T 10ecacdfb9a44d068b0df24e991612ee P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:00.666747 3054 raft_consensus.cc:1261] T 5fd1010447524b6695abb98192dc54f1 P 6568f69a78c44bb985162d2b7d0a9376 [term 1 FOLLOWER]: Refusing update from remote peer 327aa26a8e434a609ebcb382aa665fce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:00.667197 3689 consensus_queue.cc:1035] T 5fd1010447524b6695abb98192dc54f1 P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:00.668362 3661 consensus_queue.cc:1035] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 6568f69a78c44bb985162d2b7d0a9376 [LEADER]: Connected to new peer: Peer: permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:00.670977 2399 raft_consensus.cc:1261] T 5fd1010447524b6695abb98192dc54f1 P b63928177fa54227a5ff9929265ab9dd [term 1 FOLLOWER]: Refusing update from remote peer 327aa26a8e434a609ebcb382aa665fce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:00.671464 2793 raft_consensus.cc:1261] T 5fd1010447524b6695abb98192dc54f1 P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Refusing update from remote peer 327aa26a8e434a609ebcb382aa665fce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:00.672061 3551 consensus_queue.cc:1035] T 5fd1010447524b6695abb98192dc54f1 P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:00.672866 3551 consensus_queue.cc:1035] T 5fd1010447524b6695abb98192dc54f1 P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 155.683 ms
time per row: 0.0778414 ms
I20240503 01:59:00.778055 2250 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53804:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 5
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:59:00.783633 2379 tablet_service.cc:1467] Processing CreateTablet for tablet 0b669a47f1d14ce98ccab24479d39304 (DEFAULT_TABLE table=rebalance_test_table_4 [id=e8f9ec3f55d044f8b23964df20e79f65]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:00.783867 2509 tablet_service.cc:1467] Processing CreateTablet for tablet 0b669a47f1d14ce98ccab24479d39304 (DEFAULT_TABLE table=rebalance_test_table_4 [id=e8f9ec3f55d044f8b23964df20e79f65]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:00.783970 2379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b669a47f1d14ce98ccab24479d39304. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:00.784143 2509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b669a47f1d14ce98ccab24479d39304. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:00.784174 2510 tablet_service.cc:1467] Processing CreateTablet for tablet d4d07c9621134780bc7ece7bb061d934 (DEFAULT_TABLE table=rebalance_test_table_4 [id=e8f9ec3f55d044f8b23964df20e79f65]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:00.784415 2510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4d07c9621134780bc7ece7bb061d934. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:00.785388 2772 tablet_service.cc:1467] Processing CreateTablet for tablet 0b669a47f1d14ce98ccab24479d39304 (DEFAULT_TABLE table=rebalance_test_table_4 [id=e8f9ec3f55d044f8b23964df20e79f65]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:00.785547 3034 tablet_service.cc:1467] Processing CreateTablet for tablet 0b669a47f1d14ce98ccab24479d39304 (DEFAULT_TABLE table=rebalance_test_table_4 [id=e8f9ec3f55d044f8b23964df20e79f65]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:00.785696 2772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b669a47f1d14ce98ccab24479d39304. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:00.785799 3034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b669a47f1d14ce98ccab24479d39304. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:00.787029 3033 tablet_service.cc:1467] Processing CreateTablet for tablet f40f77238d1e4a34a3f7bea387c6406c (DEFAULT_TABLE table=rebalance_test_table_4 [id=e8f9ec3f55d044f8b23964df20e79f65]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:00.787261 3033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f40f77238d1e4a34a3f7bea387c6406c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:00.787829 3780 tablet_bootstrap.cc:492] T 0b669a47f1d14ce98ccab24479d39304 P 52c196c15e3348b985c689652bc49918: Bootstrap starting.
I20240503 01:59:00.788226 3035 tablet_service.cc:1467] Processing CreateTablet for tablet d4d07c9621134780bc7ece7bb061d934 (DEFAULT_TABLE table=rebalance_test_table_4 [id=e8f9ec3f55d044f8b23964df20e79f65]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:00.788466 3035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4d07c9621134780bc7ece7bb061d934. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:00.788905 3780 tablet_bootstrap.cc:654] T 0b669a47f1d14ce98ccab24479d39304 P 52c196c15e3348b985c689652bc49918: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:00.789099 3779 tablet_bootstrap.cc:492] T 0b669a47f1d14ce98ccab24479d39304 P a1a844ecd71145e4a362f4c42500fbcb: Bootstrap starting.
I20240503 01:59:00.789609 2902 tablet_service.cc:1467] Processing CreateTablet for tablet f40f77238d1e4a34a3f7bea387c6406c (DEFAULT_TABLE table=rebalance_test_table_4 [id=e8f9ec3f55d044f8b23964df20e79f65]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:00.789901 2902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f40f77238d1e4a34a3f7bea387c6406c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:00.790222 3779 tablet_bootstrap.cc:654] T 0b669a47f1d14ce98ccab24479d39304 P a1a844ecd71145e4a362f4c42500fbcb: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:00.791275 3779 tablet_bootstrap.cc:492] T 0b669a47f1d14ce98ccab24479d39304 P a1a844ecd71145e4a362f4c42500fbcb: No bootstrap required, opened a new log
I20240503 01:59:00.791358 3779 ts_tablet_manager.cc:1367] T 0b669a47f1d14ce98ccab24479d39304 P a1a844ecd71145e4a362f4c42500fbcb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:00.791446 2904 tablet_service.cc:1467] Processing CreateTablet for tablet d4d07c9621134780bc7ece7bb061d934 (DEFAULT_TABLE table=rebalance_test_table_4 [id=e8f9ec3f55d044f8b23964df20e79f65]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:00.791525 2641 tablet_service.cc:1467] Processing CreateTablet for tablet d4d07c9621134780bc7ece7bb061d934 (DEFAULT_TABLE table=rebalance_test_table_4 [id=e8f9ec3f55d044f8b23964df20e79f65]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:00.791687 2904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4d07c9621134780bc7ece7bb061d934. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:00.791776 2641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4d07c9621134780bc7ece7bb061d934. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:00.791800 3779 raft_consensus.cc:348] T 0b669a47f1d14ce98ccab24479d39304 P a1a844ecd71145e4a362f4c42500fbcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } }
I20240503 01:59:00.791918 3779 raft_consensus.cc:374] T 0b669a47f1d14ce98ccab24479d39304 P a1a844ecd71145e4a362f4c42500fbcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:00.791952 3779 raft_consensus.cc:729] T 0b669a47f1d14ce98ccab24479d39304 P a1a844ecd71145e4a362f4c42500fbcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1a844ecd71145e4a362f4c42500fbcb, State: Initialized, Role: FOLLOWER
I20240503 01:59:00.792130 3779 consensus_queue.cc:260] T 0b669a47f1d14ce98ccab24479d39304 P a1a844ecd71145e4a362f4c42500fbcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } }
I20240503 01:59:00.792208 3780 tablet_bootstrap.cc:492] T 0b669a47f1d14ce98ccab24479d39304 P 52c196c15e3348b985c689652bc49918: No bootstrap required, opened a new log
I20240503 01:59:00.792382 3780 ts_tablet_manager.cc:1367] T 0b669a47f1d14ce98ccab24479d39304 P 52c196c15e3348b985c689652bc49918: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 01:59:00.792526 3779 ts_tablet_manager.cc:1398] T 0b669a47f1d14ce98ccab24479d39304 P a1a844ecd71145e4a362f4c42500fbcb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:00.792464 2642 tablet_service.cc:1467] Processing CreateTablet for tablet 0b669a47f1d14ce98ccab24479d39304 (DEFAULT_TABLE table=rebalance_test_table_4 [id=e8f9ec3f55d044f8b23964df20e79f65]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:00.792743 3779 tablet_bootstrap.cc:492] T d4d07c9621134780bc7ece7bb061d934 P a1a844ecd71145e4a362f4c42500fbcb: Bootstrap starting.
I20240503 01:59:00.792742 2642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b669a47f1d14ce98ccab24479d39304. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:00.793226 3780 raft_consensus.cc:348] T 0b669a47f1d14ce98ccab24479d39304 P 52c196c15e3348b985c689652bc49918 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } }
I20240503 01:59:00.793409 3780 raft_consensus.cc:374] T 0b669a47f1d14ce98ccab24479d39304 P 52c196c15e3348b985c689652bc49918 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:00.793557 3780 raft_consensus.cc:729] T 0b669a47f1d14ce98ccab24479d39304 P 52c196c15e3348b985c689652bc49918 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52c196c15e3348b985c689652bc49918, State: Initialized, Role: FOLLOWER
I20240503 01:59:00.793783 3780 consensus_queue.cc:260] T 0b669a47f1d14ce98ccab24479d39304 P 52c196c15e3348b985c689652bc49918 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } }
I20240503 01:59:00.794142 3780 ts_tablet_manager.cc:1398] T 0b669a47f1d14ce98ccab24479d39304 P 52c196c15e3348b985c689652bc49918: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 01:59:00.794991 3779 tablet_bootstrap.cc:654] T d4d07c9621134780bc7ece7bb061d934 P a1a844ecd71145e4a362f4c42500fbcb: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:00.795657 3784 tablet_bootstrap.cc:492] T f40f77238d1e4a34a3f7bea387c6406c P 327aa26a8e434a609ebcb382aa665fce: Bootstrap starting.
I20240503 01:59:00.796101 2772 tablet_service.cc:1467] Processing CreateTablet for tablet f40f77238d1e4a34a3f7bea387c6406c (DEFAULT_TABLE table=rebalance_test_table_4 [id=e8f9ec3f55d044f8b23964df20e79f65]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:00.796371 2772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f40f77238d1e4a34a3f7bea387c6406c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:00.796437 3779 tablet_bootstrap.cc:492] T d4d07c9621134780bc7ece7bb061d934 P a1a844ecd71145e4a362f4c42500fbcb: No bootstrap required, opened a new log
I20240503 01:59:00.796510 3779 ts_tablet_manager.cc:1367] T d4d07c9621134780bc7ece7bb061d934 P a1a844ecd71145e4a362f4c42500fbcb: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 01:59:00.796734 3784 tablet_bootstrap.cc:654] T f40f77238d1e4a34a3f7bea387c6406c P 327aa26a8e434a609ebcb382aa665fce: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:00.797448 2380 tablet_service.cc:1467] Processing CreateTablet for tablet d4d07c9621134780bc7ece7bb061d934 (DEFAULT_TABLE table=rebalance_test_table_4 [id=e8f9ec3f55d044f8b23964df20e79f65]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:00.797705 2380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4d07c9621134780bc7ece7bb061d934. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:00.797610 3779 raft_consensus.cc:348] T d4d07c9621134780bc7ece7bb061d934 P a1a844ecd71145e4a362f4c42500fbcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } }
I20240503 01:59:00.797747 3779 raft_consensus.cc:374] T d4d07c9621134780bc7ece7bb061d934 P a1a844ecd71145e4a362f4c42500fbcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:00.797782 3779 raft_consensus.cc:729] T d4d07c9621134780bc7ece7bb061d934 P a1a844ecd71145e4a362f4c42500fbcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1a844ecd71145e4a362f4c42500fbcb, State: Initialized, Role: FOLLOWER
I20240503 01:59:00.797907 3779 consensus_queue.cc:260] T d4d07c9621134780bc7ece7bb061d934 P a1a844ecd71145e4a362f4c42500fbcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } }
I20240503 01:59:00.798184 3779 ts_tablet_manager.cc:1398] T d4d07c9621134780bc7ece7bb061d934 P a1a844ecd71145e4a362f4c42500fbcb: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:00.798771 3781 tablet_bootstrap.cc:492] T 0b669a47f1d14ce98ccab24479d39304 P 6568f69a78c44bb985162d2b7d0a9376: Bootstrap starting.
I20240503 01:59:00.798848 3784 tablet_bootstrap.cc:492] T f40f77238d1e4a34a3f7bea387c6406c P 327aa26a8e434a609ebcb382aa665fce: No bootstrap required, opened a new log
I20240503 01:59:00.798926 3784 ts_tablet_manager.cc:1367] T f40f77238d1e4a34a3f7bea387c6406c P 327aa26a8e434a609ebcb382aa665fce: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 01:59:00.799317 3784 raft_consensus.cc:348] T f40f77238d1e4a34a3f7bea387c6406c P 327aa26a8e434a609ebcb382aa665fce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } }
I20240503 01:59:00.799397 3784 raft_consensus.cc:374] T f40f77238d1e4a34a3f7bea387c6406c P 327aa26a8e434a609ebcb382aa665fce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:00.799417 3784 raft_consensus.cc:729] T f40f77238d1e4a34a3f7bea387c6406c P 327aa26a8e434a609ebcb382aa665fce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 327aa26a8e434a609ebcb382aa665fce, State: Initialized, Role: FOLLOWER
I20240503 01:59:00.799526 3784 consensus_queue.cc:260] T f40f77238d1e4a34a3f7bea387c6406c P 327aa26a8e434a609ebcb382aa665fce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } }
I20240503 01:59:00.799685 3781 tablet_bootstrap.cc:654] T 0b669a47f1d14ce98ccab24479d39304 P 6568f69a78c44bb985162d2b7d0a9376: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:00.799783 3784 ts_tablet_manager.cc:1398] T f40f77238d1e4a34a3f7bea387c6406c P 327aa26a8e434a609ebcb382aa665fce: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:00.799937 3784 tablet_bootstrap.cc:492] T d4d07c9621134780bc7ece7bb061d934 P 327aa26a8e434a609ebcb382aa665fce: Bootstrap starting.
I20240503 01:59:00.799897 2378 tablet_service.cc:1467] Processing CreateTablet for tablet f40f77238d1e4a34a3f7bea387c6406c (DEFAULT_TABLE table=rebalance_test_table_4 [id=e8f9ec3f55d044f8b23964df20e79f65]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:00.800125 2378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f40f77238d1e4a34a3f7bea387c6406c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:00.800797 3784 tablet_bootstrap.cc:654] T d4d07c9621134780bc7ece7bb061d934 P 327aa26a8e434a609ebcb382aa665fce: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:00.800863 2510 tablet_service.cc:1467] Processing CreateTablet for tablet f40f77238d1e4a34a3f7bea387c6406c (DEFAULT_TABLE table=rebalance_test_table_4 [id=e8f9ec3f55d044f8b23964df20e79f65]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:00.801095 2510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f40f77238d1e4a34a3f7bea387c6406c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:00.801142 3781 tablet_bootstrap.cc:492] T 0b669a47f1d14ce98ccab24479d39304 P 6568f69a78c44bb985162d2b7d0a9376: No bootstrap required, opened a new log
I20240503 01:59:00.801216 3781 ts_tablet_manager.cc:1367] T 0b669a47f1d14ce98ccab24479d39304 P 6568f69a78c44bb985162d2b7d0a9376: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:00.802106 3781 raft_consensus.cc:348] T 0b669a47f1d14ce98ccab24479d39304 P 6568f69a78c44bb985162d2b7d0a9376 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } }
I20240503 01:59:00.802224 3781 raft_consensus.cc:374] T 0b669a47f1d14ce98ccab24479d39304 P 6568f69a78c44bb985162d2b7d0a9376 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:00.802258 3781 raft_consensus.cc:729] T 0b669a47f1d14ce98ccab24479d39304 P 6568f69a78c44bb985162d2b7d0a9376 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6568f69a78c44bb985162d2b7d0a9376, State: Initialized, Role: FOLLOWER
I20240503 01:59:00.802631 3780 tablet_bootstrap.cc:492] T f40f77238d1e4a34a3f7bea387c6406c P 52c196c15e3348b985c689652bc49918: Bootstrap starting.
I20240503 01:59:00.803220 3779 tablet_bootstrap.cc:492] T f40f77238d1e4a34a3f7bea387c6406c P a1a844ecd71145e4a362f4c42500fbcb: Bootstrap starting.
I20240503 01:59:00.803714 3780 tablet_bootstrap.cc:654] T f40f77238d1e4a34a3f7bea387c6406c P 52c196c15e3348b985c689652bc49918: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:00.804201 3784 tablet_bootstrap.cc:492] T d4d07c9621134780bc7ece7bb061d934 P 327aa26a8e434a609ebcb382aa665fce: No bootstrap required, opened a new log
I20240503 01:59:00.804311 3779 tablet_bootstrap.cc:654] T f40f77238d1e4a34a3f7bea387c6406c P a1a844ecd71145e4a362f4c42500fbcb: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:00.804306 3784 ts_tablet_manager.cc:1367] T d4d07c9621134780bc7ece7bb061d934 P 327aa26a8e434a609ebcb382aa665fce: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240503 01:59:00.804498 3787 tablet_bootstrap.cc:492] T 0b669a47f1d14ce98ccab24479d39304 P b63928177fa54227a5ff9929265ab9dd: Bootstrap starting.
I20240503 01:59:00.804813 3784 raft_consensus.cc:348] T d4d07c9621134780bc7ece7bb061d934 P 327aa26a8e434a609ebcb382aa665fce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } }
I20240503 01:59:00.804935 3784 raft_consensus.cc:374] T d4d07c9621134780bc7ece7bb061d934 P 327aa26a8e434a609ebcb382aa665fce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:00.804977 3784 raft_consensus.cc:729] T d4d07c9621134780bc7ece7bb061d934 P 327aa26a8e434a609ebcb382aa665fce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 327aa26a8e434a609ebcb382aa665fce, State: Initialized, Role: FOLLOWER
I20240503 01:59:00.802450 3781 consensus_queue.cc:260] T 0b669a47f1d14ce98ccab24479d39304 P 6568f69a78c44bb985162d2b7d0a9376 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } }
I20240503 01:59:00.805089 3784 consensus_queue.cc:260] T d4d07c9621134780bc7ece7bb061d934 P 327aa26a8e434a609ebcb382aa665fce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } }
I20240503 01:59:00.805567 3787 tablet_bootstrap.cc:654] T 0b669a47f1d14ce98ccab24479d39304 P b63928177fa54227a5ff9929265ab9dd: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:00.806453 3781 ts_tablet_manager.cc:1398] T 0b669a47f1d14ce98ccab24479d39304 P 6568f69a78c44bb985162d2b7d0a9376: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240503 01:59:00.806653 3781 tablet_bootstrap.cc:492] T f40f77238d1e4a34a3f7bea387c6406c P 6568f69a78c44bb985162d2b7d0a9376: Bootstrap starting.
I20240503 01:59:00.807540 3781 tablet_bootstrap.cc:654] T f40f77238d1e4a34a3f7bea387c6406c P 6568f69a78c44bb985162d2b7d0a9376: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:00.808272 3784 ts_tablet_manager.cc:1398] T d4d07c9621134780bc7ece7bb061d934 P 327aa26a8e434a609ebcb382aa665fce: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 01:59:00.808279 3780 tablet_bootstrap.cc:492] T f40f77238d1e4a34a3f7bea387c6406c P 52c196c15e3348b985c689652bc49918: No bootstrap required, opened a new log
I20240503 01:59:00.808367 3780 ts_tablet_manager.cc:1367] T f40f77238d1e4a34a3f7bea387c6406c P 52c196c15e3348b985c689652bc49918: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s
I20240503 01:59:00.808590 3793 tablet_bootstrap.cc:492] T 0b669a47f1d14ce98ccab24479d39304 P b40a9a7b30d34a6db007ed8febd8a759: Bootstrap starting.
I20240503 01:59:00.808853 3780 raft_consensus.cc:348] T f40f77238d1e4a34a3f7bea387c6406c P 52c196c15e3348b985c689652bc49918 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } }
I20240503 01:59:00.808957 3780 raft_consensus.cc:374] T f40f77238d1e4a34a3f7bea387c6406c P 52c196c15e3348b985c689652bc49918 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:00.808988 3780 raft_consensus.cc:729] T f40f77238d1e4a34a3f7bea387c6406c P 52c196c15e3348b985c689652bc49918 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52c196c15e3348b985c689652bc49918, State: Initialized, Role: FOLLOWER
I20240503 01:59:00.809108 3780 consensus_queue.cc:260] T f40f77238d1e4a34a3f7bea387c6406c P 52c196c15e3348b985c689652bc49918 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } }
I20240503 01:59:00.809391 3780 ts_tablet_manager.cc:1398] T f40f77238d1e4a34a3f7bea387c6406c P 52c196c15e3348b985c689652bc49918: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:00.809695 3793 tablet_bootstrap.cc:654] T 0b669a47f1d14ce98ccab24479d39304 P b40a9a7b30d34a6db007ed8febd8a759: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:00.810537 3787 tablet_bootstrap.cc:492] T 0b669a47f1d14ce98ccab24479d39304 P b63928177fa54227a5ff9929265ab9dd: No bootstrap required, opened a new log
I20240503 01:59:00.810637 3787 ts_tablet_manager.cc:1367] T 0b669a47f1d14ce98ccab24479d39304 P b63928177fa54227a5ff9929265ab9dd: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 01:59:00.811089 3787 raft_consensus.cc:348] T 0b669a47f1d14ce98ccab24479d39304 P b63928177fa54227a5ff9929265ab9dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } }
I20240503 01:59:00.811216 3787 raft_consensus.cc:374] T 0b669a47f1d14ce98ccab24479d39304 P b63928177fa54227a5ff9929265ab9dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:00.811254 3787 raft_consensus.cc:729] T 0b669a47f1d14ce98ccab24479d39304 P b63928177fa54227a5ff9929265ab9dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b63928177fa54227a5ff9929265ab9dd, State: Initialized, Role: FOLLOWER
I20240503 01:59:00.811376 3787 consensus_queue.cc:260] T 0b669a47f1d14ce98ccab24479d39304 P b63928177fa54227a5ff9929265ab9dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } }
I20240503 01:59:00.811735 3787 ts_tablet_manager.cc:1398] T 0b669a47f1d14ce98ccab24479d39304 P b63928177fa54227a5ff9929265ab9dd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:00.812139 3793 tablet_bootstrap.cc:492] T 0b669a47f1d14ce98ccab24479d39304 P b40a9a7b30d34a6db007ed8febd8a759: No bootstrap required, opened a new log
I20240503 01:59:00.812170 3781 tablet_bootstrap.cc:492] T f40f77238d1e4a34a3f7bea387c6406c P 6568f69a78c44bb985162d2b7d0a9376: No bootstrap required, opened a new log
I20240503 01:59:00.812222 3793 ts_tablet_manager.cc:1367] T 0b669a47f1d14ce98ccab24479d39304 P b40a9a7b30d34a6db007ed8febd8a759: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 01:59:00.812243 3781 ts_tablet_manager.cc:1367] T f40f77238d1e4a34a3f7bea387c6406c P 6568f69a78c44bb985162d2b7d0a9376: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 01:59:00.812408 3787 tablet_bootstrap.cc:492] T d4d07c9621134780bc7ece7bb061d934 P b63928177fa54227a5ff9929265ab9dd: Bootstrap starting.
I20240503 01:59:00.812692 3781 raft_consensus.cc:348] T f40f77238d1e4a34a3f7bea387c6406c P 6568f69a78c44bb985162d2b7d0a9376 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } }
I20240503 01:59:00.812808 3781 raft_consensus.cc:374] T f40f77238d1e4a34a3f7bea387c6406c P 6568f69a78c44bb985162d2b7d0a9376 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:00.812844 3781 raft_consensus.cc:729] T f40f77238d1e4a34a3f7bea387c6406c P 6568f69a78c44bb985162d2b7d0a9376 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6568f69a78c44bb985162d2b7d0a9376, State: Initialized, Role: FOLLOWER
I20240503 01:59:00.812812 3793 raft_consensus.cc:348] T 0b669a47f1d14ce98ccab24479d39304 P b40a9a7b30d34a6db007ed8febd8a759 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } }
I20240503 01:59:00.812937 3793 raft_consensus.cc:374] T 0b669a47f1d14ce98ccab24479d39304 P b40a9a7b30d34a6db007ed8febd8a759 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:00.812981 3793 raft_consensus.cc:729] T 0b669a47f1d14ce98ccab24479d39304 P b40a9a7b30d34a6db007ed8febd8a759 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b40a9a7b30d34a6db007ed8febd8a759, State: Initialized, Role: FOLLOWER
I20240503 01:59:00.812954 3781 consensus_queue.cc:260] T f40f77238d1e4a34a3f7bea387c6406c P 6568f69a78c44bb985162d2b7d0a9376 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } }
I20240503 01:59:00.813117 3793 consensus_queue.cc:260] T 0b669a47f1d14ce98ccab24479d39304 P b40a9a7b30d34a6db007ed8febd8a759 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } }
I20240503 01:59:00.813231 3781 ts_tablet_manager.cc:1398] T f40f77238d1e4a34a3f7bea387c6406c P 6568f69a78c44bb985162d2b7d0a9376: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:00.813395 3781 tablet_bootstrap.cc:492] T d4d07c9621134780bc7ece7bb061d934 P 6568f69a78c44bb985162d2b7d0a9376: Bootstrap starting.
I20240503 01:59:00.813423 3793 ts_tablet_manager.cc:1398] T 0b669a47f1d14ce98ccab24479d39304 P b40a9a7b30d34a6db007ed8febd8a759: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:00.813609 3793 tablet_bootstrap.cc:492] T d4d07c9621134780bc7ece7bb061d934 P b40a9a7b30d34a6db007ed8febd8a759: Bootstrap starting.
I20240503 01:59:00.814584 3793 tablet_bootstrap.cc:654] T d4d07c9621134780bc7ece7bb061d934 P b40a9a7b30d34a6db007ed8febd8a759: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:00.814769 3781 tablet_bootstrap.cc:654] T d4d07c9621134780bc7ece7bb061d934 P 6568f69a78c44bb985162d2b7d0a9376: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:00.816066 3787 tablet_bootstrap.cc:654] T d4d07c9621134780bc7ece7bb061d934 P b63928177fa54227a5ff9929265ab9dd: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:00.817479 3793 tablet_bootstrap.cc:492] T d4d07c9621134780bc7ece7bb061d934 P b40a9a7b30d34a6db007ed8febd8a759: No bootstrap required, opened a new log
I20240503 01:59:00.817593 3793 ts_tablet_manager.cc:1367] T d4d07c9621134780bc7ece7bb061d934 P b40a9a7b30d34a6db007ed8febd8a759: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 01:59:00.817649 3781 tablet_bootstrap.cc:492] T d4d07c9621134780bc7ece7bb061d934 P 6568f69a78c44bb985162d2b7d0a9376: No bootstrap required, opened a new log
I20240503 01:59:00.817734 3781 ts_tablet_manager.cc:1367] T d4d07c9621134780bc7ece7bb061d934 P 6568f69a78c44bb985162d2b7d0a9376: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 01:59:00.817824 3787 tablet_bootstrap.cc:492] T d4d07c9621134780bc7ece7bb061d934 P b63928177fa54227a5ff9929265ab9dd: No bootstrap required, opened a new log
I20240503 01:59:00.817919 3787 ts_tablet_manager.cc:1367] T d4d07c9621134780bc7ece7bb061d934 P b63928177fa54227a5ff9929265ab9dd: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 01:59:00.818114 3781 raft_consensus.cc:348] T d4d07c9621134780bc7ece7bb061d934 P 6568f69a78c44bb985162d2b7d0a9376 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } }
I20240503 01:59:00.818200 3781 raft_consensus.cc:374] T d4d07c9621134780bc7ece7bb061d934 P 6568f69a78c44bb985162d2b7d0a9376 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:00.818121 3793 raft_consensus.cc:348] T d4d07c9621134780bc7ece7bb061d934 P b40a9a7b30d34a6db007ed8febd8a759 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } }
I20240503 01:59:00.818234 3781 raft_consensus.cc:729] T d4d07c9621134780bc7ece7bb061d934 P 6568f69a78c44bb985162d2b7d0a9376 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6568f69a78c44bb985162d2b7d0a9376, State: Initialized, Role: FOLLOWER
I20240503 01:59:00.818269 3793 raft_consensus.cc:374] T d4d07c9621134780bc7ece7bb061d934 P b40a9a7b30d34a6db007ed8febd8a759 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:00.818303 3793 raft_consensus.cc:729] T d4d07c9621134780bc7ece7bb061d934 P b40a9a7b30d34a6db007ed8febd8a759 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b40a9a7b30d34a6db007ed8febd8a759, State: Initialized, Role: FOLLOWER
I20240503 01:59:00.818339 3781 consensus_queue.cc:260] T d4d07c9621134780bc7ece7bb061d934 P 6568f69a78c44bb985162d2b7d0a9376 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } }
I20240503 01:59:00.818413 3793 consensus_queue.cc:260] T d4d07c9621134780bc7ece7bb061d934 P b40a9a7b30d34a6db007ed8febd8a759 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } }
I20240503 01:59:00.818542 3787 raft_consensus.cc:348] T d4d07c9621134780bc7ece7bb061d934 P b63928177fa54227a5ff9929265ab9dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } }
I20240503 01:59:00.818653 3787 raft_consensus.cc:374] T d4d07c9621134780bc7ece7bb061d934 P b63928177fa54227a5ff9929265ab9dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:00.818653 3781 ts_tablet_manager.cc:1398] T d4d07c9621134780bc7ece7bb061d934 P 6568f69a78c44bb985162d2b7d0a9376: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:00.818684 3787 raft_consensus.cc:729] T d4d07c9621134780bc7ece7bb061d934 P b63928177fa54227a5ff9929265ab9dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b63928177fa54227a5ff9929265ab9dd, State: Initialized, Role: FOLLOWER
I20240503 01:59:00.818697 3793 ts_tablet_manager.cc:1398] T d4d07c9621134780bc7ece7bb061d934 P b40a9a7b30d34a6db007ed8febd8a759: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:00.818825 3787 consensus_queue.cc:260] T d4d07c9621134780bc7ece7bb061d934 P b63928177fa54227a5ff9929265ab9dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } }
I20240503 01:59:00.819095 3787 ts_tablet_manager.cc:1398] T d4d07c9621134780bc7ece7bb061d934 P b63928177fa54227a5ff9929265ab9dd: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 01:59:00.819308 3787 tablet_bootstrap.cc:492] T f40f77238d1e4a34a3f7bea387c6406c P b63928177fa54227a5ff9929265ab9dd: Bootstrap starting.
I20240503 01:59:00.820219 3787 tablet_bootstrap.cc:654] T f40f77238d1e4a34a3f7bea387c6406c P b63928177fa54227a5ff9929265ab9dd: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:00.821348 3779 tablet_bootstrap.cc:492] T f40f77238d1e4a34a3f7bea387c6406c P a1a844ecd71145e4a362f4c42500fbcb: No bootstrap required, opened a new log
I20240503 01:59:00.821470 3779 ts_tablet_manager.cc:1367] T f40f77238d1e4a34a3f7bea387c6406c P a1a844ecd71145e4a362f4c42500fbcb: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.002s
I20240503 01:59:00.821553 3787 tablet_bootstrap.cc:492] T f40f77238d1e4a34a3f7bea387c6406c P b63928177fa54227a5ff9929265ab9dd: No bootstrap required, opened a new log
I20240503 01:59:00.821640 3787 ts_tablet_manager.cc:1367] T f40f77238d1e4a34a3f7bea387c6406c P b63928177fa54227a5ff9929265ab9dd: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:00.821980 3779 raft_consensus.cc:348] T f40f77238d1e4a34a3f7bea387c6406c P a1a844ecd71145e4a362f4c42500fbcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } }
I20240503 01:59:00.822019 3787 raft_consensus.cc:348] T f40f77238d1e4a34a3f7bea387c6406c P b63928177fa54227a5ff9929265ab9dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } }
I20240503 01:59:00.822108 3787 raft_consensus.cc:374] T f40f77238d1e4a34a3f7bea387c6406c P b63928177fa54227a5ff9929265ab9dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:00.822108 3779 raft_consensus.cc:374] T f40f77238d1e4a34a3f7bea387c6406c P a1a844ecd71145e4a362f4c42500fbcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:00.822151 3787 raft_consensus.cc:729] T f40f77238d1e4a34a3f7bea387c6406c P b63928177fa54227a5ff9929265ab9dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b63928177fa54227a5ff9929265ab9dd, State: Initialized, Role: FOLLOWER
I20240503 01:59:00.822151 3779 raft_consensus.cc:729] T f40f77238d1e4a34a3f7bea387c6406c P a1a844ecd71145e4a362f4c42500fbcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1a844ecd71145e4a362f4c42500fbcb, State: Initialized, Role: FOLLOWER
I20240503 01:59:00.822281 3779 consensus_queue.cc:260] T f40f77238d1e4a34a3f7bea387c6406c P a1a844ecd71145e4a362f4c42500fbcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } }
I20240503 01:59:00.822294 3787 consensus_queue.cc:260] T f40f77238d1e4a34a3f7bea387c6406c P b63928177fa54227a5ff9929265ab9dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } }
I20240503 01:59:00.822573 3779 ts_tablet_manager.cc:1398] T f40f77238d1e4a34a3f7bea387c6406c P a1a844ecd71145e4a362f4c42500fbcb: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:00.822625 3787 ts_tablet_manager.cc:1398] T f40f77238d1e4a34a3f7bea387c6406c P b63928177fa54227a5ff9929265ab9dd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:00.893289 3545 raft_consensus.cc:482] T 0b669a47f1d14ce98ccab24479d39304 P b40a9a7b30d34a6db007ed8febd8a759 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:00.893441 3545 raft_consensus.cc:504] T 0b669a47f1d14ce98ccab24479d39304 P b40a9a7b30d34a6db007ed8febd8a759 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } }
I20240503 01:59:00.893954 3545 leader_election.cc:288] T 0b669a47f1d14ce98ccab24479d39304 P b40a9a7b30d34a6db007ed8febd8a759 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b63928177fa54227a5ff9929265ab9dd (127.0.99.129:36425), a1a844ecd71145e4a362f4c42500fbcb (127.0.99.130:46033), 52c196c15e3348b985c689652bc49918 (127.0.99.132:42887), 6568f69a78c44bb985162d2b7d0a9376 (127.0.99.134:43603)
I20240503 01:59:00.894333 2531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b669a47f1d14ce98ccab24479d39304" candidate_uuid: "b40a9a7b30d34a6db007ed8febd8a759" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1a844ecd71145e4a362f4c42500fbcb" is_pre_election: true
I20240503 01:59:00.894340 3054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b669a47f1d14ce98ccab24479d39304" candidate_uuid: "b40a9a7b30d34a6db007ed8febd8a759" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6568f69a78c44bb985162d2b7d0a9376" is_pre_election: true
I20240503 01:59:00.894387 2399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b669a47f1d14ce98ccab24479d39304" candidate_uuid: "b40a9a7b30d34a6db007ed8febd8a759" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b63928177fa54227a5ff9929265ab9dd" is_pre_election: true
I20240503 01:59:00.894423 2792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b669a47f1d14ce98ccab24479d39304" candidate_uuid: "b40a9a7b30d34a6db007ed8febd8a759" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52c196c15e3348b985c689652bc49918" is_pre_election: true
I20240503 01:59:00.894471 3054 raft_consensus.cc:2454] T 0b669a47f1d14ce98ccab24479d39304 P 6568f69a78c44bb985162d2b7d0a9376 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b40a9a7b30d34a6db007ed8febd8a759 in term 0.
I20240503 01:59:00.894471 2531 raft_consensus.cc:2454] T 0b669a47f1d14ce98ccab24479d39304 P a1a844ecd71145e4a362f4c42500fbcb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b40a9a7b30d34a6db007ed8febd8a759 in term 0.
I20240503 01:59:00.894541 2399 raft_consensus.cc:2454] T 0b669a47f1d14ce98ccab24479d39304 P b63928177fa54227a5ff9929265ab9dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b40a9a7b30d34a6db007ed8febd8a759 in term 0.
I20240503 01:59:00.894560 2792 raft_consensus.cc:2454] T 0b669a47f1d14ce98ccab24479d39304 P 52c196c15e3348b985c689652bc49918 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b40a9a7b30d34a6db007ed8febd8a759 in term 0.
I20240503 01:59:00.894867 2596 leader_election.cc:302] T 0b669a47f1d14ce98ccab24479d39304 P b40a9a7b30d34a6db007ed8febd8a759 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 6568f69a78c44bb985162d2b7d0a9376, b40a9a7b30d34a6db007ed8febd8a759, b63928177fa54227a5ff9929265ab9dd; no voters:
I20240503 01:59:00.895047 3545 raft_consensus.cc:2781] T 0b669a47f1d14ce98ccab24479d39304 P b40a9a7b30d34a6db007ed8febd8a759 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:00.895097 3545 raft_consensus.cc:482] T 0b669a47f1d14ce98ccab24479d39304 P b40a9a7b30d34a6db007ed8febd8a759 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:00.895124 3545 raft_consensus.cc:3037] T 0b669a47f1d14ce98ccab24479d39304 P b40a9a7b30d34a6db007ed8febd8a759 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:00.896001 3545 raft_consensus.cc:504] T 0b669a47f1d14ce98ccab24479d39304 P b40a9a7b30d34a6db007ed8febd8a759 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } }
I20240503 01:59:00.896396 3545 leader_election.cc:288] T 0b669a47f1d14ce98ccab24479d39304 P b40a9a7b30d34a6db007ed8febd8a759 [CANDIDATE]: Term 1 election: Requested vote from peers b63928177fa54227a5ff9929265ab9dd (127.0.99.129:36425), a1a844ecd71145e4a362f4c42500fbcb (127.0.99.130:46033), 52c196c15e3348b985c689652bc49918 (127.0.99.132:42887), 6568f69a78c44bb985162d2b7d0a9376 (127.0.99.134:43603)
I20240503 01:59:00.896739 3054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b669a47f1d14ce98ccab24479d39304" candidate_uuid: "b40a9a7b30d34a6db007ed8febd8a759" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6568f69a78c44bb985162d2b7d0a9376"
I20240503 01:59:00.896845 3054 raft_consensus.cc:3037] T 0b669a47f1d14ce98ccab24479d39304 P 6568f69a78c44bb985162d2b7d0a9376 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:00.896857 2531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b669a47f1d14ce98ccab24479d39304" candidate_uuid: "b40a9a7b30d34a6db007ed8febd8a759" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1a844ecd71145e4a362f4c42500fbcb"
I20240503 01:59:00.896850 2792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b669a47f1d14ce98ccab24479d39304" candidate_uuid: "b40a9a7b30d34a6db007ed8febd8a759" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52c196c15e3348b985c689652bc49918"
I20240503 01:59:00.896868 2399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b669a47f1d14ce98ccab24479d39304" candidate_uuid: "b40a9a7b30d34a6db007ed8febd8a759" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b63928177fa54227a5ff9929265ab9dd"
I20240503 01:59:00.896955 2531 raft_consensus.cc:3037] T 0b669a47f1d14ce98ccab24479d39304 P a1a844ecd71145e4a362f4c42500fbcb [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:00.896955 2792 raft_consensus.cc:3037] T 0b669a47f1d14ce98ccab24479d39304 P 52c196c15e3348b985c689652bc49918 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:00.896962 2399 raft_consensus.cc:3037] T 0b669a47f1d14ce98ccab24479d39304 P b63928177fa54227a5ff9929265ab9dd [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:00.897818 3054 raft_consensus.cc:2454] T 0b669a47f1d14ce98ccab24479d39304 P 6568f69a78c44bb985162d2b7d0a9376 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b40a9a7b30d34a6db007ed8febd8a759 in term 1.
I20240503 01:59:00.897882 2792 raft_consensus.cc:2454] T 0b669a47f1d14ce98ccab24479d39304 P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b40a9a7b30d34a6db007ed8febd8a759 in term 1.
I20240503 01:59:00.897897 2531 raft_consensus.cc:2454] T 0b669a47f1d14ce98ccab24479d39304 P a1a844ecd71145e4a362f4c42500fbcb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b40a9a7b30d34a6db007ed8febd8a759 in term 1.
I20240503 01:59:00.897899 2399 raft_consensus.cc:2454] T 0b669a47f1d14ce98ccab24479d39304 P b63928177fa54227a5ff9929265ab9dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b40a9a7b30d34a6db007ed8febd8a759 in term 1.
I20240503 01:59:00.898187 2598 leader_election.cc:302] T 0b669a47f1d14ce98ccab24479d39304 P b40a9a7b30d34a6db007ed8febd8a759 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 6568f69a78c44bb985162d2b7d0a9376, a1a844ecd71145e4a362f4c42500fbcb, b40a9a7b30d34a6db007ed8febd8a759; no voters:
I20240503 01:59:00.898381 3545 raft_consensus.cc:2781] T 0b669a47f1d14ce98ccab24479d39304 P b40a9a7b30d34a6db007ed8febd8a759 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:00.898613 3545 raft_consensus.cc:686] T 0b669a47f1d14ce98ccab24479d39304 P b40a9a7b30d34a6db007ed8febd8a759 [term 1 LEADER]: Becoming Leader. State: Replica: b40a9a7b30d34a6db007ed8febd8a759, State: Running, Role: LEADER
I20240503 01:59:00.898774 3545 consensus_queue.cc:237] T 0b669a47f1d14ce98ccab24479d39304 P b40a9a7b30d34a6db007ed8febd8a759 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } }
I20240503 01:59:00.900599 2254 catalog_manager.cc:5484] T 0b669a47f1d14ce98ccab24479d39304 P b40a9a7b30d34a6db007ed8febd8a759 reported cstate change: term changed from 0 to 1, leader changed from <none> to b40a9a7b30d34a6db007ed8febd8a759 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "b40a9a7b30d34a6db007ed8febd8a759" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } health_report { overall_health: HEALTHY } } }
I20240503 01:59:00.989401 3633 raft_consensus.cc:482] T f40f77238d1e4a34a3f7bea387c6406c P b63928177fa54227a5ff9929265ab9dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:00.989509 3633 raft_consensus.cc:504] T f40f77238d1e4a34a3f7bea387c6406c P b63928177fa54227a5ff9929265ab9dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } }
I20240503 01:59:00.989941 3633 leader_election.cc:288] T f40f77238d1e4a34a3f7bea387c6406c P b63928177fa54227a5ff9929265ab9dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a1a844ecd71145e4a362f4c42500fbcb (127.0.99.130:46033), 327aa26a8e434a609ebcb382aa665fce (127.0.99.133:38853), 52c196c15e3348b985c689652bc49918 (127.0.99.132:42887), 6568f69a78c44bb985162d2b7d0a9376 (127.0.99.134:43603)
I20240503 01:59:00.990280 2923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40f77238d1e4a34a3f7bea387c6406c" candidate_uuid: "b63928177fa54227a5ff9929265ab9dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "327aa26a8e434a609ebcb382aa665fce" is_pre_election: true
I20240503 01:59:00.990296 2531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40f77238d1e4a34a3f7bea387c6406c" candidate_uuid: "b63928177fa54227a5ff9929265ab9dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1a844ecd71145e4a362f4c42500fbcb" is_pre_election: true
I20240503 01:59:00.990290 3054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40f77238d1e4a34a3f7bea387c6406c" candidate_uuid: "b63928177fa54227a5ff9929265ab9dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6568f69a78c44bb985162d2b7d0a9376" is_pre_election: true
I20240503 01:59:00.990377 2792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40f77238d1e4a34a3f7bea387c6406c" candidate_uuid: "b63928177fa54227a5ff9929265ab9dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52c196c15e3348b985c689652bc49918" is_pre_election: true
I20240503 01:59:00.990419 2923 raft_consensus.cc:2454] T f40f77238d1e4a34a3f7bea387c6406c P 327aa26a8e434a609ebcb382aa665fce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b63928177fa54227a5ff9929265ab9dd in term 0.
I20240503 01:59:00.990433 3054 raft_consensus.cc:2454] T f40f77238d1e4a34a3f7bea387c6406c P 6568f69a78c44bb985162d2b7d0a9376 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b63928177fa54227a5ff9929265ab9dd in term 0.
I20240503 01:59:00.990433 2531 raft_consensus.cc:2454] T f40f77238d1e4a34a3f7bea387c6406c P a1a844ecd71145e4a362f4c42500fbcb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b63928177fa54227a5ff9929265ab9dd in term 0.
I20240503 01:59:00.990511 2792 raft_consensus.cc:2454] T f40f77238d1e4a34a3f7bea387c6406c P 52c196c15e3348b985c689652bc49918 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b63928177fa54227a5ff9929265ab9dd in term 0.
I20240503 01:59:00.990767 2334 leader_election.cc:302] T f40f77238d1e4a34a3f7bea387c6406c P b63928177fa54227a5ff9929265ab9dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 327aa26a8e434a609ebcb382aa665fce, a1a844ecd71145e4a362f4c42500fbcb, b63928177fa54227a5ff9929265ab9dd; no voters:
I20240503 01:59:00.990937 3633 raft_consensus.cc:2781] T f40f77238d1e4a34a3f7bea387c6406c P b63928177fa54227a5ff9929265ab9dd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:00.990996 3633 raft_consensus.cc:482] T f40f77238d1e4a34a3f7bea387c6406c P b63928177fa54227a5ff9929265ab9dd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:00.991039 3633 raft_consensus.cc:3037] T f40f77238d1e4a34a3f7bea387c6406c P b63928177fa54227a5ff9929265ab9dd [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:00.991843 3633 raft_consensus.cc:504] T f40f77238d1e4a34a3f7bea387c6406c P b63928177fa54227a5ff9929265ab9dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } }
I20240503 01:59:00.992260 3633 leader_election.cc:288] T f40f77238d1e4a34a3f7bea387c6406c P b63928177fa54227a5ff9929265ab9dd [CANDIDATE]: Term 1 election: Requested vote from peers a1a844ecd71145e4a362f4c42500fbcb (127.0.99.130:46033), 327aa26a8e434a609ebcb382aa665fce (127.0.99.133:38853), 52c196c15e3348b985c689652bc49918 (127.0.99.132:42887), 6568f69a78c44bb985162d2b7d0a9376 (127.0.99.134:43603)
I20240503 01:59:00.992481 2531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40f77238d1e4a34a3f7bea387c6406c" candidate_uuid: "b63928177fa54227a5ff9929265ab9dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1a844ecd71145e4a362f4c42500fbcb"
I20240503 01:59:00.992518 2792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40f77238d1e4a34a3f7bea387c6406c" candidate_uuid: "b63928177fa54227a5ff9929265ab9dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52c196c15e3348b985c689652bc49918"
I20240503 01:59:00.992569 2923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40f77238d1e4a34a3f7bea387c6406c" candidate_uuid: "b63928177fa54227a5ff9929265ab9dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "327aa26a8e434a609ebcb382aa665fce"
I20240503 01:59:00.992601 2531 raft_consensus.cc:3037] T f40f77238d1e4a34a3f7bea387c6406c P a1a844ecd71145e4a362f4c42500fbcb [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:00.992614 2792 raft_consensus.cc:3037] T f40f77238d1e4a34a3f7bea387c6406c P 52c196c15e3348b985c689652bc49918 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:00.992683 2923 raft_consensus.cc:3037] T f40f77238d1e4a34a3f7bea387c6406c P 327aa26a8e434a609ebcb382aa665fce [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:00.993358 2923 raft_consensus.cc:2454] T f40f77238d1e4a34a3f7bea387c6406c P 327aa26a8e434a609ebcb382aa665fce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b63928177fa54227a5ff9929265ab9dd in term 1.
I20240503 01:59:00.993523 2531 raft_consensus.cc:2454] T f40f77238d1e4a34a3f7bea387c6406c P a1a844ecd71145e4a362f4c42500fbcb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b63928177fa54227a5ff9929265ab9dd in term 1.
I20240503 01:59:00.993523 2792 raft_consensus.cc:2454] T f40f77238d1e4a34a3f7bea387c6406c P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b63928177fa54227a5ff9929265ab9dd in term 1.
I20240503 01:59:00.993567 3054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40f77238d1e4a34a3f7bea387c6406c" candidate_uuid: "b63928177fa54227a5ff9929265ab9dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6568f69a78c44bb985162d2b7d0a9376"
I20240503 01:59:00.993674 3054 raft_consensus.cc:3037] T f40f77238d1e4a34a3f7bea387c6406c P 6568f69a78c44bb985162d2b7d0a9376 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:00.993800 2336 leader_election.cc:302] T f40f77238d1e4a34a3f7bea387c6406c P b63928177fa54227a5ff9929265ab9dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 327aa26a8e434a609ebcb382aa665fce, 52c196c15e3348b985c689652bc49918, b63928177fa54227a5ff9929265ab9dd; no voters:
I20240503 01:59:00.994014 3633 raft_consensus.cc:2781] T f40f77238d1e4a34a3f7bea387c6406c P b63928177fa54227a5ff9929265ab9dd [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:00.994187 3633 raft_consensus.cc:686] T f40f77238d1e4a34a3f7bea387c6406c P b63928177fa54227a5ff9929265ab9dd [term 1 LEADER]: Becoming Leader. State: Replica: b63928177fa54227a5ff9929265ab9dd, State: Running, Role: LEADER
I20240503 01:59:00.994364 3633 consensus_queue.cc:237] T f40f77238d1e4a34a3f7bea387c6406c P b63928177fa54227a5ff9929265ab9dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } }
I20240503 01:59:00.994583 3054 raft_consensus.cc:2454] T f40f77238d1e4a34a3f7bea387c6406c P 6568f69a78c44bb985162d2b7d0a9376 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b63928177fa54227a5ff9929265ab9dd in term 1.
I20240503 01:59:00.996392 2254 catalog_manager.cc:5484] T f40f77238d1e4a34a3f7bea387c6406c P b63928177fa54227a5ff9929265ab9dd reported cstate change: term changed from 0 to 1, leader changed from <none> to b63928177fa54227a5ff9929265ab9dd (127.0.99.129). New cstate: current_term: 1 leader_uuid: "b63928177fa54227a5ff9929265ab9dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:01.010299 3744 raft_consensus.cc:482] T d4d07c9621134780bc7ece7bb061d934 P 327aa26a8e434a609ebcb382aa665fce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:01.010386 3744 raft_consensus.cc:504] T d4d07c9621134780bc7ece7bb061d934 P 327aa26a8e434a609ebcb382aa665fce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } }
I20240503 01:59:01.010896 3744 leader_election.cc:288] T d4d07c9621134780bc7ece7bb061d934 P 327aa26a8e434a609ebcb382aa665fce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6568f69a78c44bb985162d2b7d0a9376 (127.0.99.134:43603), a1a844ecd71145e4a362f4c42500fbcb (127.0.99.130:46033), b63928177fa54227a5ff9929265ab9dd (127.0.99.129:36425), b40a9a7b30d34a6db007ed8febd8a759 (127.0.99.131:34079)
I20240503 01:59:01.011148 3054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4d07c9621134780bc7ece7bb061d934" candidate_uuid: "327aa26a8e434a609ebcb382aa665fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6568f69a78c44bb985162d2b7d0a9376" is_pre_election: true
I20240503 01:59:01.011273 3054 raft_consensus.cc:2454] T d4d07c9621134780bc7ece7bb061d934 P 6568f69a78c44bb985162d2b7d0a9376 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 327aa26a8e434a609ebcb382aa665fce in term 0.
I20240503 01:59:01.011277 2661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4d07c9621134780bc7ece7bb061d934" candidate_uuid: "327aa26a8e434a609ebcb382aa665fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b40a9a7b30d34a6db007ed8febd8a759" is_pre_election: true
I20240503 01:59:01.011404 2661 raft_consensus.cc:2454] T d4d07c9621134780bc7ece7bb061d934 P b40a9a7b30d34a6db007ed8febd8a759 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 327aa26a8e434a609ebcb382aa665fce in term 0.
I20240503 01:59:01.011430 2399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4d07c9621134780bc7ece7bb061d934" candidate_uuid: "327aa26a8e434a609ebcb382aa665fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b63928177fa54227a5ff9929265ab9dd" is_pre_election: true
I20240503 01:59:01.011549 2399 raft_consensus.cc:2454] T d4d07c9621134780bc7ece7bb061d934 P b63928177fa54227a5ff9929265ab9dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 327aa26a8e434a609ebcb382aa665fce in term 0.
I20240503 01:59:01.011844 2858 leader_election.cc:302] T d4d07c9621134780bc7ece7bb061d934 P 327aa26a8e434a609ebcb382aa665fce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 327aa26a8e434a609ebcb382aa665fce, 6568f69a78c44bb985162d2b7d0a9376, b40a9a7b30d34a6db007ed8febd8a759; no voters:
I20240503 01:59:01.011924 2531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4d07c9621134780bc7ece7bb061d934" candidate_uuid: "327aa26a8e434a609ebcb382aa665fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1a844ecd71145e4a362f4c42500fbcb" is_pre_election: true
I20240503 01:59:01.012010 3744 raft_consensus.cc:2781] T d4d07c9621134780bc7ece7bb061d934 P 327aa26a8e434a609ebcb382aa665fce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:01.012051 3744 raft_consensus.cc:482] T d4d07c9621134780bc7ece7bb061d934 P 327aa26a8e434a609ebcb382aa665fce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:01.012049 2531 raft_consensus.cc:2454] T d4d07c9621134780bc7ece7bb061d934 P a1a844ecd71145e4a362f4c42500fbcb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 327aa26a8e434a609ebcb382aa665fce in term 0.
I20240503 01:59:01.012089 3744 raft_consensus.cc:3037] T d4d07c9621134780bc7ece7bb061d934 P 327aa26a8e434a609ebcb382aa665fce [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:01.012993 3744 raft_consensus.cc:504] T d4d07c9621134780bc7ece7bb061d934 P 327aa26a8e434a609ebcb382aa665fce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } }
I20240503 01:59:01.013448 3744 leader_election.cc:288] T d4d07c9621134780bc7ece7bb061d934 P 327aa26a8e434a609ebcb382aa665fce [CANDIDATE]: Term 1 election: Requested vote from peers 6568f69a78c44bb985162d2b7d0a9376 (127.0.99.134:43603), a1a844ecd71145e4a362f4c42500fbcb (127.0.99.130:46033), b63928177fa54227a5ff9929265ab9dd (127.0.99.129:36425), b40a9a7b30d34a6db007ed8febd8a759 (127.0.99.131:34079)
I20240503 01:59:01.013679 2531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4d07c9621134780bc7ece7bb061d934" candidate_uuid: "327aa26a8e434a609ebcb382aa665fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1a844ecd71145e4a362f4c42500fbcb"
I20240503 01:59:01.013774 2531 raft_consensus.cc:3037] T d4d07c9621134780bc7ece7bb061d934 P a1a844ecd71145e4a362f4c42500fbcb [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:01.013805 2661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4d07c9621134780bc7ece7bb061d934" candidate_uuid: "327aa26a8e434a609ebcb382aa665fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b40a9a7b30d34a6db007ed8febd8a759"
I20240503 01:59:01.013896 2661 raft_consensus.cc:3037] T d4d07c9621134780bc7ece7bb061d934 P b40a9a7b30d34a6db007ed8febd8a759 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:01.013917 2399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4d07c9621134780bc7ece7bb061d934" candidate_uuid: "327aa26a8e434a609ebcb382aa665fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b63928177fa54227a5ff9929265ab9dd"
I20240503 01:59:01.014020 2399 raft_consensus.cc:3037] T d4d07c9621134780bc7ece7bb061d934 P b63928177fa54227a5ff9929265ab9dd [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:01.014645 2531 raft_consensus.cc:2454] T d4d07c9621134780bc7ece7bb061d934 P a1a844ecd71145e4a362f4c42500fbcb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 327aa26a8e434a609ebcb382aa665fce in term 1.
I20240503 01:59:01.014681 2399 raft_consensus.cc:2454] T d4d07c9621134780bc7ece7bb061d934 P b63928177fa54227a5ff9929265ab9dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 327aa26a8e434a609ebcb382aa665fce in term 1.
I20240503 01:59:01.014914 2661 raft_consensus.cc:2454] T d4d07c9621134780bc7ece7bb061d934 P b40a9a7b30d34a6db007ed8febd8a759 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 327aa26a8e434a609ebcb382aa665fce in term 1.
I20240503 01:59:01.014966 2860 leader_election.cc:302] T d4d07c9621134780bc7ece7bb061d934 P 327aa26a8e434a609ebcb382aa665fce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 327aa26a8e434a609ebcb382aa665fce, a1a844ecd71145e4a362f4c42500fbcb, b63928177fa54227a5ff9929265ab9dd; no voters:
I20240503 01:59:01.014998 3054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4d07c9621134780bc7ece7bb061d934" candidate_uuid: "327aa26a8e434a609ebcb382aa665fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6568f69a78c44bb985162d2b7d0a9376"
I20240503 01:59:01.015100 3744 raft_consensus.cc:2781] T d4d07c9621134780bc7ece7bb061d934 P 327aa26a8e434a609ebcb382aa665fce [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:01.015125 3054 raft_consensus.cc:3037] T d4d07c9621134780bc7ece7bb061d934 P 6568f69a78c44bb985162d2b7d0a9376 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:01.015190 3744 raft_consensus.cc:686] T d4d07c9621134780bc7ece7bb061d934 P 327aa26a8e434a609ebcb382aa665fce [term 1 LEADER]: Becoming Leader. State: Replica: 327aa26a8e434a609ebcb382aa665fce, State: Running, Role: LEADER
I20240503 01:59:01.015349 3744 consensus_queue.cc:237] T d4d07c9621134780bc7ece7bb061d934 P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } }
I20240503 01:59:01.016059 3054 raft_consensus.cc:2454] T d4d07c9621134780bc7ece7bb061d934 P 6568f69a78c44bb985162d2b7d0a9376 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 327aa26a8e434a609ebcb382aa665fce in term 1.
I20240503 01:59:01.017138 2254 catalog_manager.cc:5484] T d4d07c9621134780bc7ece7bb061d934 P 327aa26a8e434a609ebcb382aa665fce reported cstate change: term changed from 0 to 1, leader changed from <none> to 327aa26a8e434a609ebcb382aa665fce (127.0.99.133). New cstate: current_term: 1 leader_uuid: "327aa26a8e434a609ebcb382aa665fce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } health_report { overall_health: UNKNOWN } } }
W20240503 01:59:01.152597 3807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:01.152830 3807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:01.156004 3807 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240503 01:59:01.233209 3054 raft_consensus.cc:1261] T f40f77238d1e4a34a3f7bea387c6406c P 6568f69a78c44bb985162d2b7d0a9376 [term 1 FOLLOWER]: Refusing update from remote peer b63928177fa54227a5ff9929265ab9dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:01.233435 2792 raft_consensus.cc:1261] T f40f77238d1e4a34a3f7bea387c6406c P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Refusing update from remote peer b63928177fa54227a5ff9929265ab9dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:01.233448 2531 raft_consensus.cc:1261] T f40f77238d1e4a34a3f7bea387c6406c P a1a844ecd71145e4a362f4c42500fbcb [term 1 FOLLOWER]: Refusing update from remote peer b63928177fa54227a5ff9929265ab9dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:01.233781 2923 raft_consensus.cc:1261] T f40f77238d1e4a34a3f7bea387c6406c P 327aa26a8e434a609ebcb382aa665fce [term 1 FOLLOWER]: Refusing update from remote peer b63928177fa54227a5ff9929265ab9dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:01.233832 3633 consensus_queue.cc:1035] T f40f77238d1e4a34a3f7bea387c6406c P b63928177fa54227a5ff9929265ab9dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:01.234731 3802 consensus_queue.cc:1035] T f40f77238d1e4a34a3f7bea387c6406c P b63928177fa54227a5ff9929265ab9dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:01.235069 3802 consensus_queue.cc:1035] T f40f77238d1e4a34a3f7bea387c6406c P b63928177fa54227a5ff9929265ab9dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:01.238355 3633 consensus_queue.cc:1035] T f40f77238d1e4a34a3f7bea387c6406c P b63928177fa54227a5ff9929265ab9dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:01.245333 2661 raft_consensus.cc:1261] T d4d07c9621134780bc7ece7bb061d934 P b40a9a7b30d34a6db007ed8febd8a759 [term 1 FOLLOWER]: Refusing update from remote peer 327aa26a8e434a609ebcb382aa665fce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:01.245759 3744 consensus_queue.cc:1035] T d4d07c9621134780bc7ece7bb061d934 P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:01.247578 2399 raft_consensus.cc:1261] T d4d07c9621134780bc7ece7bb061d934 P b63928177fa54227a5ff9929265ab9dd [term 1 FOLLOWER]: Refusing update from remote peer 327aa26a8e434a609ebcb382aa665fce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:01.247983 3744 consensus_queue.cc:1035] T d4d07c9621134780bc7ece7bb061d934 P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:01.251544 2530 raft_consensus.cc:1261] T d4d07c9621134780bc7ece7bb061d934 P a1a844ecd71145e4a362f4c42500fbcb [term 1 FOLLOWER]: Refusing update from remote peer 327aa26a8e434a609ebcb382aa665fce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:01.251931 3744 consensus_queue.cc:1035] T d4d07c9621134780bc7ece7bb061d934 P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:01.253046 3055 raft_consensus.cc:1261] T d4d07c9621134780bc7ece7bb061d934 P 6568f69a78c44bb985162d2b7d0a9376 [term 1 FOLLOWER]: Refusing update from remote peer 327aa26a8e434a609ebcb382aa665fce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:01.256007 3551 consensus_queue.cc:1035] T d4d07c9621134780bc7ece7bb061d934 P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:01.270144 3053 raft_consensus.cc:1261] T 0b669a47f1d14ce98ccab24479d39304 P 6568f69a78c44bb985162d2b7d0a9376 [term 1 FOLLOWER]: Refusing update from remote peer b40a9a7b30d34a6db007ed8febd8a759: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:01.270448 2793 raft_consensus.cc:1261] T 0b669a47f1d14ce98ccab24479d39304 P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Refusing update from remote peer b40a9a7b30d34a6db007ed8febd8a759: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:01.270567 3545 consensus_queue.cc:1035] T 0b669a47f1d14ce98ccab24479d39304 P b40a9a7b30d34a6db007ed8febd8a759 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:01.270756 2400 raft_consensus.cc:1261] T 0b669a47f1d14ce98ccab24479d39304 P b63928177fa54227a5ff9929265ab9dd [term 1 FOLLOWER]: Refusing update from remote peer b40a9a7b30d34a6db007ed8febd8a759: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:01.271436 3800 consensus_queue.cc:1035] T 0b669a47f1d14ce98ccab24479d39304 P b40a9a7b30d34a6db007ed8febd8a759 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:01.271571 3545 consensus_queue.cc:1035] T 0b669a47f1d14ce98ccab24479d39304 P b40a9a7b30d34a6db007ed8febd8a759 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:01.276924 2531 raft_consensus.cc:1261] T 0b669a47f1d14ce98ccab24479d39304 P a1a844ecd71145e4a362f4c42500fbcb [term 1 FOLLOWER]: Refusing update from remote peer b40a9a7b30d34a6db007ed8febd8a759: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:01.277305 3800 consensus_queue.cc:1035] T 0b669a47f1d14ce98ccab24479d39304 P b40a9a7b30d34a6db007ed8febd8a759 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 156.469 ms
time per row: 0.0782346 ms
I20240503 01:59:01.354342 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task42rE7p/build/debug/bin/kudu
/tmp/dist-test-task42rE7p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.135:39501
--local_ip_for_outbound_sockets=127.0.99.135
--tserver_master_addrs=127.0.99.190:38489
--webserver_port=32861
--webserver_interface=127.0.99.135
--builtin_ntp_servers=127.0.99.148:35375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 01:59:01.433808 2598 consensus_peers.cc:485] T 240749b0904f4bb2a4c79b7713473411 P b40a9a7b30d34a6db007ed8febd8a759 -> Peer 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135:39501): Couldn't send request to peer 4125b1a70e864e129bd54bdc2720d7fa. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:39501: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 01:59:01.460302 3875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:01.460515 3875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:59:01.460584 3875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:01.463397 3875 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:59:01.463457 3875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:59:01.463560 3875 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.135
I20240503 01:59:01.467100 3875 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.135:39501
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.0.99.135
--webserver_port=32861
--tserver_master_addrs=127.0.99.190:38489
--never_fsync=true
--heap_profile_path=/tmp/kudu.3875
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.135
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:59:01.467872 3875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:59:01.468946 3875 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:01.477774 3883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:01.478520 3880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:01.479712 3881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:01.480715 3875 server_base.cc:1034] running on GCE node
I20240503 01:59:01.481043 3875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:59:01.481490 3875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:59:01.482651 3875 hybrid_clock.cc:648] HybridClock initialized: now 1714701541482631 us; error 29 us; skew 500 ppm
I20240503 01:59:01.484547 3875 webserver.cc:460] Webserver started at http://127.0.99.135:32861/ using document root <none> and password file <none>
I20240503 01:59:01.485146 3875 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:01.485226 3875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:01.488838 3875 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20240503 01:59:01.490907 3889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:01.492002 3875 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:01.492094 3875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "4125b1a70e864e129bd54bdc2720d7fa"
format_stamp: "Formatted at 2024-05-03 01:58:53 on dist-test-slave-dtz3"
I20240503 01:59:01.492570 3875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:01.522056 3875 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:01.522619 3875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:59:01.522754 3875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:01.523222 3875 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:01.524318 3896 ts_tablet_manager.cc:516] Loading tablet metadata (0/1 complete)
I20240503 01:59:01.527212 3875 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240503 01:59:01.527277 3875 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.003s user 0.000s sys 0.000s
I20240503 01:59:01.527352 3875 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240503 01:59:01.528854 3875 ts_tablet_manager.cc:590] Registered 1 tablets
I20240503 01:59:01.528899 3875 ts_tablet_manager.cc:569] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:01.529193 3896 tablet_bootstrap.cc:492] T 240749b0904f4bb2a4c79b7713473411 P 4125b1a70e864e129bd54bdc2720d7fa: Bootstrap starting.
I20240503 01:59:01.544198 3875 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.135:39501
I20240503 01:59:01.544263 4003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.135:39501 every 8 connection(s)
I20240503 01:59:01.545210 3875 server_base.cc:1166] Dumped server information to /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 01:59:01.549556 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task42rE7p/build/debug/bin/kudu as pid 3875
I20240503 01:59:01.549716 398 test_util.cc:268] Using random seed: 273131155
I20240503 01:59:01.553262 3896 log.cc:824] T 240749b0904f4bb2a4c79b7713473411 P 4125b1a70e864e129bd54bdc2720d7fa: Log is configured to *not* fsync() on all Append() calls
I20240503 01:59:01.554910 4004 heartbeater.cc:347] Connected to a master server at 127.0.99.190:38489
I20240503 01:59:01.555127 4004 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:01.555784 4004 heartbeater.cc:511] Master 127.0.99.190:38489 requested a full tablet report, sending...
I20240503 01:59:01.555845 3896 tablet_bootstrap.cc:492] T 240749b0904f4bb2a4c79b7713473411 P 4125b1a70e864e129bd54bdc2720d7fa: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:01.556134 3896 tablet_bootstrap.cc:492] T 240749b0904f4bb2a4c79b7713473411 P 4125b1a70e864e129bd54bdc2720d7fa: Bootstrap complete.
I20240503 01:59:01.557385 3896 ts_tablet_manager.cc:1367] T 240749b0904f4bb2a4c79b7713473411 P 4125b1a70e864e129bd54bdc2720d7fa: Time spent bootstrapping tablet: real 0.028s user 0.004s sys 0.022s
I20240503 01:59:01.557637 2251 ts_manager.cc:194] Re-registered known tserver with Master: 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135:39501)
I20240503 01:59:01.558326 2251 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.135:49223
I20240503 01:59:01.560246 3896 raft_consensus.cc:348] T 240749b0904f4bb2a4c79b7713473411 P 4125b1a70e864e129bd54bdc2720d7fa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } }
I20240503 01:59:01.560655 3896 raft_consensus.cc:729] T 240749b0904f4bb2a4c79b7713473411 P 4125b1a70e864e129bd54bdc2720d7fa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4125b1a70e864e129bd54bdc2720d7fa, State: Initialized, Role: FOLLOWER
I20240503 01:59:01.561072 3896 consensus_queue.cc:260] T 240749b0904f4bb2a4c79b7713473411 P 4125b1a70e864e129bd54bdc2720d7fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } }
I20240503 01:59:01.561681 4004 heartbeater.cc:503] Master 127.0.99.190:38489 was elected leader, sending a full tablet report...
I20240503 01:59:01.561719 398 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_0 already exists
I20240503 01:59:01.561821 3896 ts_tablet_manager.cc:1398] T 240749b0904f4bb2a4c79b7713473411 P 4125b1a70e864e129bd54bdc2720d7fa: Time spent starting tablet: real 0.004s user 0.004s sys 0.001s
I20240503 01:59:01.561914 398 test_util.cc:268] Using random seed: 273143352
I20240503 01:59:01.572329 398 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_1 already exists
I20240503 01:59:01.572484 398 test_util.cc:268] Using random seed: 273153922
I20240503 01:59:01.587148 398 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_2 already exists
I20240503 01:59:01.587319 398 test_util.cc:268] Using random seed: 273168757
I20240503 01:59:01.603408 398 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_3 already exists
I20240503 01:59:01.603611 398 test_util.cc:268] Using random seed: 273185048
I20240503 01:59:01.620965 398 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_4 already exists
W20240503 01:59:01.796665 4005 tablet.cc:2369] T 240749b0904f4bb2a4c79b7713473411 P 4125b1a70e864e129bd54bdc2720d7fa: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 01:59:02.104552 2860 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.132:42887, user_credentials={real_user=slave}} blocked reactor thread for 46329.5us
I20240503 01:59:02.477169 3938 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:02.503562 2510 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:02.560668 3035 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 01:59:02.563768 2378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:02.569856 2772 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:02.573962 2904 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 01:59:02.598959 2860 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.130:46033, user_credentials={real_user=slave}} blocked reactor thread for 61280.5us
I20240503 01:59:02.630832 2641 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:03.162513 3938 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:03.182934 3035 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 01:59:03.183233 2772 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:03.238911 2510 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:03.278035 2904 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 01:59:03.280717 2641 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 01:59:03.293920 2336 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.130:46033, user_credentials={real_user=slave}} blocked reactor thread for 88034.3us
I20240503 01:59:03.476148 2378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:03.851289 2923 consensus_queue.cc:237] T 9d84cbdf02c14873b8838be296977aaf P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 1.34, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: true } }
I20240503 01:59:03.853503 3052 raft_consensus.cc:1261] T 9d84cbdf02c14873b8838be296977aaf P 6568f69a78c44bb985162d2b7d0a9376 [term 1 FOLLOWER]: Refusing update from remote peer 327aa26a8e434a609ebcb382aa665fce: Log matching property violated. Preceding OpId in replica: term: 1 index: 34. Preceding OpId from leader: term: 1 index: 35. (index mismatch)
I20240503 01:59:03.853929 4039 consensus_queue.cc:1035] T 9d84cbdf02c14873b8838be296977aaf P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20240503 01:59:03.859614 2659 raft_consensus.cc:1261] T 9d84cbdf02c14873b8838be296977aaf P b40a9a7b30d34a6db007ed8febd8a759 [term 1 FOLLOWER]: Refusing update from remote peer 327aa26a8e434a609ebcb382aa665fce: Log matching property violated. Preceding OpId in replica: term: 1 index: 34. Preceding OpId from leader: term: 1 index: 35. (index mismatch)
I20240503 01:59:03.860203 2790 raft_consensus.cc:1261] T 9d84cbdf02c14873b8838be296977aaf P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Refusing update from remote peer 327aa26a8e434a609ebcb382aa665fce: Log matching property violated. Preceding OpId in replica: term: 1 index: 34. Preceding OpId from leader: term: 1 index: 35. (index mismatch)
I20240503 01:59:03.861680 2528 raft_consensus.cc:1261] T 9d84cbdf02c14873b8838be296977aaf P a1a844ecd71145e4a362f4c42500fbcb [term 1 FOLLOWER]: Refusing update from remote peer 327aa26a8e434a609ebcb382aa665fce: Log matching property violated. Preceding OpId in replica: term: 1 index: 32. Preceding OpId from leader: term: 1 index: 35. (index mismatch)
I20240503 01:59:03.862583 4197 consensus_queue.cc:1035] T 9d84cbdf02c14873b8838be296977aaf P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20240503 01:59:03.869009 4197 consensus_queue.cc:1035] T 9d84cbdf02c14873b8838be296977aaf P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20240503 01:59:03.895125 4039 consensus_queue.cc:1035] T 9d84cbdf02c14873b8838be296977aaf P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 32, Time since last communication: 0.000s
I20240503 01:59:03.907969 3870 raft_consensus.cc:2932] T 9d84cbdf02c14873b8838be296977aaf P 327aa26a8e434a609ebcb382aa665fce [term 1 LEADER]: Committing config change with OpId 1.35: config changed from index -1 to 35, NON_VOTER 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) added. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: true } } }
I20240503 01:59:03.911015 3052 raft_consensus.cc:2932] T 9d84cbdf02c14873b8838be296977aaf P 6568f69a78c44bb985162d2b7d0a9376 [term 1 FOLLOWER]: Committing config change with OpId 1.35: config changed from index -1 to 35, NON_VOTER 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) added. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: true } } }
I20240503 01:59:03.912667 2790 raft_consensus.cc:2932] T 9d84cbdf02c14873b8838be296977aaf P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Committing config change with OpId 1.35: config changed from index -1 to 35, NON_VOTER 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) added. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: true } } }
I20240503 01:59:03.916052 2528 raft_consensus.cc:2932] T 9d84cbdf02c14873b8838be296977aaf P a1a844ecd71145e4a362f4c42500fbcb [term 1 FOLLOWER]: Committing config change with OpId 1.35: config changed from index -1 to 35, NON_VOTER 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) added. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: true } } }
I20240503 01:59:03.916265 2250 catalog_manager.cc:5484] T 9d84cbdf02c14873b8838be296977aaf P 327aa26a8e434a609ebcb382aa665fce reported cstate change: config changed from index -1 to 35, NON_VOTER 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "327aa26a8e434a609ebcb382aa665fce" committed_config { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:03.950196 2659 raft_consensus.cc:2932] T 9d84cbdf02c14873b8838be296977aaf P b40a9a7b30d34a6db007ed8febd8a759 [term 1 FOLLOWER]: Committing config change with OpId 1.35: config changed from index -1 to 35, NON_VOTER 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) added. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: true } } }
W20240503 01:59:03.967499 2860 consensus_peers.cc:485] T 9d84cbdf02c14873b8838be296977aaf P 327aa26a8e434a609ebcb382aa665fce -> Peer 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135:39501): Couldn't send request to peer 4125b1a70e864e129bd54bdc2720d7fa. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9d84cbdf02c14873b8838be296977aaf. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:04.103830 2656 consensus_queue.cc:237] T 240749b0904f4bb2a4c79b7713473411 P b40a9a7b30d34a6db007ed8febd8a759 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { replace: true } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { promote: true } }
W20240503 01:59:04.106014 2596 consensus_peers.cc:485] T 240749b0904f4bb2a4c79b7713473411 P b40a9a7b30d34a6db007ed8febd8a759 -> Peer 327aa26a8e434a609ebcb382aa665fce (127.0.99.133:38853): Couldn't send request to peer 327aa26a8e434a609ebcb382aa665fce. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 240749b0904f4bb2a4c79b7713473411. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:04.107023 3958 raft_consensus.cc:1261] T 240749b0904f4bb2a4c79b7713473411 P 4125b1a70e864e129bd54bdc2720d7fa [term 1 FOLLOWER]: Refusing update from remote peer b40a9a7b30d34a6db007ed8febd8a759: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:04.107335 3052 raft_consensus.cc:1261] T 240749b0904f4bb2a4c79b7713473411 P 6568f69a78c44bb985162d2b7d0a9376 [term 1 FOLLOWER]: Refusing update from remote peer b40a9a7b30d34a6db007ed8febd8a759: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:04.107743 4250 consensus_queue.cc:1035] T 240749b0904f4bb2a4c79b7713473411 P b40a9a7b30d34a6db007ed8febd8a759 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 01:59:04.108098 4250 consensus_queue.cc:1035] T 240749b0904f4bb2a4c79b7713473411 P b40a9a7b30d34a6db007ed8febd8a759 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 01:59:04.108570 2395 raft_consensus.cc:1261] T 240749b0904f4bb2a4c79b7713473411 P b63928177fa54227a5ff9929265ab9dd [term 1 FOLLOWER]: Refusing update from remote peer b40a9a7b30d34a6db007ed8febd8a759: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:04.109035 4250 consensus_queue.cc:1035] T 240749b0904f4bb2a4c79b7713473411 P b40a9a7b30d34a6db007ed8febd8a759 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 01:59:04.113974 2791 raft_consensus.cc:1261] T 240749b0904f4bb2a4c79b7713473411 P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Refusing update from remote peer b40a9a7b30d34a6db007ed8febd8a759: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:04.114713 3869 consensus_queue.cc:1035] T 240749b0904f4bb2a4c79b7713473411 P b40a9a7b30d34a6db007ed8febd8a759 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 01:59:04.115554 3869 raft_consensus.cc:2932] T 240749b0904f4bb2a4c79b7713473411 P b40a9a7b30d34a6db007ed8febd8a759 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 327aa26a8e434a609ebcb382aa665fce (127.0.99.133) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { replace: true } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { promote: true } } }
I20240503 01:59:04.117656 2790 raft_consensus.cc:2932] T 240749b0904f4bb2a4c79b7713473411 P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 327aa26a8e434a609ebcb382aa665fce (127.0.99.133) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { replace: true } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { promote: true } } }
I20240503 01:59:04.118096 3055 raft_consensus.cc:2932] T 240749b0904f4bb2a4c79b7713473411 P 6568f69a78c44bb985162d2b7d0a9376 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 327aa26a8e434a609ebcb382aa665fce (127.0.99.133) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { replace: true } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { promote: true } } }
I20240503 01:59:04.118019 3958 raft_consensus.cc:2932] T 240749b0904f4bb2a4c79b7713473411 P 4125b1a70e864e129bd54bdc2720d7fa [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 327aa26a8e434a609ebcb382aa665fce (127.0.99.133) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { replace: true } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { promote: true } } }
I20240503 01:59:04.121677 2253 catalog_manager.cc:5484] T 240749b0904f4bb2a4c79b7713473411 P 6568f69a78c44bb985162d2b7d0a9376 reported cstate change: config changed from index -1 to 2, NON_VOTER 327aa26a8e434a609ebcb382aa665fce (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "b40a9a7b30d34a6db007ed8febd8a759" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { replace: true } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { promote: true } } }
I20240503 01:59:04.126139 2400 raft_consensus.cc:2932] T 240749b0904f4bb2a4c79b7713473411 P b63928177fa54227a5ff9929265ab9dd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 327aa26a8e434a609ebcb382aa665fce (127.0.99.133) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { replace: true } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { promote: true } } }
I20240503 01:59:04.168210 2923 consensus_queue.cc:237] T 5fd1010447524b6695abb98192dc54f1 P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 1.24, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46033 } attrs { promote: true } }
I20240503 01:59:04.172184 3054 raft_consensus.cc:1261] T 5fd1010447524b6695abb98192dc54f1 P 6568f69a78c44bb985162d2b7d0a9376 [term 1 FOLLOWER]: Refusing update from remote peer 327aa26a8e434a609ebcb382aa665fce: Log matching property violated. Preceding OpId in replica: term: 1 index: 24. Preceding OpId from leader: term: 1 index: 25. (index mismatch)
I20240503 01:59:04.173301 4214 consensus_queue.cc:1035] T 5fd1010447524b6695abb98192dc54f1 P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 23, Time since last communication: 0.000s
I20240503 01:59:04.192382 2661 raft_consensus.cc:1261] T 5fd1010447524b6695abb98192dc54f1 P b40a9a7b30d34a6db007ed8febd8a759 [term 1 FOLLOWER]: Refusing update from remote peer 327aa26a8e434a609ebcb382aa665fce: Log matching property violated. Preceding OpId in replica: term: 1 index: 24. Preceding OpId from leader: term: 1 index: 25. (index mismatch)
I20240503 01:59:04.193146 4098 consensus_queue.cc:1035] T 5fd1010447524b6695abb98192dc54f1 P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 23, Time since last communication: 0.000s
I20240503 01:59:04.203362 2791 raft_consensus.cc:1261] T 5fd1010447524b6695abb98192dc54f1 P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Refusing update from remote peer 327aa26a8e434a609ebcb382aa665fce: Log matching property violated. Preceding OpId in replica: term: 1 index: 23. Preceding OpId from leader: term: 1 index: 25. (index mismatch)
W20240503 01:59:04.205513 2860 consensus_peers.cc:485] T 5fd1010447524b6695abb98192dc54f1 P 327aa26a8e434a609ebcb382aa665fce -> Peer a1a844ecd71145e4a362f4c42500fbcb (127.0.99.130:46033): Couldn't send request to peer a1a844ecd71145e4a362f4c42500fbcb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5fd1010447524b6695abb98192dc54f1. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:04.203271 2397 raft_consensus.cc:1261] T 5fd1010447524b6695abb98192dc54f1 P b63928177fa54227a5ff9929265ab9dd [term 1 FOLLOWER]: Refusing update from remote peer 327aa26a8e434a609ebcb382aa665fce: Log matching property violated. Preceding OpId in replica: term: 1 index: 23. Preceding OpId from leader: term: 1 index: 25. (index mismatch)
I20240503 01:59:04.207702 4039 consensus_queue.cc:1035] T 5fd1010447524b6695abb98192dc54f1 P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 23, Time since last communication: 0.000s
I20240503 01:59:04.208365 4039 consensus_queue.cc:1035] T 5fd1010447524b6695abb98192dc54f1 P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 23, Time since last communication: 0.000s
I20240503 01:59:04.211869 4098 raft_consensus.cc:2932] T 5fd1010447524b6695abb98192dc54f1 P 327aa26a8e434a609ebcb382aa665fce [term 1 LEADER]: Committing config change with OpId 1.25: config changed from index -1 to 25, NON_VOTER a1a844ecd71145e4a362f4c42500fbcb (127.0.99.130) added. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46033 } attrs { promote: true } } }
I20240503 01:59:04.215457 3055 raft_consensus.cc:2932] T 5fd1010447524b6695abb98192dc54f1 P 6568f69a78c44bb985162d2b7d0a9376 [term 1 FOLLOWER]: Committing config change with OpId 1.25: config changed from index -1 to 25, NON_VOTER a1a844ecd71145e4a362f4c42500fbcb (127.0.99.130) added. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46033 } attrs { promote: true } } }
I20240503 01:59:04.218320 2656 raft_consensus.cc:2932] T 5fd1010447524b6695abb98192dc54f1 P b40a9a7b30d34a6db007ed8febd8a759 [term 1 FOLLOWER]: Committing config change with OpId 1.25: config changed from index -1 to 25, NON_VOTER a1a844ecd71145e4a362f4c42500fbcb (127.0.99.130) added. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46033 } attrs { promote: true } } }
I20240503 01:59:04.221951 2251 catalog_manager.cc:5484] T 5fd1010447524b6695abb98192dc54f1 P b40a9a7b30d34a6db007ed8febd8a759 reported cstate change: config changed from index -1 to 25, NON_VOTER a1a844ecd71145e4a362f4c42500fbcb (127.0.99.130) added. New cstate: current_term: 1 leader_uuid: "327aa26a8e434a609ebcb382aa665fce" committed_config { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46033 } attrs { promote: true } } }
I20240503 01:59:04.227850 2394 raft_consensus.cc:2932] T 5fd1010447524b6695abb98192dc54f1 P b63928177fa54227a5ff9929265ab9dd [term 1 FOLLOWER]: Committing config change with OpId 1.25: config changed from index -1 to 25, NON_VOTER a1a844ecd71145e4a362f4c42500fbcb (127.0.99.130) added. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46033 } attrs { promote: true } } }
I20240503 01:59:04.240552 2790 raft_consensus.cc:2932] T 5fd1010447524b6695abb98192dc54f1 P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Committing config change with OpId 1.25: config changed from index -1 to 25, NON_VOTER a1a844ecd71145e4a362f4c42500fbcb (127.0.99.130) added. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46033 } attrs { promote: true } } }
I20240503 01:59:04.358758 4285 ts_tablet_manager.cc:902] T 9d84cbdf02c14873b8838be296977aaf P 4125b1a70e864e129bd54bdc2720d7fa: Initiating tablet copy from peer 327aa26a8e434a609ebcb382aa665fce (127.0.99.133:38853)
I20240503 01:59:04.362891 4285 tablet_copy_client.cc:308] T 9d84cbdf02c14873b8838be296977aaf P 4125b1a70e864e129bd54bdc2720d7fa: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.133:38853
I20240503 01:59:04.377501 2944 tablet_copy_service.cc:140] P 327aa26a8e434a609ebcb382aa665fce: Received BeginTabletCopySession request for tablet 9d84cbdf02c14873b8838be296977aaf from peer 4125b1a70e864e129bd54bdc2720d7fa ({username='slave'} at 127.0.99.135:39289)
I20240503 01:59:04.377652 2944 tablet_copy_service.cc:161] P 327aa26a8e434a609ebcb382aa665fce: Beginning new tablet copy session on tablet 9d84cbdf02c14873b8838be296977aaf from peer 4125b1a70e864e129bd54bdc2720d7fa at {username='slave'} at 127.0.99.135:39289: session id = 4125b1a70e864e129bd54bdc2720d7fa-9d84cbdf02c14873b8838be296977aaf
I20240503 01:59:04.379348 2944 tablet_copy_source_session.cc:205] T 9d84cbdf02c14873b8838be296977aaf P 327aa26a8e434a609ebcb382aa665fce: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:04.383162 4285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d84cbdf02c14873b8838be296977aaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:04.389807 4285 tablet_copy_client.cc:783] T 9d84cbdf02c14873b8838be296977aaf P 4125b1a70e864e129bd54bdc2720d7fa: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:04.390141 4285 tablet_copy_client.cc:647] T 9d84cbdf02c14873b8838be296977aaf P 4125b1a70e864e129bd54bdc2720d7fa: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:04.392297 4285 tablet_copy_client.cc:523] T 9d84cbdf02c14873b8838be296977aaf P 4125b1a70e864e129bd54bdc2720d7fa: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:04.393759 4285 tablet_bootstrap.cc:492] T 9d84cbdf02c14873b8838be296977aaf P 4125b1a70e864e129bd54bdc2720d7fa: Bootstrap starting.
I20240503 01:59:04.463474 2395 consensus_queue.cc:237] T 19b3a90e13194997bde2b9a806f8f2b5 P b63928177fa54227a5ff9929265ab9dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 64, Committed index: 64, Last appended: 1.64, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 65 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } attrs { replace: true } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: true } }
I20240503 01:59:04.465090 3052 raft_consensus.cc:1261] T 19b3a90e13194997bde2b9a806f8f2b5 P 6568f69a78c44bb985162d2b7d0a9376 [term 1 FOLLOWER]: Refusing update from remote peer b63928177fa54227a5ff9929265ab9dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 64. Preceding OpId from leader: term: 1 index: 65. (index mismatch)
I20240503 01:59:04.465546 3802 consensus_queue.cc:1035] T 19b3a90e13194997bde2b9a806f8f2b5 P b63928177fa54227a5ff9929265ab9dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 65, Last known committed idx: 64, Time since last communication: 0.000s
I20240503 01:59:04.465939 2790 raft_consensus.cc:1261] T 19b3a90e13194997bde2b9a806f8f2b5 P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Refusing update from remote peer b63928177fa54227a5ff9929265ab9dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 64. Preceding OpId from leader: term: 1 index: 65. (index mismatch)
I20240503 01:59:04.466682 4132 consensus_queue.cc:1035] T 19b3a90e13194997bde2b9a806f8f2b5 P b63928177fa54227a5ff9929265ab9dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 65, Last known committed idx: 64, Time since last communication: 0.000s
I20240503 01:59:04.470852 2529 raft_consensus.cc:1261] T 19b3a90e13194997bde2b9a806f8f2b5 P a1a844ecd71145e4a362f4c42500fbcb [term 1 FOLLOWER]: Refusing update from remote peer b63928177fa54227a5ff9929265ab9dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 64. Preceding OpId from leader: term: 1 index: 65. (index mismatch)
I20240503 01:59:04.471657 3802 raft_consensus.cc:2932] T 19b3a90e13194997bde2b9a806f8f2b5 P b63928177fa54227a5ff9929265ab9dd [term 1 LEADER]: Committing config change with OpId 1.65: config changed from index -1 to 65, NON_VOTER 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) added. New config: { opid_index: 65 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } attrs { replace: true } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: true } } }
I20240503 01:59:04.472988 2661 raft_consensus.cc:1261] T 19b3a90e13194997bde2b9a806f8f2b5 P b40a9a7b30d34a6db007ed8febd8a759 [term 1 FOLLOWER]: Refusing update from remote peer b63928177fa54227a5ff9929265ab9dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 62. Preceding OpId from leader: term: 1 index: 65. (index mismatch)
I20240503 01:59:04.473968 2791 raft_consensus.cc:2932] T 19b3a90e13194997bde2b9a806f8f2b5 P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Committing config change with OpId 1.65: config changed from index -1 to 65, NON_VOTER 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) added. New config: { opid_index: 65 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } attrs { replace: true } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: true } } }
I20240503 01:59:04.474280 3842 consensus_queue.cc:1035] T 19b3a90e13194997bde2b9a806f8f2b5 P b63928177fa54227a5ff9929265ab9dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 65, Last known committed idx: 63, Time since last communication: 0.000s
I20240503 01:59:04.474267 3052 raft_consensus.cc:2932] T 19b3a90e13194997bde2b9a806f8f2b5 P 6568f69a78c44bb985162d2b7d0a9376 [term 1 FOLLOWER]: Committing config change with OpId 1.65: config changed from index -1 to 65, NON_VOTER 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) added. New config: { opid_index: 65 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } attrs { replace: true } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: true } } }
I20240503 01:59:04.476495 4233 consensus_queue.cc:1035] T 19b3a90e13194997bde2b9a806f8f2b5 P b63928177fa54227a5ff9929265ab9dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 65, Last known committed idx: 61, Time since last communication: 0.000s
I20240503 01:59:04.476722 2253 catalog_manager.cc:5484] T 19b3a90e13194997bde2b9a806f8f2b5 P 52c196c15e3348b985c689652bc49918 reported cstate change: config changed from index -1 to 65, NON_VOTER 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "b63928177fa54227a5ff9929265ab9dd" committed_config { opid_index: 65 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } attrs { replace: true } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: true } } }
I20240503 01:59:04.484414 2524 raft_consensus.cc:2932] T 19b3a90e13194997bde2b9a806f8f2b5 P a1a844ecd71145e4a362f4c42500fbcb [term 1 FOLLOWER]: Committing config change with OpId 1.65: config changed from index -1 to 65, NON_VOTER 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) added. New config: { opid_index: 65 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } attrs { replace: true } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: true } } }
I20240503 01:59:04.506326 4285 tablet_bootstrap.cc:492] T 9d84cbdf02c14873b8838be296977aaf P 4125b1a70e864e129bd54bdc2720d7fa: Bootstrap replayed 1/1 log segments. Stats: ops{read=44 overwritten=0 applied=44 ignored=0} inserts{seen=675 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:04.506759 4285 tablet_bootstrap.cc:492] T 9d84cbdf02c14873b8838be296977aaf P 4125b1a70e864e129bd54bdc2720d7fa: Bootstrap complete.
I20240503 01:59:04.506996 4285 ts_tablet_manager.cc:1367] T 9d84cbdf02c14873b8838be296977aaf P 4125b1a70e864e129bd54bdc2720d7fa: Time spent bootstrapping tablet: real 0.113s user 0.058s sys 0.004s
I20240503 01:59:04.507562 4285 raft_consensus.cc:348] T 9d84cbdf02c14873b8838be296977aaf P 4125b1a70e864e129bd54bdc2720d7fa [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: true } }
I20240503 01:59:04.507799 4285 raft_consensus.cc:729] T 9d84cbdf02c14873b8838be296977aaf P 4125b1a70e864e129bd54bdc2720d7fa [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4125b1a70e864e129bd54bdc2720d7fa, State: Initialized, Role: LEARNER
I20240503 01:59:04.507987 4285 consensus_queue.cc:260] T 9d84cbdf02c14873b8838be296977aaf P 4125b1a70e864e129bd54bdc2720d7fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 44, Last appended: 1.44, Last appended by leader: 44, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: true } }
I20240503 01:59:04.510004 4285 ts_tablet_manager.cc:1398] T 9d84cbdf02c14873b8838be296977aaf P 4125b1a70e864e129bd54bdc2720d7fa: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 01:59:04.510624 2944 tablet_copy_service.cc:342] P 327aa26a8e434a609ebcb382aa665fce: Request end of tablet copy session 4125b1a70e864e129bd54bdc2720d7fa-9d84cbdf02c14873b8838be296977aaf received from {username='slave'} at 127.0.99.135:39289
I20240503 01:59:04.510727 2944 tablet_copy_service.cc:433] P 327aa26a8e434a609ebcb382aa665fce: ending tablet copy session 4125b1a70e864e129bd54bdc2720d7fa-9d84cbdf02c14873b8838be296977aaf on tablet 9d84cbdf02c14873b8838be296977aaf with peer 4125b1a70e864e129bd54bdc2720d7fa
W20240503 01:59:04.536276 2336 consensus_peers.cc:485] T 19b3a90e13194997bde2b9a806f8f2b5 P b63928177fa54227a5ff9929265ab9dd -> Peer 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135:39501): Couldn't send request to peer 4125b1a70e864e129bd54bdc2720d7fa. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 19b3a90e13194997bde2b9a806f8f2b5. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:04.563163 2659 raft_consensus.cc:2932] T 19b3a90e13194997bde2b9a806f8f2b5 P b40a9a7b30d34a6db007ed8febd8a759 [term 1 FOLLOWER]: Committing config change with OpId 1.65: config changed from index -1 to 65, NON_VOTER 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) added. New config: { opid_index: 65 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } attrs { replace: true } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: true } } }
I20240503 01:59:04.588610 4298 ts_tablet_manager.cc:902] T 5fd1010447524b6695abb98192dc54f1 P a1a844ecd71145e4a362f4c42500fbcb: Initiating tablet copy from peer 327aa26a8e434a609ebcb382aa665fce (127.0.99.133:38853)
I20240503 01:59:04.593045 4298 tablet_copy_client.cc:308] T 5fd1010447524b6695abb98192dc54f1 P a1a844ecd71145e4a362f4c42500fbcb: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.133:38853
I20240503 01:59:04.594838 2944 tablet_copy_service.cc:140] P 327aa26a8e434a609ebcb382aa665fce: Received BeginTabletCopySession request for tablet 5fd1010447524b6695abb98192dc54f1 from peer a1a844ecd71145e4a362f4c42500fbcb ({username='slave'} at 127.0.99.130:35863)
I20240503 01:59:04.594944 2944 tablet_copy_service.cc:161] P 327aa26a8e434a609ebcb382aa665fce: Beginning new tablet copy session on tablet 5fd1010447524b6695abb98192dc54f1 from peer a1a844ecd71145e4a362f4c42500fbcb at {username='slave'} at 127.0.99.130:35863: session id = a1a844ecd71145e4a362f4c42500fbcb-5fd1010447524b6695abb98192dc54f1
I20240503 01:59:04.596575 2944 tablet_copy_source_session.cc:205] T 5fd1010447524b6695abb98192dc54f1 P 327aa26a8e434a609ebcb382aa665fce: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:04.598991 4298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fd1010447524b6695abb98192dc54f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:04.601861 4298 tablet_copy_client.cc:783] T 5fd1010447524b6695abb98192dc54f1 P a1a844ecd71145e4a362f4c42500fbcb: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:04.602272 4298 tablet_copy_client.cc:647] T 5fd1010447524b6695abb98192dc54f1 P a1a844ecd71145e4a362f4c42500fbcb: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:04.607172 4298 tablet_copy_client.cc:523] T 5fd1010447524b6695abb98192dc54f1 P a1a844ecd71145e4a362f4c42500fbcb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:04.609297 4298 tablet_bootstrap.cc:492] T 5fd1010447524b6695abb98192dc54f1 P a1a844ecd71145e4a362f4c42500fbcb: Bootstrap starting.
I20240503 01:59:04.611515 2923 consensus_queue.cc:237] T a6e17a2c4ba0482e96d04eb9d3ba854a P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 54, Committed index: 54, Last appended: 1.55, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 56 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } attrs { replace: true } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { promote: true } }
I20240503 01:59:04.614764 3052 raft_consensus.cc:1261] T a6e17a2c4ba0482e96d04eb9d3ba854a P 6568f69a78c44bb985162d2b7d0a9376 [term 1 FOLLOWER]: Refusing update from remote peer 327aa26a8e434a609ebcb382aa665fce: Log matching property violated. Preceding OpId in replica: term: 1 index: 55. Preceding OpId from leader: term: 1 index: 56. (index mismatch)
I20240503 01:59:04.615394 3551 consensus_queue.cc:1035] T a6e17a2c4ba0482e96d04eb9d3ba854a P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 56, Last known committed idx: 54, Time since last communication: 0.000s
I20240503 01:59:04.616288 2529 raft_consensus.cc:1261] T a6e17a2c4ba0482e96d04eb9d3ba854a P a1a844ecd71145e4a362f4c42500fbcb [term 1 FOLLOWER]: Refusing update from remote peer 327aa26a8e434a609ebcb382aa665fce: Log matching property violated. Preceding OpId in replica: term: 1 index: 55. Preceding OpId from leader: term: 1 index: 56. (index mismatch)
I20240503 01:59:04.617369 2789 raft_consensus.cc:1261] T a6e17a2c4ba0482e96d04eb9d3ba854a P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Refusing update from remote peer 327aa26a8e434a609ebcb382aa665fce: Log matching property violated. Preceding OpId in replica: term: 1 index: 54. Preceding OpId from leader: term: 1 index: 56. (index mismatch)
I20240503 01:59:04.617332 3551 consensus_queue.cc:1035] T a6e17a2c4ba0482e96d04eb9d3ba854a P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 56, Last known committed idx: 54, Time since last communication: 0.000s
I20240503 01:59:04.620332 4039 consensus_queue.cc:1035] T a6e17a2c4ba0482e96d04eb9d3ba854a P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 56, Last known committed idx: 54, Time since last communication: 0.000s
I20240503 01:59:04.638273 4039 raft_consensus.cc:2932] T a6e17a2c4ba0482e96d04eb9d3ba854a P 327aa26a8e434a609ebcb382aa665fce [term 1 LEADER]: Committing config change with OpId 1.56: config changed from index -1 to 56, NON_VOTER b63928177fa54227a5ff9929265ab9dd (127.0.99.129) added. New config: { opid_index: 56 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } attrs { replace: true } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { promote: true } } }
I20240503 01:59:04.639729 2792 raft_consensus.cc:2932] T a6e17a2c4ba0482e96d04eb9d3ba854a P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Committing config change with OpId 1.56: config changed from index -1 to 56, NON_VOTER b63928177fa54227a5ff9929265ab9dd (127.0.99.129) added. New config: { opid_index: 56 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } attrs { replace: true } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { promote: true } } }
I20240503 01:59:04.641043 2528 raft_consensus.cc:2932] T a6e17a2c4ba0482e96d04eb9d3ba854a P a1a844ecd71145e4a362f4c42500fbcb [term 1 FOLLOWER]: Committing config change with OpId 1.56: config changed from index -1 to 56, NON_VOTER b63928177fa54227a5ff9929265ab9dd (127.0.99.129) added. New config: { opid_index: 56 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } attrs { replace: true } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { promote: true } } }
I20240503 01:59:04.642555 2251 catalog_manager.cc:5484] T a6e17a2c4ba0482e96d04eb9d3ba854a P 327aa26a8e434a609ebcb382aa665fce reported cstate change: config changed from index -1 to 56, NON_VOTER b63928177fa54227a5ff9929265ab9dd (127.0.99.129) added. New cstate: current_term: 1 leader_uuid: "327aa26a8e434a609ebcb382aa665fce" committed_config { opid_index: 56 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:04.648375 3054 raft_consensus.cc:2932] T a6e17a2c4ba0482e96d04eb9d3ba854a P 6568f69a78c44bb985162d2b7d0a9376 [term 1 FOLLOWER]: Committing config change with OpId 1.56: config changed from index -1 to 56, NON_VOTER b63928177fa54227a5ff9929265ab9dd (127.0.99.129) added. New config: { opid_index: 56 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } attrs { replace: true } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { promote: true } } }
I20240503 01:59:04.661101 2395 consensus_queue.cc:237] T c10343409a2a423eb0b21759bfc9ba61 P b63928177fa54227a5ff9929265ab9dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 45, Committed index: 45, Last appended: 1.45, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 46 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { replace: true } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: true } }
I20240503 01:59:04.663116 2921 raft_consensus.cc:1261] T c10343409a2a423eb0b21759bfc9ba61 P 327aa26a8e434a609ebcb382aa665fce [term 1 FOLLOWER]: Refusing update from remote peer b63928177fa54227a5ff9929265ab9dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 45. Preceding OpId from leader: term: 1 index: 46. (index mismatch)
I20240503 01:59:04.663518 4132 consensus_queue.cc:1035] T c10343409a2a423eb0b21759bfc9ba61 P b63928177fa54227a5ff9929265ab9dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 46, Last known committed idx: 45, Time since last communication: 0.000s
W20240503 01:59:04.665257 2336 consensus_peers.cc:485] T c10343409a2a423eb0b21759bfc9ba61 P b63928177fa54227a5ff9929265ab9dd -> Peer 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135:39501): Couldn't send request to peer 4125b1a70e864e129bd54bdc2720d7fa. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c10343409a2a423eb0b21759bfc9ba61. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:04.667821 2789 raft_consensus.cc:1261] T c10343409a2a423eb0b21759bfc9ba61 P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Refusing update from remote peer b63928177fa54227a5ff9929265ab9dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 45. Preceding OpId from leader: term: 1 index: 46. (index mismatch)
I20240503 01:59:04.669267 4132 consensus_queue.cc:1035] T c10343409a2a423eb0b21759bfc9ba61 P b63928177fa54227a5ff9929265ab9dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 46, Last known committed idx: 44, Time since last communication: 0.000s
I20240503 01:59:04.675576 2658 raft_consensus.cc:1261] T c10343409a2a423eb0b21759bfc9ba61 P b40a9a7b30d34a6db007ed8febd8a759 [term 1 FOLLOWER]: Refusing update from remote peer b63928177fa54227a5ff9929265ab9dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 45. Preceding OpId from leader: term: 1 index: 46. (index mismatch)
I20240503 01:59:04.677078 2530 raft_consensus.cc:1261] T c10343409a2a423eb0b21759bfc9ba61 P a1a844ecd71145e4a362f4c42500fbcb [term 1 FOLLOWER]: Refusing update from remote peer b63928177fa54227a5ff9929265ab9dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 45. Preceding OpId from leader: term: 1 index: 46. (index mismatch)
I20240503 01:59:04.677775 4132 raft_consensus.cc:2932] T c10343409a2a423eb0b21759bfc9ba61 P b63928177fa54227a5ff9929265ab9dd [term 1 LEADER]: Committing config change with OpId 1.46: config changed from index -1 to 46, NON_VOTER 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) added. New config: { opid_index: 46 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { replace: true } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: true } } }
I20240503 01:59:04.677855 2789 raft_consensus.cc:2932] T c10343409a2a423eb0b21759bfc9ba61 P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Committing config change with OpId 1.46: config changed from index -1 to 46, NON_VOTER 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) added. New config: { opid_index: 46 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { replace: true } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: true } } }
I20240503 01:59:04.679674 4132 consensus_queue.cc:1035] T c10343409a2a423eb0b21759bfc9ba61 P b63928177fa54227a5ff9929265ab9dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 46, Last known committed idx: 45, Time since last communication: 0.000s
I20240503 01:59:04.680289 2657 raft_consensus.cc:1261] T a6e17a2c4ba0482e96d04eb9d3ba854a P b40a9a7b30d34a6db007ed8febd8a759 [term 1 FOLLOWER]: Refusing update from remote peer 327aa26a8e434a609ebcb382aa665fce: Log matching property violated. Preceding OpId in replica: term: 1 index: 54. Preceding OpId from leader: term: 1 index: 56. (index mismatch)
I20240503 01:59:04.681908 2530 raft_consensus.cc:2932] T c10343409a2a423eb0b21759bfc9ba61 P a1a844ecd71145e4a362f4c42500fbcb [term 1 FOLLOWER]: Committing config change with OpId 1.46: config changed from index -1 to 46, NON_VOTER 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) added. New config: { opid_index: 46 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { replace: true } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: true } } }
I20240503 01:59:04.682045 2253 catalog_manager.cc:5484] T c10343409a2a423eb0b21759bfc9ba61 P b63928177fa54227a5ff9929265ab9dd reported cstate change: config changed from index -1 to 46, NON_VOTER 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "b63928177fa54227a5ff9929265ab9dd" committed_config { opid_index: 46 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:04.686420 2921 raft_consensus.cc:2932] T c10343409a2a423eb0b21759bfc9ba61 P 327aa26a8e434a609ebcb382aa665fce [term 1 FOLLOWER]: Committing config change with OpId 1.46: config changed from index -1 to 46, NON_VOTER 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) added. New config: { opid_index: 46 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { replace: true } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: true } } }
I20240503 01:59:04.693445 4214 consensus_queue.cc:1035] T a6e17a2c4ba0482e96d04eb9d3ba854a P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 56, Last known committed idx: 54, Time since last communication: 0.000s
I20240503 01:59:04.695820 4132 consensus_queue.cc:1035] T c10343409a2a423eb0b21759bfc9ba61 P b63928177fa54227a5ff9929265ab9dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 46, Last known committed idx: 44, Time since last communication: 0.000s
I20240503 01:59:04.699672 2660 raft_consensus.cc:2932] T c10343409a2a423eb0b21759bfc9ba61 P b40a9a7b30d34a6db007ed8febd8a759 [term 1 FOLLOWER]: Committing config change with OpId 1.46: config changed from index -1 to 46, NON_VOTER 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) added. New config: { opid_index: 46 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { replace: true } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: true } } }
I20240503 01:59:04.705160 2662 raft_consensus.cc:2932] T a6e17a2c4ba0482e96d04eb9d3ba854a P b40a9a7b30d34a6db007ed8febd8a759 [term 1 FOLLOWER]: Committing config change with OpId 1.56: config changed from index -1 to 56, NON_VOTER b63928177fa54227a5ff9929265ab9dd (127.0.99.129) added. New config: { opid_index: 56 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } attrs { replace: true } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { promote: true } } }
I20240503 01:59:04.710021 4311 ts_tablet_manager.cc:902] T 240749b0904f4bb2a4c79b7713473411 P 327aa26a8e434a609ebcb382aa665fce: Initiating tablet copy from peer b40a9a7b30d34a6db007ed8febd8a759 (127.0.99.131:34079)
I20240503 01:59:04.714809 4311 tablet_copy_client.cc:308] T 240749b0904f4bb2a4c79b7713473411 P 327aa26a8e434a609ebcb382aa665fce: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:34079
W20240503 01:59:04.710494 2860 consensus_peers.cc:485] T a6e17a2c4ba0482e96d04eb9d3ba854a P 327aa26a8e434a609ebcb382aa665fce -> Peer b63928177fa54227a5ff9929265ab9dd (127.0.99.129:36425): Couldn't send request to peer b63928177fa54227a5ff9929265ab9dd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a6e17a2c4ba0482e96d04eb9d3ba854a. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:04.725778 3958 raft_consensus.cc:1203] T 9d84cbdf02c14873b8838be296977aaf P 4125b1a70e864e129bd54bdc2720d7fa [term 1 LEARNER]: Deduplicated request from leader. Original: 1.34->[1.35-1.52] Dedup: 1.44->[1.45-1.52]
I20240503 01:59:04.734164 2682 tablet_copy_service.cc:140] P b40a9a7b30d34a6db007ed8febd8a759: Received BeginTabletCopySession request for tablet 240749b0904f4bb2a4c79b7713473411 from peer 327aa26a8e434a609ebcb382aa665fce ({username='slave'} at 127.0.99.133:47839)
I20240503 01:59:04.734321 2682 tablet_copy_service.cc:161] P b40a9a7b30d34a6db007ed8febd8a759: Beginning new tablet copy session on tablet 240749b0904f4bb2a4c79b7713473411 from peer 327aa26a8e434a609ebcb382aa665fce at {username='slave'} at 127.0.99.133:47839: session id = 327aa26a8e434a609ebcb382aa665fce-240749b0904f4bb2a4c79b7713473411
I20240503 01:59:04.735908 2682 tablet_copy_source_session.cc:205] T 240749b0904f4bb2a4c79b7713473411 P b40a9a7b30d34a6db007ed8febd8a759: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:04.742749 4311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 240749b0904f4bb2a4c79b7713473411. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:04.800158 2659 consensus_queue.cc:237] T 568a05972208414c90f7f4e2ea2c10a9 P b40a9a7b30d34a6db007ed8febd8a759 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 78, Committed index: 78, Last appended: 1.78, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: true } }
I20240503 01:59:04.800621 4311 tablet_copy_client.cc:783] T 240749b0904f4bb2a4c79b7713473411 P 327aa26a8e434a609ebcb382aa665fce: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:04.800915 4311 tablet_copy_client.cc:647] T 240749b0904f4bb2a4c79b7713473411 P 327aa26a8e434a609ebcb382aa665fce: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:04.802969 2529 raft_consensus.cc:1261] T 568a05972208414c90f7f4e2ea2c10a9 P a1a844ecd71145e4a362f4c42500fbcb [term 1 FOLLOWER]: Refusing update from remote peer b40a9a7b30d34a6db007ed8febd8a759: Log matching property violated. Preceding OpId in replica: term: 1 index: 78. Preceding OpId from leader: term: 1 index: 79. (index mismatch)
I20240503 01:59:04.803613 4253 consensus_queue.cc:1035] T 568a05972208414c90f7f4e2ea2c10a9 P b40a9a7b30d34a6db007ed8febd8a759 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 79, Last known committed idx: 78, Time since last communication: 0.000s
W20240503 01:59:04.804069 2598 consensus_peers.cc:485] T 568a05972208414c90f7f4e2ea2c10a9 P b40a9a7b30d34a6db007ed8febd8a759 -> Peer 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135:39501): Couldn't send request to peer 4125b1a70e864e129bd54bdc2720d7fa. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 568a05972208414c90f7f4e2ea2c10a9. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:04.804430 2793 raft_consensus.cc:1261] T 568a05972208414c90f7f4e2ea2c10a9 P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Refusing update from remote peer b40a9a7b30d34a6db007ed8febd8a759: Log matching property violated. Preceding OpId in replica: term: 1 index: 78. Preceding OpId from leader: term: 1 index: 79. (index mismatch)
I20240503 01:59:04.805016 2400 raft_consensus.cc:1261] T 568a05972208414c90f7f4e2ea2c10a9 P b63928177fa54227a5ff9929265ab9dd [term 1 FOLLOWER]: Refusing update from remote peer b40a9a7b30d34a6db007ed8febd8a759: Log matching property violated. Preceding OpId in replica: term: 1 index: 78. Preceding OpId from leader: term: 1 index: 79. (index mismatch)
I20240503 01:59:04.805215 4250 consensus_queue.cc:1035] T 568a05972208414c90f7f4e2ea2c10a9 P b40a9a7b30d34a6db007ed8febd8a759 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 79, Last known committed idx: 78, Time since last communication: 0.000s
I20240503 01:59:04.805788 2921 raft_consensus.cc:1261] T 568a05972208414c90f7f4e2ea2c10a9 P 327aa26a8e434a609ebcb382aa665fce [term 1 FOLLOWER]: Refusing update from remote peer b40a9a7b30d34a6db007ed8febd8a759: Log matching property violated. Preceding OpId in replica: term: 1 index: 78. Preceding OpId from leader: term: 1 index: 79. (index mismatch)
I20240503 01:59:04.806277 4253 consensus_queue.cc:1035] T 568a05972208414c90f7f4e2ea2c10a9 P b40a9a7b30d34a6db007ed8febd8a759 [LEADER]: Connected to new peer: Peer: permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 79, Last known committed idx: 78, Time since last communication: 0.000s
I20240503 01:59:04.807053 4250 raft_consensus.cc:2932] T 568a05972208414c90f7f4e2ea2c10a9 P b40a9a7b30d34a6db007ed8febd8a759 [term 1 LEADER]: Committing config change with OpId 1.79: config changed from index -1 to 79, NON_VOTER 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) added. New config: { opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: true } } }
I20240503 01:59:04.808437 2921 raft_consensus.cc:2932] T 568a05972208414c90f7f4e2ea2c10a9 P 327aa26a8e434a609ebcb382aa665fce [term 1 FOLLOWER]: Committing config change with OpId 1.79: config changed from index -1 to 79, NON_VOTER 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) added. New config: { opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: true } } }
I20240503 01:59:04.810784 2793 raft_consensus.cc:2932] T 568a05972208414c90f7f4e2ea2c10a9 P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Committing config change with OpId 1.79: config changed from index -1 to 79, NON_VOTER 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) added. New config: { opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: true } } }
I20240503 01:59:04.811462 2251 catalog_manager.cc:5484] T 568a05972208414c90f7f4e2ea2c10a9 P b40a9a7b30d34a6db007ed8febd8a759 reported cstate change: config changed from index -1 to 79, NON_VOTER 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "b40a9a7b30d34a6db007ed8febd8a759" committed_config { opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:04.818428 2531 raft_consensus.cc:2932] T 568a05972208414c90f7f4e2ea2c10a9 P a1a844ecd71145e4a362f4c42500fbcb [term 1 FOLLOWER]: Committing config change with OpId 1.79: config changed from index -1 to 79, NON_VOTER 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) added. New config: { opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: true } } }
I20240503 01:59:04.827517 4327 consensus_queue.cc:1035] T 568a05972208414c90f7f4e2ea2c10a9 P b40a9a7b30d34a6db007ed8febd8a759 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 79, Last known committed idx: 78, Time since last communication: 0.000s
I20240503 01:59:04.830904 2400 raft_consensus.cc:2932] T 568a05972208414c90f7f4e2ea2c10a9 P b63928177fa54227a5ff9929265ab9dd [term 1 FOLLOWER]: Committing config change with OpId 1.79: config changed from index -1 to 79, NON_VOTER 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) added. New config: { opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: true } } }
I20240503 01:59:04.906886 4285 ts_tablet_manager.cc:902] T 19b3a90e13194997bde2b9a806f8f2b5 P 4125b1a70e864e129bd54bdc2720d7fa: Initiating tablet copy from peer b63928177fa54227a5ff9929265ab9dd (127.0.99.129:36425)
I20240503 01:59:04.907578 4285 tablet_copy_client.cc:308] T 19b3a90e13194997bde2b9a806f8f2b5 P 4125b1a70e864e129bd54bdc2720d7fa: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:36425
I20240503 01:59:04.925535 4311 tablet_copy_client.cc:523] T 240749b0904f4bb2a4c79b7713473411 P 327aa26a8e434a609ebcb382aa665fce: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:04.931339 4311 tablet_bootstrap.cc:492] T 240749b0904f4bb2a4c79b7713473411 P 327aa26a8e434a609ebcb382aa665fce: Bootstrap starting.
I20240503 01:59:04.939777 4311 tablet_bootstrap.cc:492] T 240749b0904f4bb2a4c79b7713473411 P 327aa26a8e434a609ebcb382aa665fce: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:04.940099 4311 tablet_bootstrap.cc:492] T 240749b0904f4bb2a4c79b7713473411 P 327aa26a8e434a609ebcb382aa665fce: Bootstrap complete.
I20240503 01:59:04.940284 4311 ts_tablet_manager.cc:1367] T 240749b0904f4bb2a4c79b7713473411 P 327aa26a8e434a609ebcb382aa665fce: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20240503 01:59:04.940634 4311 raft_consensus.cc:348] T 240749b0904f4bb2a4c79b7713473411 P 327aa26a8e434a609ebcb382aa665fce [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { replace: true } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { promote: true } }
I20240503 01:59:04.940801 4311 raft_consensus.cc:729] T 240749b0904f4bb2a4c79b7713473411 P 327aa26a8e434a609ebcb382aa665fce [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 327aa26a8e434a609ebcb382aa665fce, State: Initialized, Role: LEARNER
I20240503 01:59:04.940898 4311 consensus_queue.cc:260] T 240749b0904f4bb2a4c79b7713473411 P 327aa26a8e434a609ebcb382aa665fce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { replace: true } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { promote: true } }
I20240503 01:59:04.941170 4311 ts_tablet_manager.cc:1398] T 240749b0904f4bb2a4c79b7713473411 P 327aa26a8e434a609ebcb382aa665fce: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:04.979715 2682 tablet_copy_service.cc:342] P b40a9a7b30d34a6db007ed8febd8a759: Request end of tablet copy session 327aa26a8e434a609ebcb382aa665fce-240749b0904f4bb2a4c79b7713473411 received from {username='slave'} at 127.0.99.133:47839
I20240503 01:59:04.979837 2682 tablet_copy_service.cc:433] P b40a9a7b30d34a6db007ed8febd8a759: ending tablet copy session 327aa26a8e434a609ebcb382aa665fce-240749b0904f4bb2a4c79b7713473411 on tablet 240749b0904f4bb2a4c79b7713473411 with peer 327aa26a8e434a609ebcb382aa665fce
I20240503 01:59:04.986146 4216 raft_consensus.cc:1050] T 9d84cbdf02c14873b8838be296977aaf P 327aa26a8e434a609ebcb382aa665fce: attempting to promote NON_VOTER 4125b1a70e864e129bd54bdc2720d7fa to VOTER
I20240503 01:59:04.986714 4216 consensus_queue.cc:237] T 9d84cbdf02c14873b8838be296977aaf P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 54, Committed index: 54, Last appended: 1.54, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 55 OBSOLETE_local: false peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } }
I20240503 01:59:04.994952 2661 raft_consensus.cc:1261] T 9d84cbdf02c14873b8838be296977aaf P b40a9a7b30d34a6db007ed8febd8a759 [term 1 FOLLOWER]: Refusing update from remote peer 327aa26a8e434a609ebcb382aa665fce: Log matching property violated. Preceding OpId in replica: term: 1 index: 54. Preceding OpId from leader: term: 1 index: 55. (index mismatch)
I20240503 01:59:04.995262 3052 raft_consensus.cc:1261] T 9d84cbdf02c14873b8838be296977aaf P 6568f69a78c44bb985162d2b7d0a9376 [term 1 FOLLOWER]: Refusing update from remote peer 327aa26a8e434a609ebcb382aa665fce: Log matching property violated. Preceding OpId in replica: term: 1 index: 54. Preceding OpId from leader: term: 1 index: 55. (index mismatch)
I20240503 01:59:04.996333 4197 consensus_queue.cc:1035] T 9d84cbdf02c14873b8838be296977aaf P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 55, Last known committed idx: 54, Time since last communication: 0.000s
I20240503 01:59:04.999507 3551 consensus_queue.cc:1035] T 9d84cbdf02c14873b8838be296977aaf P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 55, Last known committed idx: 54, Time since last communication: 0.000s
I20240503 01:59:05.013978 2420 tablet_copy_service.cc:140] P b63928177fa54227a5ff9929265ab9dd: Received BeginTabletCopySession request for tablet 19b3a90e13194997bde2b9a806f8f2b5 from peer 4125b1a70e864e129bd54bdc2720d7fa ({username='slave'} at 127.0.99.135:45173)
I20240503 01:59:05.014132 2420 tablet_copy_service.cc:161] P b63928177fa54227a5ff9929265ab9dd: Beginning new tablet copy session on tablet 19b3a90e13194997bde2b9a806f8f2b5 from peer 4125b1a70e864e129bd54bdc2720d7fa at {username='slave'} at 127.0.99.135:45173: session id = 4125b1a70e864e129bd54bdc2720d7fa-19b3a90e13194997bde2b9a806f8f2b5
I20240503 01:59:05.015638 2420 tablet_copy_source_session.cc:205] T 19b3a90e13194997bde2b9a806f8f2b5 P b63928177fa54227a5ff9929265ab9dd: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:05.016981 4285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19b3a90e13194997bde2b9a806f8f2b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:05.019544 4285 tablet_copy_client.cc:783] T 19b3a90e13194997bde2b9a806f8f2b5 P 4125b1a70e864e129bd54bdc2720d7fa: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:05.019726 4285 tablet_copy_client.cc:647] T 19b3a90e13194997bde2b9a806f8f2b5 P 4125b1a70e864e129bd54bdc2720d7fa: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:05.022730 4285 tablet_copy_client.cc:523] T 19b3a90e13194997bde2b9a806f8f2b5 P 4125b1a70e864e129bd54bdc2720d7fa: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:05.024545 4285 tablet_bootstrap.cc:492] T 19b3a90e13194997bde2b9a806f8f2b5 P 4125b1a70e864e129bd54bdc2720d7fa: Bootstrap starting.
I20240503 01:59:05.034263 2921 raft_consensus.cc:1203] T 240749b0904f4bb2a4c79b7713473411 P 327aa26a8e434a609ebcb382aa665fce [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 01:59:05.037416 2792 raft_consensus.cc:1261] T 9d84cbdf02c14873b8838be296977aaf P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Refusing update from remote peer 327aa26a8e434a609ebcb382aa665fce: Log matching property violated. Preceding OpId in replica: term: 1 index: 54. Preceding OpId from leader: term: 1 index: 55. (index mismatch)
I20240503 01:59:05.038241 3958 raft_consensus.cc:1261] T 9d84cbdf02c14873b8838be296977aaf P 4125b1a70e864e129bd54bdc2720d7fa [term 1 LEARNER]: Refusing update from remote peer 327aa26a8e434a609ebcb382aa665fce: Log matching property violated. Preceding OpId in replica: term: 1 index: 54. Preceding OpId from leader: term: 1 index: 55. (index mismatch)
I20240503 01:59:05.039345 2530 raft_consensus.cc:1261] T 9d84cbdf02c14873b8838be296977aaf P a1a844ecd71145e4a362f4c42500fbcb [term 1 FOLLOWER]: Refusing update from remote peer 327aa26a8e434a609ebcb382aa665fce: Log matching property violated. Preceding OpId in replica: term: 1 index: 54. Preceding OpId from leader: term: 1 index: 55. (index mismatch)
I20240503 01:59:05.054366 4098 consensus_queue.cc:1035] T 9d84cbdf02c14873b8838be296977aaf P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 55, Last known committed idx: 54, Time since last communication: 0.021s
I20240503 01:59:05.104373 4214 consensus_queue.cc:1035] T 9d84cbdf02c14873b8838be296977aaf P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 55, Last known committed idx: 54, Time since last communication: 0.000s
I20240503 01:59:05.104919 4214 consensus_queue.cc:1035] T 9d84cbdf02c14873b8838be296977aaf P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 55, Last known committed idx: 54, Time since last communication: 0.000s
I20240503 01:59:05.133360 3551 raft_consensus.cc:2932] T 9d84cbdf02c14873b8838be296977aaf P 327aa26a8e434a609ebcb382aa665fce [term 1 LEADER]: Committing config change with OpId 1.55: config changed from index 35 to 55, 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 55 OBSOLETE_local: false peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:05.136420 3054 raft_consensus.cc:2932] T 9d84cbdf02c14873b8838be296977aaf P 6568f69a78c44bb985162d2b7d0a9376 [term 1 FOLLOWER]: Committing config change with OpId 1.55: config changed from index 35 to 55, 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 55 OBSOLETE_local: false peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:05.142474 2661 raft_consensus.cc:2932] T 9d84cbdf02c14873b8838be296977aaf P b40a9a7b30d34a6db007ed8febd8a759 [term 1 FOLLOWER]: Committing config change with OpId 1.55: config changed from index 35 to 55, 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 55 OBSOLETE_local: false peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:05.146028 4349 ts_tablet_manager.cc:902] T a6e17a2c4ba0482e96d04eb9d3ba854a P b63928177fa54227a5ff9929265ab9dd: Initiating tablet copy from peer 327aa26a8e434a609ebcb382aa665fce (127.0.99.133:38853)
I20240503 01:59:05.147152 4349 tablet_copy_client.cc:308] T a6e17a2c4ba0482e96d04eb9d3ba854a P b63928177fa54227a5ff9929265ab9dd: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.133:38853
I20240503 01:59:05.147701 2944 tablet_copy_service.cc:140] P 327aa26a8e434a609ebcb382aa665fce: Received BeginTabletCopySession request for tablet a6e17a2c4ba0482e96d04eb9d3ba854a from peer b63928177fa54227a5ff9929265ab9dd ({username='slave'} at 127.0.99.129:33595)
I20240503 01:59:05.147791 2944 tablet_copy_service.cc:161] P 327aa26a8e434a609ebcb382aa665fce: Beginning new tablet copy session on tablet a6e17a2c4ba0482e96d04eb9d3ba854a from peer b63928177fa54227a5ff9929265ab9dd at {username='slave'} at 127.0.99.129:33595: session id = b63928177fa54227a5ff9929265ab9dd-a6e17a2c4ba0482e96d04eb9d3ba854a
I20240503 01:59:05.148874 2944 tablet_copy_source_session.cc:205] T a6e17a2c4ba0482e96d04eb9d3ba854a P 327aa26a8e434a609ebcb382aa665fce: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:05.151307 2924 consensus_queue.cc:237] T d4d07c9621134780bc7ece7bb061d934 P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 53, Committed index: 53, Last appended: 1.54, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 55 OBSOLETE_local: false peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } attrs { replace: true } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: true } }
I20240503 01:59:05.153381 2661 raft_consensus.cc:1261] T d4d07c9621134780bc7ece7bb061d934 P b40a9a7b30d34a6db007ed8febd8a759 [term 1 FOLLOWER]: Refusing update from remote peer 327aa26a8e434a609ebcb382aa665fce: Log matching property violated. Preceding OpId in replica: term: 1 index: 54. Preceding OpId from leader: term: 1 index: 55. (index mismatch)
I20240503 01:59:05.153878 4098 consensus_queue.cc:1035] T d4d07c9621134780bc7ece7bb061d934 P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 55, Last known committed idx: 53, Time since last communication: 0.000s
I20240503 01:59:05.154629 3052 raft_consensus.cc:1261] T d4d07c9621134780bc7ece7bb061d934 P 6568f69a78c44bb985162d2b7d0a9376 [term 1 FOLLOWER]: Refusing update from remote peer 327aa26a8e434a609ebcb382aa665fce: Log matching property violated. Preceding OpId in replica: term: 1 index: 54. Preceding OpId from leader: term: 1 index: 55. (index mismatch)
I20240503 01:59:05.155042 4098 consensus_queue.cc:1035] T d4d07c9621134780bc7ece7bb061d934 P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 55, Last known committed idx: 53, Time since last communication: 0.000s
I20240503 01:59:05.155261 2253 catalog_manager.cc:5484] T 9d84cbdf02c14873b8838be296977aaf P 6568f69a78c44bb985162d2b7d0a9376 reported cstate change: config changed from index 35 to 55, 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "327aa26a8e434a609ebcb382aa665fce" committed_config { opid_index: 55 OBSOLETE_local: false peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:05.158169 4214 raft_consensus.cc:2932] T d4d07c9621134780bc7ece7bb061d934 P 327aa26a8e434a609ebcb382aa665fce [term 1 LEADER]: Committing config change with OpId 1.55: config changed from index -1 to 55, NON_VOTER 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) added. New config: { opid_index: 55 OBSOLETE_local: false peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } attrs { replace: true } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: true } } }
I20240503 01:59:05.159758 2530 raft_consensus.cc:2932] T 9d84cbdf02c14873b8838be296977aaf P a1a844ecd71145e4a362f4c42500fbcb [term 1 FOLLOWER]: Committing config change with OpId 1.55: config changed from index 35 to 55, 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 55 OBSOLETE_local: false peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:05.163355 4349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6e17a2c4ba0482e96d04eb9d3ba854a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:05.163545 2530 raft_consensus.cc:1261] T d4d07c9621134780bc7ece7bb061d934 P a1a844ecd71145e4a362f4c42500fbcb [term 1 FOLLOWER]: Refusing update from remote peer 327aa26a8e434a609ebcb382aa665fce: Log matching property violated. Preceding OpId in replica: term: 1 index: 53. Preceding OpId from leader: term: 1 index: 55. (index mismatch)
I20240503 01:59:05.163867 3958 raft_consensus.cc:2932] T 9d84cbdf02c14873b8838be296977aaf P 4125b1a70e864e129bd54bdc2720d7fa [term 1 FOLLOWER]: Committing config change with OpId 1.55: config changed from index 35 to 55, 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 55 OBSOLETE_local: false peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:05.163625 2400 raft_consensus.cc:1261] T d4d07c9621134780bc7ece7bb061d934 P b63928177fa54227a5ff9929265ab9dd [term 1 FOLLOWER]: Refusing update from remote peer 327aa26a8e434a609ebcb382aa665fce: Log matching property violated. Preceding OpId in replica: term: 1 index: 52. Preceding OpId from leader: term: 1 index: 55. (index mismatch)
I20240503 01:59:05.166832 4350 ts_tablet_manager.cc:902] T c10343409a2a423eb0b21759bfc9ba61 P 4125b1a70e864e129bd54bdc2720d7fa: Initiating tablet copy from peer b63928177fa54227a5ff9929265ab9dd (127.0.99.129:36425)
I20240503 01:59:05.167455 4350 tablet_copy_client.cc:308] T c10343409a2a423eb0b21759bfc9ba61 P 4125b1a70e864e129bd54bdc2720d7fa: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:36425
I20240503 01:59:05.167711 4349 tablet_copy_client.cc:783] T a6e17a2c4ba0482e96d04eb9d3ba854a P b63928177fa54227a5ff9929265ab9dd: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:05.167904 4349 tablet_copy_client.cc:647] T a6e17a2c4ba0482e96d04eb9d3ba854a P b63928177fa54227a5ff9929265ab9dd: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:05.168735 2420 tablet_copy_service.cc:140] P b63928177fa54227a5ff9929265ab9dd: Received BeginTabletCopySession request for tablet c10343409a2a423eb0b21759bfc9ba61 from peer 4125b1a70e864e129bd54bdc2720d7fa ({username='slave'} at 127.0.99.135:45173)
I20240503 01:59:05.168812 2420 tablet_copy_service.cc:161] P b63928177fa54227a5ff9929265ab9dd: Beginning new tablet copy session on tablet c10343409a2a423eb0b21759bfc9ba61 from peer 4125b1a70e864e129bd54bdc2720d7fa at {username='slave'} at 127.0.99.135:45173: session id = 4125b1a70e864e129bd54bdc2720d7fa-c10343409a2a423eb0b21759bfc9ba61
I20240503 01:59:05.169703 2420 tablet_copy_source_session.cc:205] T c10343409a2a423eb0b21759bfc9ba61 P b63928177fa54227a5ff9929265ab9dd: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:05.177093 3052 raft_consensus.cc:2932] T d4d07c9621134780bc7ece7bb061d934 P 6568f69a78c44bb985162d2b7d0a9376 [term 1 FOLLOWER]: Committing config change with OpId 1.55: config changed from index -1 to 55, NON_VOTER 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) added. New config: { opid_index: 55 OBSOLETE_local: false peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } attrs { replace: true } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: true } } }
I20240503 01:59:05.182737 4350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c10343409a2a423eb0b21759bfc9ba61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:05.183969 2661 raft_consensus.cc:2932] T d4d07c9621134780bc7ece7bb061d934 P b40a9a7b30d34a6db007ed8febd8a759 [term 1 FOLLOWER]: Committing config change with OpId 1.55: config changed from index -1 to 55, NON_VOTER 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) added. New config: { opid_index: 55 OBSOLETE_local: false peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } attrs { replace: true } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: true } } }
I20240503 01:59:05.185098 4350 tablet_copy_client.cc:783] T c10343409a2a423eb0b21759bfc9ba61 P 4125b1a70e864e129bd54bdc2720d7fa: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:05.185286 4350 tablet_copy_client.cc:647] T c10343409a2a423eb0b21759bfc9ba61 P 4125b1a70e864e129bd54bdc2720d7fa: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:05.186033 2253 catalog_manager.cc:5484] T d4d07c9621134780bc7ece7bb061d934 P 6568f69a78c44bb985162d2b7d0a9376 reported cstate change: config changed from index -1 to 55, NON_VOTER 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "327aa26a8e434a609ebcb382aa665fce" committed_config { opid_index: 55 OBSOLETE_local: false peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } attrs { replace: true } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: true } } }
I20240503 01:59:05.196101 4214 consensus_queue.cc:1035] T d4d07c9621134780bc7ece7bb061d934 P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 55, Last known committed idx: 53, Time since last communication: 0.000s
I20240503 01:59:05.207937 4365 ts_tablet_manager.cc:902] T 568a05972208414c90f7f4e2ea2c10a9 P 4125b1a70e864e129bd54bdc2720d7fa: Initiating tablet copy from peer b40a9a7b30d34a6db007ed8febd8a759 (127.0.99.131:34079)
I20240503 01:59:05.208753 4365 tablet_copy_client.cc:308] T 568a05972208414c90f7f4e2ea2c10a9 P 4125b1a70e864e129bd54bdc2720d7fa: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:34079
I20240503 01:59:05.212584 2529 raft_consensus.cc:2932] T d4d07c9621134780bc7ece7bb061d934 P a1a844ecd71145e4a362f4c42500fbcb [term 1 FOLLOWER]: Committing config change with OpId 1.55: config changed from index -1 to 55, NON_VOTER 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) added. New config: { opid_index: 55 OBSOLETE_local: false peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } attrs { replace: true } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: true } } }
W20240503 01:59:05.219096 2860 consensus_peers.cc:485] T d4d07c9621134780bc7ece7bb061d934 P 327aa26a8e434a609ebcb382aa665fce -> Peer 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135:39501): Couldn't send request to peer 4125b1a70e864e129bd54bdc2720d7fa. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d4d07c9621134780bc7ece7bb061d934. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:05.221616 4214 consensus_queue.cc:1035] T d4d07c9621134780bc7ece7bb061d934 P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 55, Last known committed idx: 52, Time since last communication: 0.000s
I20240503 01:59:05.222972 2790 raft_consensus.cc:2932] T 9d84cbdf02c14873b8838be296977aaf P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Committing config change with OpId 1.55: config changed from index 35 to 55, 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 55 OBSOLETE_local: false peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:05.228473 2682 tablet_copy_service.cc:140] P b40a9a7b30d34a6db007ed8febd8a759: Received BeginTabletCopySession request for tablet 568a05972208414c90f7f4e2ea2c10a9 from peer 4125b1a70e864e129bd54bdc2720d7fa ({username='slave'} at 127.0.99.135:51755)
I20240503 01:59:05.228575 2682 tablet_copy_service.cc:161] P b40a9a7b30d34a6db007ed8febd8a759: Beginning new tablet copy session on tablet 568a05972208414c90f7f4e2ea2c10a9 from peer 4125b1a70e864e129bd54bdc2720d7fa at {username='slave'} at 127.0.99.135:51755: session id = 4125b1a70e864e129bd54bdc2720d7fa-568a05972208414c90f7f4e2ea2c10a9
I20240503 01:59:05.229595 2682 tablet_copy_source_session.cc:205] T 568a05972208414c90f7f4e2ea2c10a9 P b40a9a7b30d34a6db007ed8febd8a759: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:05.231318 4365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 568a05972208414c90f7f4e2ea2c10a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:05.235334 4365 tablet_copy_client.cc:783] T 568a05972208414c90f7f4e2ea2c10a9 P 4125b1a70e864e129bd54bdc2720d7fa: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:05.235925 4365 tablet_copy_client.cc:647] T 568a05972208414c90f7f4e2ea2c10a9 P 4125b1a70e864e129bd54bdc2720d7fa: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:05.239709 4365 tablet_copy_client.cc:523] T 568a05972208414c90f7f4e2ea2c10a9 P 4125b1a70e864e129bd54bdc2720d7fa: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:05.241001 4365 tablet_bootstrap.cc:492] T 568a05972208414c90f7f4e2ea2c10a9 P 4125b1a70e864e129bd54bdc2720d7fa: Bootstrap starting.
I20240503 01:59:05.252014 2400 raft_consensus.cc:2932] T d4d07c9621134780bc7ece7bb061d934 P b63928177fa54227a5ff9929265ab9dd [term 1 FOLLOWER]: Committing config change with OpId 1.55: config changed from index -1 to 55, NON_VOTER 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) added. New config: { opid_index: 55 OBSOLETE_local: false peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } attrs { replace: true } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: true } } }
I20240503 01:59:05.281965 4350 tablet_copy_client.cc:523] T c10343409a2a423eb0b21759bfc9ba61 P 4125b1a70e864e129bd54bdc2720d7fa: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:05.283545 4350 tablet_bootstrap.cc:492] T c10343409a2a423eb0b21759bfc9ba61 P 4125b1a70e864e129bd54bdc2720d7fa: Bootstrap starting.
W20240503 01:59:05.319929 2598 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.132:42887, user_credentials={real_user=slave}} blocked reactor thread for 51791.4us
I20240503 01:59:05.328475 2921 consensus_queue.cc:237] T 9d84cbdf02c14873b8838be296977aaf P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 57, Committed index: 57, Last appended: 1.58, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 59 OBSOLETE_local: false peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } }
I20240503 01:59:05.330003 3052 raft_consensus.cc:1261] T 9d84cbdf02c14873b8838be296977aaf P 6568f69a78c44bb985162d2b7d0a9376 [term 1 FOLLOWER]: Refusing update from remote peer 327aa26a8e434a609ebcb382aa665fce: Log matching property violated. Preceding OpId in replica: term: 1 index: 58. Preceding OpId from leader: term: 1 index: 59. (index mismatch)
I20240503 01:59:05.331547 4098 consensus_queue.cc:1035] T 9d84cbdf02c14873b8838be296977aaf P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 59, Last known committed idx: 57, Time since last communication: 0.000s
I20240503 01:59:05.370144 2529 raft_consensus.cc:1261] T 9d84cbdf02c14873b8838be296977aaf P a1a844ecd71145e4a362f4c42500fbcb [term 1 FOLLOWER]: Refusing update from remote peer 327aa26a8e434a609ebcb382aa665fce: Log matching property violated. Preceding OpId in replica: term: 1 index: 57. Preceding OpId from leader: term: 1 index: 59. (index mismatch)
I20240503 01:59:05.370214 2793 raft_consensus.cc:1261] T 9d84cbdf02c14873b8838be296977aaf P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Refusing update from remote peer 327aa26a8e434a609ebcb382aa665fce: Log matching property violated. Preceding OpId in replica: term: 1 index: 57. Preceding OpId from leader: term: 1 index: 59. (index mismatch)
I20240503 01:59:05.370383 3957 raft_consensus.cc:1261] T 9d84cbdf02c14873b8838be296977aaf P 4125b1a70e864e129bd54bdc2720d7fa [term 1 FOLLOWER]: Refusing update from remote peer 327aa26a8e434a609ebcb382aa665fce: Log matching property violated. Preceding OpId in replica: term: 1 index: 58. Preceding OpId from leader: term: 1 index: 59. (index mismatch)
I20240503 01:59:05.416469 3551 consensus_queue.cc:1035] T 9d84cbdf02c14873b8838be296977aaf P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 59, Last known committed idx: 57, Time since last communication: 0.000s
I20240503 01:59:05.419124 3551 consensus_queue.cc:1035] T 9d84cbdf02c14873b8838be296977aaf P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 59, Last known committed idx: 57, Time since last communication: 0.000s
I20240503 01:59:05.420284 3551 consensus_queue.cc:1035] T 9d84cbdf02c14873b8838be296977aaf P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 59, Last known committed idx: 57, Time since last communication: 0.000s
I20240503 01:59:05.441314 4349 tablet_copy_client.cc:523] T a6e17a2c4ba0482e96d04eb9d3ba854a P b63928177fa54227a5ff9929265ab9dd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:05.443434 4349 tablet_bootstrap.cc:492] T a6e17a2c4ba0482e96d04eb9d3ba854a P b63928177fa54227a5ff9929265ab9dd: Bootstrap starting.
I20240503 01:59:05.445679 3545 raft_consensus.cc:1050] T 240749b0904f4bb2a4c79b7713473411 P b40a9a7b30d34a6db007ed8febd8a759: attempting to promote NON_VOTER 327aa26a8e434a609ebcb382aa665fce to VOTER
I20240503 01:59:05.446192 3545 consensus_queue.cc:237] T 240749b0904f4bb2a4c79b7713473411 P b40a9a7b30d34a6db007ed8febd8a759 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { replace: true } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { promote: false } }
I20240503 01:59:05.448119 2921 raft_consensus.cc:1261] T 240749b0904f4bb2a4c79b7713473411 P 327aa26a8e434a609ebcb382aa665fce [term 1 LEARNER]: Refusing update from remote peer b40a9a7b30d34a6db007ed8febd8a759: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 01:59:05.448604 4251 consensus_queue.cc:1035] T 240749b0904f4bb2a4c79b7713473411 P b40a9a7b30d34a6db007ed8febd8a759 [LEADER]: Connected to new peer: Peer: permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 01:59:05.451957 3054 raft_consensus.cc:1261] T 240749b0904f4bb2a4c79b7713473411 P 6568f69a78c44bb985162d2b7d0a9376 [term 1 FOLLOWER]: Refusing update from remote peer b40a9a7b30d34a6db007ed8febd8a759: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 01:59:05.452944 4335 consensus_queue.cc:1035] T 240749b0904f4bb2a4c79b7713473411 P b40a9a7b30d34a6db007ed8febd8a759 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 01:59:05.461849 3957 raft_consensus.cc:1261] T 240749b0904f4bb2a4c79b7713473411 P 4125b1a70e864e129bd54bdc2720d7fa [term 1 FOLLOWER]: Refusing update from remote peer b40a9a7b30d34a6db007ed8febd8a759: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 01:59:05.462327 2792 raft_consensus.cc:1261] T 240749b0904f4bb2a4c79b7713473411 P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Refusing update from remote peer b40a9a7b30d34a6db007ed8febd8a759: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 01:59:05.469458 2394 raft_consensus.cc:1261] T 240749b0904f4bb2a4c79b7713473411 P b63928177fa54227a5ff9929265ab9dd [term 1 FOLLOWER]: Refusing update from remote peer b40a9a7b30d34a6db007ed8febd8a759: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 01:59:05.472074 4214 raft_consensus.cc:2932] T 9d84cbdf02c14873b8838be296977aaf P 327aa26a8e434a609ebcb382aa665fce [term 1 LEADER]: Committing config change with OpId 1.59: config changed from index 55 to 59, VOTER b40a9a7b30d34a6db007ed8febd8a759 (127.0.99.131) evicted. New config: { opid_index: 59 OBSOLETE_local: false peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:05.474575 2243 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9d84cbdf02c14873b8838be296977aaf with cas_config_opid_index 55: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:05.476280 3054 raft_consensus.cc:2932] T 9d84cbdf02c14873b8838be296977aaf P 6568f69a78c44bb985162d2b7d0a9376 [term 1 FOLLOWER]: Committing config change with OpId 1.59: config changed from index 55 to 59, VOTER b40a9a7b30d34a6db007ed8febd8a759 (127.0.99.131) evicted. New config: { opid_index: 59 OBSOLETE_local: false peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:05.479341 2253 catalog_manager.cc:5484] T 9d84cbdf02c14873b8838be296977aaf P 6568f69a78c44bb985162d2b7d0a9376 reported cstate change: config changed from index 55 to 59, VOTER b40a9a7b30d34a6db007ed8febd8a759 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "327aa26a8e434a609ebcb382aa665fce" committed_config { opid_index: 59 OBSOLETE_local: false peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:05.480100 4327 consensus_queue.cc:1035] T 240749b0904f4bb2a4c79b7713473411 P b40a9a7b30d34a6db007ed8febd8a759 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 01:59:05.482486 2641 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d84cbdf02c14873b8838be296977aaf with delete_type TABLET_DATA_TOMBSTONED (TS b40a9a7b30d34a6db007ed8febd8a759 not found in new config with opid_index 59) from {username='slave'} at 127.0.0.1:40954
I20240503 01:59:05.486847 4327 consensus_queue.cc:1035] T 240749b0904f4bb2a4c79b7713473411 P b40a9a7b30d34a6db007ed8febd8a759 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 01:59:05.488869 4250 consensus_queue.cc:1035] T 240749b0904f4bb2a4c79b7713473411 P b40a9a7b30d34a6db007ed8febd8a759 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 01:59:05.514483 2790 raft_consensus.cc:2932] T 9d84cbdf02c14873b8838be296977aaf P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Committing config change with OpId 1.59: config changed from index 55 to 59, VOTER b40a9a7b30d34a6db007ed8febd8a759 (127.0.99.131) evicted. New config: { opid_index: 59 OBSOLETE_local: false peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:05.528147 4388 tablet_replica.cc:330] T 9d84cbdf02c14873b8838be296977aaf P b40a9a7b30d34a6db007ed8febd8a759: stopping tablet replica
I20240503 01:59:05.528791 4388 raft_consensus.cc:2229] T 9d84cbdf02c14873b8838be296977aaf P b40a9a7b30d34a6db007ed8febd8a759 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:05.544916 4388 pending_rounds.cc:62] T 9d84cbdf02c14873b8838be296977aaf P b40a9a7b30d34a6db007ed8febd8a759: Trying to abort 1 pending ops.
I20240503 01:59:05.545110 4388 pending_rounds.cc:68] T 9d84cbdf02c14873b8838be296977aaf P b40a9a7b30d34a6db007ed8febd8a759: Aborting op as it isn't in flight: id { term: 1 index: 58 } timestamp: 7023417529407463424 op_type: WRITE_OP write_request { tablet_id: "9d84cbdf02c14873b8838be296977aaf" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\224\223\363l,B\312\354\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""2e3ecff83b8f7500" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023417529342402560 authz_token { token_data: "\010\221\214\321\261\006\"3\n\005slave\022*\n d0846ac337c64bf1ac218add4c796fff\020\001\030\001 \001(\001" signature: "<redacted>""f\"w\200@\373>yk\204\026\017`\207;\333\274NS\371\203\003G\254\205\216\317jn9\304S&^\247\331\351\325\026\034\360\211\277\271k\365\212\336\224KO\223\005\375\245mo\223\2328U\tg\216" signing_key_seq_num: 0 } } request_id { client_id: "45556a5dc8664c03ba879df420f1330c" seq_no: 158 first_incomplete_seq_no: 158 attempt_no: 0 }
I20240503 01:59:05.546103 4388 raft_consensus.cc:2258] T 9d84cbdf02c14873b8838be296977aaf P b40a9a7b30d34a6db007ed8febd8a759 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:05.548213 4326 raft_consensus.cc:2932] T 240749b0904f4bb2a4c79b7713473411 P b40a9a7b30d34a6db007ed8febd8a759 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 327aa26a8e434a609ebcb382aa665fce (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { replace: true } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { promote: false } } }
I20240503 01:59:05.550055 3957 raft_consensus.cc:2932] T 9d84cbdf02c14873b8838be296977aaf P 4125b1a70e864e129bd54bdc2720d7fa [term 1 FOLLOWER]: Committing config change with OpId 1.59: config changed from index 55 to 59, VOTER b40a9a7b30d34a6db007ed8febd8a759 (127.0.99.131) evicted. New config: { opid_index: 59 OBSOLETE_local: false peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:05.551689 3054 raft_consensus.cc:2932] T 240749b0904f4bb2a4c79b7713473411 P 6568f69a78c44bb985162d2b7d0a9376 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 327aa26a8e434a609ebcb382aa665fce (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { replace: true } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { promote: false } } }
I20240503 01:59:05.553687 2922 raft_consensus.cc:2932] T 240749b0904f4bb2a4c79b7713473411 P 327aa26a8e434a609ebcb382aa665fce [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 327aa26a8e434a609ebcb382aa665fce (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { replace: true } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { promote: false } } }
I20240503 01:59:05.555737 4388 ts_tablet_manager.cc:1875] T 9d84cbdf02c14873b8838be296977aaf P b40a9a7b30d34a6db007ed8febd8a759: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:05.556919 2250 catalog_manager.cc:5484] T 240749b0904f4bb2a4c79b7713473411 P 6568f69a78c44bb985162d2b7d0a9376 reported cstate change: config changed from index 2 to 3, 327aa26a8e434a609ebcb382aa665fce (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b40a9a7b30d34a6db007ed8febd8a759" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { replace: true } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { promote: false } } }
I20240503 01:59:05.563248 2524 raft_consensus.cc:2932] T 9d84cbdf02c14873b8838be296977aaf P a1a844ecd71145e4a362f4c42500fbcb [term 1 FOLLOWER]: Committing config change with OpId 1.59: config changed from index 55 to 59, VOTER b40a9a7b30d34a6db007ed8febd8a759 (127.0.99.131) evicted. New config: { opid_index: 59 OBSOLETE_local: false peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:05.566790 3957 raft_consensus.cc:2932] T 240749b0904f4bb2a4c79b7713473411 P 4125b1a70e864e129bd54bdc2720d7fa [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 327aa26a8e434a609ebcb382aa665fce (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { replace: true } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { promote: false } } }
I20240503 01:59:05.575222 2789 raft_consensus.cc:2932] T 240749b0904f4bb2a4c79b7713473411 P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 327aa26a8e434a609ebcb382aa665fce (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { replace: true } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { promote: false } } }
I20240503 01:59:05.647014 2659 consensus_queue.cc:237] T 240749b0904f4bb2a4c79b7713473411 P b40a9a7b30d34a6db007ed8febd8a759 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { promote: false } }
I20240503 01:59:05.648396 2921 raft_consensus.cc:1261] T 240749b0904f4bb2a4c79b7713473411 P 327aa26a8e434a609ebcb382aa665fce [term 1 FOLLOWER]: Refusing update from remote peer b40a9a7b30d34a6db007ed8febd8a759: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:05.648864 4250 consensus_queue.cc:1035] T 240749b0904f4bb2a4c79b7713473411 P b40a9a7b30d34a6db007ed8febd8a759 [LEADER]: Connected to new peer: Peer: permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:05.650962 3055 raft_consensus.cc:1261] T 240749b0904f4bb2a4c79b7713473411 P 6568f69a78c44bb985162d2b7d0a9376 [term 1 FOLLOWER]: Refusing update from remote peer b40a9a7b30d34a6db007ed8febd8a759: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:05.651420 4250 consensus_queue.cc:1035] T 240749b0904f4bb2a4c79b7713473411 P b40a9a7b30d34a6db007ed8febd8a759 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:05.654552 3957 raft_consensus.cc:1261] T 240749b0904f4bb2a4c79b7713473411 P 4125b1a70e864e129bd54bdc2720d7fa [term 1 FOLLOWER]: Refusing update from remote peer b40a9a7b30d34a6db007ed8febd8a759: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:05.654850 4335 raft_consensus.cc:2932] T 240749b0904f4bb2a4c79b7713473411 P b40a9a7b30d34a6db007ed8febd8a759 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b63928177fa54227a5ff9929265ab9dd (127.0.99.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { promote: false } } }
I20240503 01:59:05.655272 3055 raft_consensus.cc:2932] T 240749b0904f4bb2a4c79b7713473411 P 6568f69a78c44bb985162d2b7d0a9376 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b63928177fa54227a5ff9929265ab9dd (127.0.99.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { promote: false } } }
I20240503 01:59:05.655962 4388 ts_tablet_manager.cc:1888] T 9d84cbdf02c14873b8838be296977aaf P b40a9a7b30d34a6db007ed8febd8a759: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.58
I20240503 01:59:05.656069 4388 log.cc:1196] T 9d84cbdf02c14873b8838be296977aaf P b40a9a7b30d34a6db007ed8febd8a759: Deleting WAL directory at /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/9d84cbdf02c14873b8838be296977aaf
I20240503 01:59:05.657244 2242 catalog_manager.cc:4830] TS b40a9a7b30d34a6db007ed8febd8a759 (127.0.99.131:34079): tablet 9d84cbdf02c14873b8838be296977aaf (table rebalance_test_table_1 [id=d0846ac337c64bf1ac218add4c796fff]) successfully deleted
I20240503 01:59:05.657750 2242 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 240749b0904f4bb2a4c79b7713473411 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:05.658432 2921 raft_consensus.cc:2932] T 240749b0904f4bb2a4c79b7713473411 P 327aa26a8e434a609ebcb382aa665fce [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b63928177fa54227a5ff9929265ab9dd (127.0.99.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { promote: false } } }
I20240503 01:59:05.658577 2790 raft_consensus.cc:1261] T 240749b0904f4bb2a4c79b7713473411 P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Refusing update from remote peer b40a9a7b30d34a6db007ed8febd8a759: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:05.662312 4298 tablet_bootstrap.cc:492] T 5fd1010447524b6695abb98192dc54f1 P a1a844ecd71145e4a362f4c42500fbcb: Bootstrap replayed 1/1 log segments. Stats: ops{read=31 overwritten=0 applied=30 ignored=0} inserts{seen=715 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 01:59:05.663206 4335 consensus_queue.cc:1035] T 240749b0904f4bb2a4c79b7713473411 P b40a9a7b30d34a6db007ed8febd8a759 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:05.664202 2251 catalog_manager.cc:5484] T 240749b0904f4bb2a4c79b7713473411 P b40a9a7b30d34a6db007ed8febd8a759 reported cstate change: config changed from index 3 to 4, VOTER b63928177fa54227a5ff9929265ab9dd (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "b40a9a7b30d34a6db007ed8febd8a759" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:05.664273 3957 raft_consensus.cc:2932] T 240749b0904f4bb2a4c79b7713473411 P 4125b1a70e864e129bd54bdc2720d7fa [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b63928177fa54227a5ff9929265ab9dd (127.0.99.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { promote: false } } }
I20240503 01:59:05.666831 3545 consensus_queue.cc:1035] T 240749b0904f4bb2a4c79b7713473411 P b40a9a7b30d34a6db007ed8febd8a759 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:05.675288 4298 tablet_bootstrap.cc:492] T 5fd1010447524b6695abb98192dc54f1 P a1a844ecd71145e4a362f4c42500fbcb: Bootstrap complete.
I20240503 01:59:05.677512 4298 ts_tablet_manager.cc:1367] T 5fd1010447524b6695abb98192dc54f1 P a1a844ecd71145e4a362f4c42500fbcb: Time spent bootstrapping tablet: real 1.068s user 0.046s sys 0.005s
I20240503 01:59:05.677709 2790 raft_consensus.cc:2932] T 240749b0904f4bb2a4c79b7713473411 P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b63928177fa54227a5ff9929265ab9dd (127.0.99.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { promote: false } } }
I20240503 01:59:05.678414 4298 raft_consensus.cc:348] T 5fd1010447524b6695abb98192dc54f1 P a1a844ecd71145e4a362f4c42500fbcb [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46033 } attrs { promote: true } }
I20240503 01:59:05.682416 2378 tablet_service.cc:1514] Processing DeleteTablet for tablet 240749b0904f4bb2a4c79b7713473411 with delete_type TABLET_DATA_TOMBSTONED (TS b63928177fa54227a5ff9929265ab9dd not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:44104
I20240503 01:59:05.687741 4298 raft_consensus.cc:729] T 5fd1010447524b6695abb98192dc54f1 P a1a844ecd71145e4a362f4c42500fbcb [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a1a844ecd71145e4a362f4c42500fbcb, State: Initialized, Role: LEARNER
I20240503 01:59:05.688469 4298 consensus_queue.cc:260] T 5fd1010447524b6695abb98192dc54f1 P a1a844ecd71145e4a362f4c42500fbcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 30, Last appended: 1.31, Last appended by leader: 31, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46033 } attrs { promote: true } }
I20240503 01:59:05.692440 4298 ts_tablet_manager.cc:1398] T 5fd1010447524b6695abb98192dc54f1 P a1a844ecd71145e4a362f4c42500fbcb: Time spent starting tablet: real 0.014s user 0.001s sys 0.000s
I20240503 01:59:05.716300 4410 ts_tablet_manager.cc:902] T d4d07c9621134780bc7ece7bb061d934 P 4125b1a70e864e129bd54bdc2720d7fa: Initiating tablet copy from peer 327aa26a8e434a609ebcb382aa665fce (127.0.99.133:38853)
I20240503 01:59:05.717187 4410 tablet_copy_client.cc:308] T d4d07c9621134780bc7ece7bb061d934 P 4125b1a70e864e129bd54bdc2720d7fa: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.133:38853
I20240503 01:59:05.725179 4365 tablet_bootstrap.cc:492] T 568a05972208414c90f7f4e2ea2c10a9 P 4125b1a70e864e129bd54bdc2720d7fa: Bootstrap replayed 1/1 log segments. Stats: ops{read=94 overwritten=0 applied=94 ignored=0} inserts{seen=779 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:05.725687 4365 tablet_bootstrap.cc:492] T 568a05972208414c90f7f4e2ea2c10a9 P 4125b1a70e864e129bd54bdc2720d7fa: Bootstrap complete.
I20240503 01:59:05.725951 4365 ts_tablet_manager.cc:1367] T 568a05972208414c90f7f4e2ea2c10a9 P 4125b1a70e864e129bd54bdc2720d7fa: Time spent bootstrapping tablet: real 0.485s user 0.087s sys 0.012s
I20240503 01:59:05.726511 2944 tablet_copy_service.cc:342] P 327aa26a8e434a609ebcb382aa665fce: Request end of tablet copy session a1a844ecd71145e4a362f4c42500fbcb-5fd1010447524b6695abb98192dc54f1 received from {username='slave'} at 127.0.99.130:35863
I20240503 01:59:05.726608 2944 tablet_copy_service.cc:433] P 327aa26a8e434a609ebcb382aa665fce: ending tablet copy session a1a844ecd71145e4a362f4c42500fbcb-5fd1010447524b6695abb98192dc54f1 on tablet 5fd1010447524b6695abb98192dc54f1 with peer a1a844ecd71145e4a362f4c42500fbcb
I20240503 01:59:05.726517 4365 raft_consensus.cc:348] T 568a05972208414c90f7f4e2ea2c10a9 P 4125b1a70e864e129bd54bdc2720d7fa [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: true } }
I20240503 01:59:05.726799 4365 raft_consensus.cc:729] T 568a05972208414c90f7f4e2ea2c10a9 P 4125b1a70e864e129bd54bdc2720d7fa [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4125b1a70e864e129bd54bdc2720d7fa, State: Initialized, Role: LEARNER
I20240503 01:59:05.726939 4365 consensus_queue.cc:260] T 568a05972208414c90f7f4e2ea2c10a9 P 4125b1a70e864e129bd54bdc2720d7fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 94, Last appended: 1.94, Last appended by leader: 94, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: true } }
I20240503 01:59:05.727299 2944 tablet_copy_service.cc:140] P 327aa26a8e434a609ebcb382aa665fce: Received BeginTabletCopySession request for tablet d4d07c9621134780bc7ece7bb061d934 from peer 4125b1a70e864e129bd54bdc2720d7fa ({username='slave'} at 127.0.99.135:39289)
I20240503 01:59:05.727396 2944 tablet_copy_service.cc:161] P 327aa26a8e434a609ebcb382aa665fce: Beginning new tablet copy session on tablet d4d07c9621134780bc7ece7bb061d934 from peer 4125b1a70e864e129bd54bdc2720d7fa at {username='slave'} at 127.0.99.135:39289: session id = 4125b1a70e864e129bd54bdc2720d7fa-d4d07c9621134780bc7ece7bb061d934
I20240503 01:59:05.728298 2944 tablet_copy_source_session.cc:205] T d4d07c9621134780bc7ece7bb061d934 P 327aa26a8e434a609ebcb382aa665fce: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:05.729489 4365 ts_tablet_manager.cc:1398] T 568a05972208414c90f7f4e2ea2c10a9 P 4125b1a70e864e129bd54bdc2720d7fa: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 01:59:05.731587 4410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4d07c9621134780bc7ece7bb061d934. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:05.734333 4410 tablet_copy_client.cc:783] T d4d07c9621134780bc7ece7bb061d934 P 4125b1a70e864e129bd54bdc2720d7fa: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:05.734663 4410 tablet_copy_client.cc:647] T d4d07c9621134780bc7ece7bb061d934 P 4125b1a70e864e129bd54bdc2720d7fa: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:05.735424 2682 tablet_copy_service.cc:342] P b40a9a7b30d34a6db007ed8febd8a759: Request end of tablet copy session 4125b1a70e864e129bd54bdc2720d7fa-568a05972208414c90f7f4e2ea2c10a9 received from {username='slave'} at 127.0.99.135:51755
I20240503 01:59:05.735518 2682 tablet_copy_service.cc:433] P b40a9a7b30d34a6db007ed8febd8a759: ending tablet copy session 4125b1a70e864e129bd54bdc2720d7fa-568a05972208414c90f7f4e2ea2c10a9 on tablet 568a05972208414c90f7f4e2ea2c10a9 with peer 4125b1a70e864e129bd54bdc2720d7fa
I20240503 01:59:05.748245 4410 tablet_copy_client.cc:523] T d4d07c9621134780bc7ece7bb061d934 P 4125b1a70e864e129bd54bdc2720d7fa: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:05.767915 3957 raft_consensus.cc:1203] T 568a05972208414c90f7f4e2ea2c10a9 P 4125b1a70e864e129bd54bdc2720d7fa [term 1 LEARNER]: Deduplicated request from leader. Original: 1.78->[1.79-1.100] Dedup: 1.94->[1.95-1.100]
I20240503 01:59:05.780500 4409 tablet_replica.cc:330] T 240749b0904f4bb2a4c79b7713473411 P b63928177fa54227a5ff9929265ab9dd: stopping tablet replica
I20240503 01:59:05.780970 4409 raft_consensus.cc:2229] T 240749b0904f4bb2a4c79b7713473411 P b63928177fa54227a5ff9929265ab9dd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:05.781414 4409 pending_rounds.cc:62] T 240749b0904f4bb2a4c79b7713473411 P b63928177fa54227a5ff9929265ab9dd: Trying to abort 1 pending ops.
I20240503 01:59:05.781546 4409 pending_rounds.cc:68] T 240749b0904f4bb2a4c79b7713473411 P b63928177fa54227a5ff9929265ab9dd: Aborting op as it isn't in flight: id { term: 1 index: 3 } timestamp: 7023417530146398208 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "240749b0904f4bb2a4c79b7713473411" old_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { replace: true } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { promote: true } } } new_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { replace: true } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { promote: false } } } }
I20240503 01:59:05.781736 4409 raft_consensus.cc:2894] T 240749b0904f4bb2a4c79b7713473411 P b63928177fa54227a5ff9929265ab9dd [term 1 FOLLOWER]: Aborting config change with OpId 1.3: Aborted: Op aborted
I20240503 01:59:05.781813 4409 raft_consensus.cc:2866] T 240749b0904f4bb2a4c79b7713473411 P b63928177fa54227a5ff9929265ab9dd [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240503 01:59:05.781867 4409 raft_consensus.cc:2258] T 240749b0904f4bb2a4c79b7713473411 P b63928177fa54227a5ff9929265ab9dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:05.786582 4409 ts_tablet_manager.cc:1875] T 240749b0904f4bb2a4c79b7713473411 P b63928177fa54227a5ff9929265ab9dd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:05.814358 4410 tablet_bootstrap.cc:492] T d4d07c9621134780bc7ece7bb061d934 P 4125b1a70e864e129bd54bdc2720d7fa: Bootstrap starting.
I20240503 01:59:05.822463 2520 raft_consensus.cc:1203] T 5fd1010447524b6695abb98192dc54f1 P a1a844ecd71145e4a362f4c42500fbcb [term 1 LEARNER]: Deduplicated request from leader. Original: 1.24->[1.25-1.36] Dedup: 1.31->[1.32-1.36]
I20240503 01:59:05.874122 4409 ts_tablet_manager.cc:1888] T 240749b0904f4bb2a4c79b7713473411 P b63928177fa54227a5ff9929265ab9dd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 01:59:05.874261 4409 log.cc:1196] T 240749b0904f4bb2a4c79b7713473411 P b63928177fa54227a5ff9929265ab9dd: Deleting WAL directory at /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/240749b0904f4bb2a4c79b7713473411
I20240503 01:59:05.904357 2245 catalog_manager.cc:4830] TS b63928177fa54227a5ff9929265ab9dd (127.0.99.129:36425): tablet 240749b0904f4bb2a4c79b7713473411 (table TestTable [id=e423c6476d1b43779e0a839bf55a520d]) successfully deleted
I20240503 01:59:05.914175 4250 raft_consensus.cc:1050] T 568a05972208414c90f7f4e2ea2c10a9 P b40a9a7b30d34a6db007ed8febd8a759: attempting to promote NON_VOTER 4125b1a70e864e129bd54bdc2720d7fa to VOTER
I20240503 01:59:05.914813 4250 consensus_queue.cc:237] T 568a05972208414c90f7f4e2ea2c10a9 P b40a9a7b30d34a6db007ed8febd8a759 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 103, Committed index: 103, Last appended: 1.103, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } }
I20240503 01:59:05.918180 2924 raft_consensus.cc:1261] T 568a05972208414c90f7f4e2ea2c10a9 P 327aa26a8e434a609ebcb382aa665fce [term 1 FOLLOWER]: Refusing update from remote peer b40a9a7b30d34a6db007ed8febd8a759: Log matching property violated. Preceding OpId in replica: term: 1 index: 103. Preceding OpId from leader: term: 1 index: 104. (index mismatch)
I20240503 01:59:05.918937 4335 consensus_queue.cc:1035] T 568a05972208414c90f7f4e2ea2c10a9 P b40a9a7b30d34a6db007ed8febd8a759 [LEADER]: Connected to new peer: Peer: permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 104, Last known committed idx: 103, Time since last communication: 0.000s
I20240503 01:59:05.920156 3957 raft_consensus.cc:1261] T 568a05972208414c90f7f4e2ea2c10a9 P 4125b1a70e864e129bd54bdc2720d7fa [term 1 LEARNER]: Refusing update from remote peer b40a9a7b30d34a6db007ed8febd8a759: Log matching property violated. Preceding OpId in replica: term: 1 index: 103. Preceding OpId from leader: term: 1 index: 104. (index mismatch)
I20240503 01:59:05.920392 2789 raft_consensus.cc:1261] T 568a05972208414c90f7f4e2ea2c10a9 P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Refusing update from remote peer b40a9a7b30d34a6db007ed8febd8a759: Log matching property violated. Preceding OpId in replica: term: 1 index: 103. Preceding OpId from leader: term: 1 index: 104. (index mismatch)
I20240503 01:59:05.921839 3545 consensus_queue.cc:1035] T 568a05972208414c90f7f4e2ea2c10a9 P b40a9a7b30d34a6db007ed8febd8a759 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 104, Last known committed idx: 103, Time since last communication: 0.000s
I20240503 01:59:05.921967 4250 consensus_queue.cc:1035] T 568a05972208414c90f7f4e2ea2c10a9 P b40a9a7b30d34a6db007ed8febd8a759 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 104, Last known committed idx: 103, Time since last communication: 0.000s
I20240503 01:59:05.924322 4250 raft_consensus.cc:2932] T 568a05972208414c90f7f4e2ea2c10a9 P b40a9a7b30d34a6db007ed8febd8a759 [term 1 LEADER]: Committing config change with OpId 1.104: config changed from index 79 to 104, 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:05.926359 2531 raft_consensus.cc:1261] T 568a05972208414c90f7f4e2ea2c10a9 P a1a844ecd71145e4a362f4c42500fbcb [term 1 FOLLOWER]: Refusing update from remote peer b40a9a7b30d34a6db007ed8febd8a759: Log matching property violated. Preceding OpId in replica: term: 1 index: 103. Preceding OpId from leader: term: 1 index: 104. (index mismatch)
I20240503 01:59:05.926364 3957 raft_consensus.cc:2932] T 568a05972208414c90f7f4e2ea2c10a9 P 4125b1a70e864e129bd54bdc2720d7fa [term 1 FOLLOWER]: Committing config change with OpId 1.104: config changed from index 79 to 104, 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:05.926831 4335 consensus_queue.cc:1035] T 568a05972208414c90f7f4e2ea2c10a9 P b40a9a7b30d34a6db007ed8febd8a759 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 104, Last known committed idx: 103, Time since last communication: 0.000s
I20240503 01:59:05.928097 2520 raft_consensus.cc:2932] T 568a05972208414c90f7f4e2ea2c10a9 P a1a844ecd71145e4a362f4c42500fbcb [term 1 FOLLOWER]: Committing config change with OpId 1.104: config changed from index 79 to 104, 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:05.936197 4285 tablet_bootstrap.cc:492] T 19b3a90e13194997bde2b9a806f8f2b5 P 4125b1a70e864e129bd54bdc2720d7fa: Bootstrap replayed 1/1 log segments. Stats: ops{read=77 overwritten=0 applied=77 ignored=0} inserts{seen=762 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:05.940307 4349 tablet_bootstrap.cc:492] T a6e17a2c4ba0482e96d04eb9d3ba854a P b63928177fa54227a5ff9929265ab9dd: Bootstrap replayed 1/1 log segments. Stats: ops{read=61 overwritten=0 applied=61 ignored=0} inserts{seen=746 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:05.942838 4349 tablet_bootstrap.cc:492] T a6e17a2c4ba0482e96d04eb9d3ba854a P b63928177fa54227a5ff9929265ab9dd: Bootstrap complete.
I20240503 01:59:05.943095 4349 ts_tablet_manager.cc:1367] T a6e17a2c4ba0482e96d04eb9d3ba854a P b63928177fa54227a5ff9929265ab9dd: Time spent bootstrapping tablet: real 0.500s user 0.064s sys 0.012s
I20240503 01:59:05.943624 4349 raft_consensus.cc:348] T a6e17a2c4ba0482e96d04eb9d3ba854a P b63928177fa54227a5ff9929265ab9dd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 56 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } attrs { replace: true } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { promote: true } }
I20240503 01:59:05.944063 4285 tablet_bootstrap.cc:492] T 19b3a90e13194997bde2b9a806f8f2b5 P 4125b1a70e864e129bd54bdc2720d7fa: Bootstrap complete.
I20240503 01:59:05.944154 4349 raft_consensus.cc:729] T a6e17a2c4ba0482e96d04eb9d3ba854a P b63928177fa54227a5ff9929265ab9dd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b63928177fa54227a5ff9929265ab9dd, State: Initialized, Role: LEARNER
I20240503 01:59:05.944324 4349 consensus_queue.cc:260] T a6e17a2c4ba0482e96d04eb9d3ba854a P b63928177fa54227a5ff9929265ab9dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 61, Last appended: 1.61, Last appended by leader: 61, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 56 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } attrs { replace: true } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { promote: true } }
I20240503 01:59:05.944679 4285 ts_tablet_manager.cc:1367] T 19b3a90e13194997bde2b9a806f8f2b5 P 4125b1a70e864e129bd54bdc2720d7fa: Time spent bootstrapping tablet: real 0.920s user 0.073s sys 0.014s
I20240503 01:59:05.945061 4349 ts_tablet_manager.cc:1398] T a6e17a2c4ba0482e96d04eb9d3ba854a P b63928177fa54227a5ff9929265ab9dd: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:05.945613 4285 raft_consensus.cc:348] T 19b3a90e13194997bde2b9a806f8f2b5 P 4125b1a70e864e129bd54bdc2720d7fa [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 65 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } attrs { replace: true } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: true } }
I20240503 01:59:05.945982 4285 raft_consensus.cc:729] T 19b3a90e13194997bde2b9a806f8f2b5 P 4125b1a70e864e129bd54bdc2720d7fa [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4125b1a70e864e129bd54bdc2720d7fa, State: Initialized, Role: LEARNER
I20240503 01:59:05.946166 4285 consensus_queue.cc:260] T 19b3a90e13194997bde2b9a806f8f2b5 P 4125b1a70e864e129bd54bdc2720d7fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 77, Last appended: 1.77, Last appended by leader: 77, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 65 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } attrs { replace: true } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: true } }
I20240503 01:59:05.946276 2944 tablet_copy_service.cc:342] P 327aa26a8e434a609ebcb382aa665fce: Request end of tablet copy session b63928177fa54227a5ff9929265ab9dd-a6e17a2c4ba0482e96d04eb9d3ba854a received from {username='slave'} at 127.0.99.129:33595
I20240503 01:59:05.946350 2944 tablet_copy_service.cc:433] P 327aa26a8e434a609ebcb382aa665fce: ending tablet copy session b63928177fa54227a5ff9929265ab9dd-a6e17a2c4ba0482e96d04eb9d3ba854a on tablet a6e17a2c4ba0482e96d04eb9d3ba854a with peer b63928177fa54227a5ff9929265ab9dd
I20240503 01:59:05.942519 4350 tablet_bootstrap.cc:492] T c10343409a2a423eb0b21759bfc9ba61 P 4125b1a70e864e129bd54bdc2720d7fa: Bootstrap replayed 1/1 log segments. Stats: ops{read=51 overwritten=0 applied=51 ignored=0} inserts{seen=723 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:05.946642 4350 tablet_bootstrap.cc:492] T c10343409a2a423eb0b21759bfc9ba61 P 4125b1a70e864e129bd54bdc2720d7fa: Bootstrap complete.
I20240503 01:59:05.946861 4350 ts_tablet_manager.cc:1367] T c10343409a2a423eb0b21759bfc9ba61 P 4125b1a70e864e129bd54bdc2720d7fa: Time spent bootstrapping tablet: real 0.663s user 0.056s sys 0.007s
I20240503 01:59:05.949193 2396 raft_consensus.cc:1261] T 568a05972208414c90f7f4e2ea2c10a9 P b63928177fa54227a5ff9929265ab9dd [term 1 FOLLOWER]: Refusing update from remote peer b40a9a7b30d34a6db007ed8febd8a759: Log matching property violated. Preceding OpId in replica: term: 1 index: 103. Preceding OpId from leader: term: 1 index: 104. (index mismatch)
I20240503 01:59:05.951907 4350 raft_consensus.cc:348] T c10343409a2a423eb0b21759bfc9ba61 P 4125b1a70e864e129bd54bdc2720d7fa [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 46 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { replace: true } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: true } }
I20240503 01:59:05.952100 4350 raft_consensus.cc:729] T c10343409a2a423eb0b21759bfc9ba61 P 4125b1a70e864e129bd54bdc2720d7fa [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4125b1a70e864e129bd54bdc2720d7fa, State: Initialized, Role: LEARNER
I20240503 01:59:05.952250 4350 consensus_queue.cc:260] T c10343409a2a423eb0b21759bfc9ba61 P 4125b1a70e864e129bd54bdc2720d7fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 51, Last appended: 1.51, Last appended by leader: 51, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 46 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { replace: true } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: true } }
I20240503 01:59:05.954337 4350 ts_tablet_manager.cc:1398] T c10343409a2a423eb0b21759bfc9ba61 P 4125b1a70e864e129bd54bdc2720d7fa: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240503 01:59:05.956439 2420 tablet_copy_service.cc:342] P b63928177fa54227a5ff9929265ab9dd: Request end of tablet copy session 4125b1a70e864e129bd54bdc2720d7fa-c10343409a2a423eb0b21759bfc9ba61 received from {username='slave'} at 127.0.99.135:45173
I20240503 01:59:05.956532 2420 tablet_copy_service.cc:433] P b63928177fa54227a5ff9929265ab9dd: ending tablet copy session 4125b1a70e864e129bd54bdc2720d7fa-c10343409a2a423eb0b21759bfc9ba61 on tablet c10343409a2a423eb0b21759bfc9ba61 with peer 4125b1a70e864e129bd54bdc2720d7fa
I20240503 01:59:05.960672 4335 consensus_queue.cc:1035] T 568a05972208414c90f7f4e2ea2c10a9 P b40a9a7b30d34a6db007ed8febd8a759 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 104, Last known committed idx: 103, Time since last communication: 0.000s
I20240503 01:59:05.962740 2396 raft_consensus.cc:2932] T 568a05972208414c90f7f4e2ea2c10a9 P b63928177fa54227a5ff9929265ab9dd [term 1 FOLLOWER]: Committing config change with OpId 1.104: config changed from index 79 to 104, 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:05.976806 4285 ts_tablet_manager.cc:1398] T 19b3a90e13194997bde2b9a806f8f2b5 P 4125b1a70e864e129bd54bdc2720d7fa: Time spent starting tablet: real 0.032s user 0.001s sys 0.000s
I20240503 01:59:05.979620 2420 tablet_copy_service.cc:342] P b63928177fa54227a5ff9929265ab9dd: Request end of tablet copy session 4125b1a70e864e129bd54bdc2720d7fa-19b3a90e13194997bde2b9a806f8f2b5 received from {username='slave'} at 127.0.99.135:45173
I20240503 01:59:05.979717 2420 tablet_copy_service.cc:433] P b63928177fa54227a5ff9929265ab9dd: ending tablet copy session 4125b1a70e864e129bd54bdc2720d7fa-19b3a90e13194997bde2b9a806f8f2b5 on tablet 19b3a90e13194997bde2b9a806f8f2b5 with peer 4125b1a70e864e129bd54bdc2720d7fa
I20240503 01:59:05.983774 2251 catalog_manager.cc:5484] T 568a05972208414c90f7f4e2ea2c10a9 P 4125b1a70e864e129bd54bdc2720d7fa reported cstate change: config changed from index 79 to 104, 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b40a9a7b30d34a6db007ed8febd8a759" committed_config { opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:05.986689 2397 raft_consensus.cc:1203] T a6e17a2c4ba0482e96d04eb9d3ba854a P b63928177fa54227a5ff9929265ab9dd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.55->[1.56-1.64] Dedup: 1.61->[1.62-1.64]
I20240503 01:59:05.998209 2789 raft_consensus.cc:2932] T 568a05972208414c90f7f4e2ea2c10a9 P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Committing config change with OpId 1.104: config changed from index 79 to 104, 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:06.002068 2924 raft_consensus.cc:2932] T 568a05972208414c90f7f4e2ea2c10a9 P 327aa26a8e434a609ebcb382aa665fce [term 1 FOLLOWER]: Committing config change with OpId 1.104: config changed from index 79 to 104, 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:06.015460 4197 raft_consensus.cc:1050] T 5fd1010447524b6695abb98192dc54f1 P 327aa26a8e434a609ebcb382aa665fce: attempting to promote NON_VOTER a1a844ecd71145e4a362f4c42500fbcb to VOTER
I20240503 01:59:06.017174 4197 consensus_queue.cc:237] T 5fd1010447524b6695abb98192dc54f1 P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 38, Committed index: 38, Last appended: 1.38, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } attrs { promote: false } }
I20240503 01:59:06.019646 3054 raft_consensus.cc:1261] T 5fd1010447524b6695abb98192dc54f1 P 6568f69a78c44bb985162d2b7d0a9376 [term 1 FOLLOWER]: Refusing update from remote peer 327aa26a8e434a609ebcb382aa665fce: Log matching property violated. Preceding OpId in replica: term: 1 index: 38. Preceding OpId from leader: term: 1 index: 39. (index mismatch)
I20240503 01:59:06.020027 4216 consensus_queue.cc:1035] T 5fd1010447524b6695abb98192dc54f1 P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 39, Last known committed idx: 38, Time since last communication: 0.000s
I20240503 01:59:06.022576 2659 raft_consensus.cc:1261] T 5fd1010447524b6695abb98192dc54f1 P b40a9a7b30d34a6db007ed8febd8a759 [term 1 FOLLOWER]: Refusing update from remote peer 327aa26a8e434a609ebcb382aa665fce: Log matching property violated. Preceding OpId in replica: term: 1 index: 38. Preceding OpId from leader: term: 1 index: 39. (index mismatch)
I20240503 01:59:06.023350 4039 consensus_queue.cc:1035] T 5fd1010447524b6695abb98192dc54f1 P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 39, Last known committed idx: 38, Time since last communication: 0.000s
I20240503 01:59:06.032011 2660 consensus_queue.cc:237] T 568a05972208414c90f7f4e2ea2c10a9 P b40a9a7b30d34a6db007ed8febd8a759 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 105, Committed index: 105, Last appended: 1.106, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } }
I20240503 01:59:06.040833 3958 raft_consensus.cc:1261] T 568a05972208414c90f7f4e2ea2c10a9 P 4125b1a70e864e129bd54bdc2720d7fa [term 1 FOLLOWER]: Refusing update from remote peer b40a9a7b30d34a6db007ed8febd8a759: Log matching property violated. Preceding OpId in replica: term: 1 index: 106. Preceding OpId from leader: term: 1 index: 107. (index mismatch)
I20240503 01:59:06.042953 3545 consensus_queue.cc:1035] T 568a05972208414c90f7f4e2ea2c10a9 P b40a9a7b30d34a6db007ed8febd8a759 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 107, Last known committed idx: 105, Time since last communication: 0.000s
I20240503 01:59:06.047349 2531 raft_consensus.cc:1261] T 5fd1010447524b6695abb98192dc54f1 P a1a844ecd71145e4a362f4c42500fbcb [term 1 LEARNER]: Refusing update from remote peer 327aa26a8e434a609ebcb382aa665fce: Log matching property violated. Preceding OpId in replica: term: 1 index: 38. Preceding OpId from leader: term: 1 index: 39. (index mismatch)
I20240503 01:59:06.047958 2792 raft_consensus.cc:1261] T 5fd1010447524b6695abb98192dc54f1 P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Refusing update from remote peer 327aa26a8e434a609ebcb382aa665fce: Log matching property violated. Preceding OpId in replica: term: 1 index: 38. Preceding OpId from leader: term: 1 index: 39. (index mismatch)
I20240503 01:59:06.061388 2394 raft_consensus.cc:1261] T 568a05972208414c90f7f4e2ea2c10a9 P b63928177fa54227a5ff9929265ab9dd [term 1 FOLLOWER]: Refusing update from remote peer b40a9a7b30d34a6db007ed8febd8a759: Log matching property violated. Preceding OpId in replica: term: 1 index: 106. Preceding OpId from leader: term: 1 index: 107. (index mismatch)
I20240503 01:59:06.063938 4335 consensus_queue.cc:1035] T 568a05972208414c90f7f4e2ea2c10a9 P b40a9a7b30d34a6db007ed8febd8a759 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 107, Last known committed idx: 105, Time since last communication: 0.000s
I20240503 01:59:06.068010 3958 raft_consensus.cc:1203] T c10343409a2a423eb0b21759bfc9ba61 P 4125b1a70e864e129bd54bdc2720d7fa [term 1 LEARNER]: Deduplicated request from leader. Original: 1.45->[1.46-1.54] Dedup: 1.51->[1.52-1.54]
I20240503 01:59:06.072837 2394 raft_consensus.cc:1261] T 5fd1010447524b6695abb98192dc54f1 P b63928177fa54227a5ff9929265ab9dd [term 1 FOLLOWER]: Refusing update from remote peer 327aa26a8e434a609ebcb382aa665fce: Log matching property violated. Preceding OpId in replica: term: 1 index: 37. Preceding OpId from leader: term: 1 index: 39. (index mismatch)
I20240503 01:59:06.078771 2791 raft_consensus.cc:1261] T 568a05972208414c90f7f4e2ea2c10a9 P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Refusing update from remote peer b40a9a7b30d34a6db007ed8febd8a759: Log matching property violated. Preceding OpId in replica: term: 1 index: 105. Preceding OpId from leader: term: 1 index: 107. (index mismatch)
I20240503 01:59:06.079352 3545 consensus_queue.cc:1035] T 568a05972208414c90f7f4e2ea2c10a9 P b40a9a7b30d34a6db007ed8febd8a759 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 107, Last known committed idx: 105, Time since last communication: 0.000s
I20240503 01:59:06.081590 4250 raft_consensus.cc:2932] T 568a05972208414c90f7f4e2ea2c10a9 P b40a9a7b30d34a6db007ed8febd8a759 [term 1 LEADER]: Committing config change with OpId 1.107: config changed from index 104 to 107, VOTER a1a844ecd71145e4a362f4c42500fbcb (127.0.99.130) evicted. New config: { opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:06.082561 2787 raft_consensus.cc:2932] T 568a05972208414c90f7f4e2ea2c10a9 P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Committing config change with OpId 1.107: config changed from index 104 to 107, VOTER a1a844ecd71145e4a362f4c42500fbcb (127.0.99.130) evicted. New config: { opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:06.084518 2242 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 568a05972208414c90f7f4e2ea2c10a9 with cas_config_opid_index 104: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:06.085951 3957 raft_consensus.cc:2932] T 568a05972208414c90f7f4e2ea2c10a9 P 4125b1a70e864e129bd54bdc2720d7fa [term 1 FOLLOWER]: Committing config change with OpId 1.107: config changed from index 104 to 107, VOTER a1a844ecd71145e4a362f4c42500fbcb (127.0.99.130) evicted. New config: { opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:06.086899 4216 consensus_queue.cc:1035] T 5fd1010447524b6695abb98192dc54f1 P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 39, Last known committed idx: 37, Time since last communication: 0.000s
I20240503 01:59:06.090086 2251 catalog_manager.cc:5484] T 568a05972208414c90f7f4e2ea2c10a9 P 4125b1a70e864e129bd54bdc2720d7fa reported cstate change: config changed from index 104 to 107, VOTER a1a844ecd71145e4a362f4c42500fbcb (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "b40a9a7b30d34a6db007ed8febd8a759" committed_config { opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:06.093220 2510 tablet_service.cc:1514] Processing DeleteTablet for tablet 568a05972208414c90f7f4e2ea2c10a9 with delete_type TABLET_DATA_TOMBSTONED (TS a1a844ecd71145e4a362f4c42500fbcb not found in new config with opid_index 107) from {username='slave'} at 127.0.0.1:59126
I20240503 01:59:06.102973 2921 raft_consensus.cc:1261] T 568a05972208414c90f7f4e2ea2c10a9 P 327aa26a8e434a609ebcb382aa665fce [term 1 FOLLOWER]: Refusing update from remote peer b40a9a7b30d34a6db007ed8febd8a759: Log matching property violated. Preceding OpId in replica: term: 1 index: 105. Preceding OpId from leader: term: 1 index: 107. (index mismatch)
I20240503 01:59:06.103837 4250 consensus_queue.cc:1035] T 568a05972208414c90f7f4e2ea2c10a9 P b40a9a7b30d34a6db007ed8febd8a759 [LEADER]: Connected to new peer: Peer: permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 107, Last known committed idx: 105, Time since last communication: 0.000s
I20240503 01:59:06.109292 2924 raft_consensus.cc:2932] T 568a05972208414c90f7f4e2ea2c10a9 P 327aa26a8e434a609ebcb382aa665fce [term 1 FOLLOWER]: Committing config change with OpId 1.107: config changed from index 104 to 107, VOTER a1a844ecd71145e4a362f4c42500fbcb (127.0.99.130) evicted. New config: { opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:06.110213 4216 consensus_queue.cc:1035] T 5fd1010447524b6695abb98192dc54f1 P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 39, Last known committed idx: 37, Time since last communication: 0.000s
I20240503 01:59:06.135613 4432 tablet_replica.cc:330] T 568a05972208414c90f7f4e2ea2c10a9 P a1a844ecd71145e4a362f4c42500fbcb: stopping tablet replica
I20240503 01:59:06.140875 4432 raft_consensus.cc:2229] T 568a05972208414c90f7f4e2ea2c10a9 P a1a844ecd71145e4a362f4c42500fbcb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:06.141235 4432 pending_rounds.cc:62] T 568a05972208414c90f7f4e2ea2c10a9 P a1a844ecd71145e4a362f4c42500fbcb: Trying to abort 1 pending ops.
I20240503 01:59:06.141319 4432 pending_rounds.cc:68] T 568a05972208414c90f7f4e2ea2c10a9 P a1a844ecd71145e4a362f4c42500fbcb: Aborting op as it isn't in flight: id { term: 1 index: 106 } timestamp: 7023417532374056960 op_type: WRITE_OP write_request { tablet_id: "568a05972208414c90f7f4e2ea2c10a9" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\270\225\223<\222w\243\246\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""410ddf3deb767e51" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023417532080721920 authz_token { token_data: "\010\221\214\321\261\006\"3\n\005slave\022*\n 2de29504ce33484ba12dd9985ab742a9\020\001\030\001 \001(\001" signature: "<redacted>""\024\264\251Q\231\371%s\361I\035\220\355\332\234\352\004T\024\300\332=\004\331\302\020\3459\254\251d\211\204\316n\345\223M\020r\224\362,1G\324\264\250\357[Y\310\211t\356\002e9\323\364\341_\2447" signing_key_seq_num: 0 } } request_id { client_id: "f06eac613c694c3aaf48086938797abc" seq_no: 273 first_incomplete_seq_no: 273 attempt_no: 0 }
I20240503 01:59:06.141649 4432 raft_consensus.cc:2258] T 568a05972208414c90f7f4e2ea2c10a9 P a1a844ecd71145e4a362f4c42500fbcb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:06.143551 4432 ts_tablet_manager.cc:1875] T 568a05972208414c90f7f4e2ea2c10a9 P a1a844ecd71145e4a362f4c42500fbcb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:06.148182 4098 consensus_queue.cc:1035] T 5fd1010447524b6695abb98192dc54f1 P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 39, Last known committed idx: 38, Time since last communication: 0.000s
I20240503 01:59:06.150687 2396 raft_consensus.cc:2932] T 568a05972208414c90f7f4e2ea2c10a9 P b63928177fa54227a5ff9929265ab9dd [term 1 FOLLOWER]: Committing config change with OpId 1.107: config changed from index 104 to 107, VOTER a1a844ecd71145e4a362f4c42500fbcb (127.0.99.130) evicted. New config: { opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:06.157055 4216 raft_consensus.cc:2932] T 5fd1010447524b6695abb98192dc54f1 P 327aa26a8e434a609ebcb382aa665fce [term 1 LEADER]: Committing config change with OpId 1.39: config changed from index 25 to 39, a1a844ecd71145e4a362f4c42500fbcb (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } attrs { promote: false } } }
I20240503 01:59:06.159431 3055 raft_consensus.cc:2932] T 5fd1010447524b6695abb98192dc54f1 P 6568f69a78c44bb985162d2b7d0a9376 [term 1 FOLLOWER]: Committing config change with OpId 1.39: config changed from index 25 to 39, a1a844ecd71145e4a362f4c42500fbcb (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } attrs { promote: false } } }
I20240503 01:59:06.159824 2661 raft_consensus.cc:2932] T 5fd1010447524b6695abb98192dc54f1 P b40a9a7b30d34a6db007ed8febd8a759 [term 1 FOLLOWER]: Committing config change with OpId 1.39: config changed from index 25 to 39, a1a844ecd71145e4a362f4c42500fbcb (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } attrs { promote: false } } }
I20240503 01:59:06.163081 4432 ts_tablet_manager.cc:1888] T 568a05972208414c90f7f4e2ea2c10a9 P a1a844ecd71145e4a362f4c42500fbcb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.106
I20240503 01:59:06.163195 4432 log.cc:1196] T 568a05972208414c90f7f4e2ea2c10a9 P a1a844ecd71145e4a362f4c42500fbcb: Deleting WAL directory at /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/568a05972208414c90f7f4e2ea2c10a9
I20240503 01:59:06.163762 2245 catalog_manager.cc:4830] TS a1a844ecd71145e4a362f4c42500fbcb (127.0.99.130:46033): tablet 568a05972208414c90f7f4e2ea2c10a9 (table rebalance_test_table_0 [id=2de29504ce33484ba12dd9985ab742a9]) successfully deleted
I20240503 01:59:06.167102 2253 catalog_manager.cc:5484] T 5fd1010447524b6695abb98192dc54f1 P b40a9a7b30d34a6db007ed8febd8a759 reported cstate change: config changed from index 25 to 39, a1a844ecd71145e4a362f4c42500fbcb (127.0.99.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "327aa26a8e434a609ebcb382aa665fce" committed_config { opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } attrs { promote: false } } }
I20240503 01:59:06.180946 2394 raft_consensus.cc:2932] T 5fd1010447524b6695abb98192dc54f1 P b63928177fa54227a5ff9929265ab9dd [term 1 FOLLOWER]: Committing config change with OpId 1.39: config changed from index 25 to 39, a1a844ecd71145e4a362f4c42500fbcb (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } attrs { promote: false } } }
I20240503 01:59:06.209455 2792 raft_consensus.cc:2932] T 5fd1010447524b6695abb98192dc54f1 P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Committing config change with OpId 1.39: config changed from index 25 to 39, a1a844ecd71145e4a362f4c42500fbcb (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } attrs { promote: false } } }
I20240503 01:59:06.217305 2531 raft_consensus.cc:2932] T 5fd1010447524b6695abb98192dc54f1 P a1a844ecd71145e4a362f4c42500fbcb [term 1 FOLLOWER]: Committing config change with OpId 1.39: config changed from index 25 to 39, a1a844ecd71145e4a362f4c42500fbcb (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } attrs { promote: false } } }
I20240503 01:59:06.227403 4410 tablet_bootstrap.cc:492] T d4d07c9621134780bc7ece7bb061d934 P 4125b1a70e864e129bd54bdc2720d7fa: Bootstrap replayed 1/1 log segments. Stats: ops{read=69 overwritten=0 applied=69 ignored=0} inserts{seen=700 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:06.227733 4410 tablet_bootstrap.cc:492] T d4d07c9621134780bc7ece7bb061d934 P 4125b1a70e864e129bd54bdc2720d7fa: Bootstrap complete.
I20240503 01:59:06.227926 4410 ts_tablet_manager.cc:1367] T d4d07c9621134780bc7ece7bb061d934 P 4125b1a70e864e129bd54bdc2720d7fa: Time spent bootstrapping tablet: real 0.414s user 0.079s sys 0.002s
I20240503 01:59:06.229408 4410 raft_consensus.cc:348] T d4d07c9621134780bc7ece7bb061d934 P 4125b1a70e864e129bd54bdc2720d7fa [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 55 OBSOLETE_local: false peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } attrs { replace: true } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: true } }
I20240503 01:59:06.233973 4410 raft_consensus.cc:729] T d4d07c9621134780bc7ece7bb061d934 P 4125b1a70e864e129bd54bdc2720d7fa [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4125b1a70e864e129bd54bdc2720d7fa, State: Initialized, Role: LEARNER
I20240503 01:59:06.234558 4410 consensus_queue.cc:260] T d4d07c9621134780bc7ece7bb061d934 P 4125b1a70e864e129bd54bdc2720d7fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 69, Last appended: 1.69, Last appended by leader: 69, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 55 OBSOLETE_local: false peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } attrs { replace: true } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: true } }
I20240503 01:59:06.235028 4410 ts_tablet_manager.cc:1398] T d4d07c9621134780bc7ece7bb061d934 P 4125b1a70e864e129bd54bdc2720d7fa: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240503 01:59:06.239338 2944 tablet_copy_service.cc:342] P 327aa26a8e434a609ebcb382aa665fce: Request end of tablet copy session 4125b1a70e864e129bd54bdc2720d7fa-d4d07c9621134780bc7ece7bb061d934 received from {username='slave'} at 127.0.99.135:39289
I20240503 01:59:06.239429 2944 tablet_copy_service.cc:433] P 327aa26a8e434a609ebcb382aa665fce: ending tablet copy session 4125b1a70e864e129bd54bdc2720d7fa-d4d07c9621134780bc7ece7bb061d934 on tablet d4d07c9621134780bc7ece7bb061d934 with peer 4125b1a70e864e129bd54bdc2720d7fa
I20240503 01:59:06.314476 3955 raft_consensus.cc:1203] T 19b3a90e13194997bde2b9a806f8f2b5 P 4125b1a70e864e129bd54bdc2720d7fa [term 1 LEARNER]: Deduplicated request from leader. Original: 1.64->[1.65-1.100] Dedup: 1.77->[1.78-1.100]
I20240503 01:59:06.315135 2920 consensus_queue.cc:237] T 5fd1010447524b6695abb98192dc54f1 P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 40, Committed index: 40, Last appended: 1.41, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 42 OBSOLETE_local: false peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } attrs { promote: false } }
I20240503 01:59:06.316926 3054 raft_consensus.cc:1261] T 5fd1010447524b6695abb98192dc54f1 P 6568f69a78c44bb985162d2b7d0a9376 [term 1 FOLLOWER]: Refusing update from remote peer 327aa26a8e434a609ebcb382aa665fce: Log matching property violated. Preceding OpId in replica: term: 1 index: 41. Preceding OpId from leader: term: 1 index: 42. (index mismatch)
I20240503 01:59:06.317373 3870 consensus_queue.cc:1035] T 5fd1010447524b6695abb98192dc54f1 P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 42, Last known committed idx: 40, Time since last communication: 0.000s
I20240503 01:59:06.330904 4332 raft_consensus.cc:1050] T c10343409a2a423eb0b21759bfc9ba61 P b63928177fa54227a5ff9929265ab9dd: attempting to promote NON_VOTER 4125b1a70e864e129bd54bdc2720d7fa to VOTER
I20240503 01:59:06.331418 4332 consensus_queue.cc:237] T c10343409a2a423eb0b21759bfc9ba61 P b63928177fa54227a5ff9929265ab9dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 55, Committed index: 55, Last appended: 1.55, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 56 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { replace: true } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } }
I20240503 01:59:06.334026 2921 raft_consensus.cc:1261] T c10343409a2a423eb0b21759bfc9ba61 P 327aa26a8e434a609ebcb382aa665fce [term 1 FOLLOWER]: Refusing update from remote peer b63928177fa54227a5ff9929265ab9dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 55. Preceding OpId from leader: term: 1 index: 56. (index mismatch)
I20240503 01:59:06.334232 2790 raft_consensus.cc:1261] T c10343409a2a423eb0b21759bfc9ba61 P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Refusing update from remote peer b63928177fa54227a5ff9929265ab9dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 55. Preceding OpId from leader: term: 1 index: 56. (index mismatch)
I20240503 01:59:06.334415 3802 consensus_queue.cc:1035] T c10343409a2a423eb0b21759bfc9ba61 P b63928177fa54227a5ff9929265ab9dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 56, Last known committed idx: 54, Time since last communication: 0.000s
I20240503 01:59:06.336956 4132 consensus_queue.cc:1035] T c10343409a2a423eb0b21759bfc9ba61 P b63928177fa54227a5ff9929265ab9dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 56, Last known committed idx: 54, Time since last communication: 0.000s
I20240503 01:59:06.337284 3957 raft_consensus.cc:1261] T c10343409a2a423eb0b21759bfc9ba61 P 4125b1a70e864e129bd54bdc2720d7fa [term 1 LEARNER]: Refusing update from remote peer b63928177fa54227a5ff9929265ab9dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 55. Preceding OpId from leader: term: 1 index: 56. (index mismatch)
I20240503 01:59:06.339939 4132 consensus_queue.cc:1035] T c10343409a2a423eb0b21759bfc9ba61 P b63928177fa54227a5ff9929265ab9dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 56, Last known committed idx: 54, Time since last communication: 0.000s
I20240503 01:59:06.340498 2530 raft_consensus.cc:1261] T c10343409a2a423eb0b21759bfc9ba61 P a1a844ecd71145e4a362f4c42500fbcb [term 1 FOLLOWER]: Refusing update from remote peer b63928177fa54227a5ff9929265ab9dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 55. Preceding OpId from leader: term: 1 index: 56. (index mismatch)
I20240503 01:59:06.348872 4132 consensus_queue.cc:1035] T c10343409a2a423eb0b21759bfc9ba61 P b63928177fa54227a5ff9929265ab9dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 56, Last known committed idx: 54, Time since last communication: 0.000s
I20240503 01:59:06.349640 2662 raft_consensus.cc:1261] T c10343409a2a423eb0b21759bfc9ba61 P b40a9a7b30d34a6db007ed8febd8a759 [term 1 FOLLOWER]: Refusing update from remote peer b63928177fa54227a5ff9929265ab9dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 55. Preceding OpId from leader: term: 1 index: 56. (index mismatch)
I20240503 01:59:06.350903 2530 raft_consensus.cc:1261] T 5fd1010447524b6695abb98192dc54f1 P a1a844ecd71145e4a362f4c42500fbcb [term 1 FOLLOWER]: Refusing update from remote peer 327aa26a8e434a609ebcb382aa665fce: Log matching property violated. Preceding OpId in replica: term: 1 index: 41. Preceding OpId from leader: term: 1 index: 42. (index mismatch)
I20240503 01:59:06.353585 4439 consensus_queue.cc:1035] T c10343409a2a423eb0b21759bfc9ba61 P b63928177fa54227a5ff9929265ab9dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 56, Last known committed idx: 55, Time since last communication: 0.000s
I20240503 01:59:06.362475 2789 raft_consensus.cc:1261] T 5fd1010447524b6695abb98192dc54f1 P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Refusing update from remote peer 327aa26a8e434a609ebcb382aa665fce: Log matching property violated. Preceding OpId in replica: term: 1 index: 41. Preceding OpId from leader: term: 1 index: 42. (index mismatch)
I20240503 01:59:06.363138 3551 consensus_queue.cc:1035] T 5fd1010447524b6695abb98192dc54f1 P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 42, Last known committed idx: 40, Time since last communication: 0.000s
I20240503 01:59:06.363328 4437 raft_consensus.cc:2932] T c10343409a2a423eb0b21759bfc9ba61 P b63928177fa54227a5ff9929265ab9dd [term 1 LEADER]: Committing config change with OpId 1.56: config changed from index 46 to 56, 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 56 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { replace: true } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:06.364023 2662 raft_consensus.cc:2932] T c10343409a2a423eb0b21759bfc9ba61 P b40a9a7b30d34a6db007ed8febd8a759 [term 1 FOLLOWER]: Committing config change with OpId 1.56: config changed from index 46 to 56, 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 56 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { replace: true } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:06.364197 2397 raft_consensus.cc:1261] T 5fd1010447524b6695abb98192dc54f1 P b63928177fa54227a5ff9929265ab9dd [term 1 FOLLOWER]: Refusing update from remote peer 327aa26a8e434a609ebcb382aa665fce: Log matching property violated. Preceding OpId in replica: term: 1 index: 41. Preceding OpId from leader: term: 1 index: 42. (index mismatch)
I20240503 01:59:06.367630 3551 consensus_queue.cc:1035] T 5fd1010447524b6695abb98192dc54f1 P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 42, Last known committed idx: 40, Time since last communication: 0.000s
I20240503 01:59:06.368726 2924 raft_consensus.cc:2932] T c10343409a2a423eb0b21759bfc9ba61 P 327aa26a8e434a609ebcb382aa665fce [term 1 FOLLOWER]: Committing config change with OpId 1.56: config changed from index 46 to 56, 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 56 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { replace: true } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:06.370646 2253 catalog_manager.cc:5484] T c10343409a2a423eb0b21759bfc9ba61 P b40a9a7b30d34a6db007ed8febd8a759 reported cstate change: config changed from index 46 to 56, 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b63928177fa54227a5ff9929265ab9dd" committed_config { opid_index: 56 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { replace: true } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:06.378240 3870 raft_consensus.cc:1050] T a6e17a2c4ba0482e96d04eb9d3ba854a P 327aa26a8e434a609ebcb382aa665fce: attempting to promote NON_VOTER b63928177fa54227a5ff9929265ab9dd to VOTER
I20240503 01:59:06.378861 3870 consensus_queue.cc:237] T a6e17a2c4ba0482e96d04eb9d3ba854a P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 65, Committed index: 65, Last appended: 1.65, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } attrs { replace: true } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { promote: false } }
I20240503 01:59:06.380292 3055 raft_consensus.cc:1261] T a6e17a2c4ba0482e96d04eb9d3ba854a P 6568f69a78c44bb985162d2b7d0a9376 [term 1 FOLLOWER]: Refusing update from remote peer 327aa26a8e434a609ebcb382aa665fce: Log matching property violated. Preceding OpId in replica: term: 1 index: 65. Preceding OpId from leader: term: 1 index: 66. (index mismatch)
I20240503 01:59:06.380707 3551 consensus_queue.cc:1035] T a6e17a2c4ba0482e96d04eb9d3ba854a P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 66, Last known committed idx: 65, Time since last communication: 0.000s
I20240503 01:59:06.382238 3551 consensus_queue.cc:1035] T 5fd1010447524b6695abb98192dc54f1 P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 42, Last known committed idx: 40, Time since last communication: 0.000s
I20240503 01:59:06.385049 3957 raft_consensus.cc:2932] T c10343409a2a423eb0b21759bfc9ba61 P 4125b1a70e864e129bd54bdc2720d7fa [term 1 FOLLOWER]: Committing config change with OpId 1.56: config changed from index 46 to 56, 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 56 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { replace: true } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:06.386646 2787 raft_consensus.cc:2932] T c10343409a2a423eb0b21759bfc9ba61 P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Committing config change with OpId 1.56: config changed from index 46 to 56, 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 56 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { replace: true } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:06.387590 2524 raft_consensus.cc:2932] T c10343409a2a423eb0b21759bfc9ba61 P a1a844ecd71145e4a362f4c42500fbcb [term 1 FOLLOWER]: Committing config change with OpId 1.56: config changed from index 46 to 56, 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 56 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { replace: true } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:06.393342 2662 raft_consensus.cc:1261] T a6e17a2c4ba0482e96d04eb9d3ba854a P b40a9a7b30d34a6db007ed8febd8a759 [term 1 FOLLOWER]: Refusing update from remote peer 327aa26a8e434a609ebcb382aa665fce: Log matching property violated. Preceding OpId in replica: term: 1 index: 65. Preceding OpId from leader: term: 1 index: 66. (index mismatch)
I20240503 01:59:06.394003 3551 consensus_queue.cc:1035] T a6e17a2c4ba0482e96d04eb9d3ba854a P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 66, Last known committed idx: 65, Time since last communication: 0.000s
I20240503 01:59:06.420019 2397 consensus_queue.cc:237] T c10343409a2a423eb0b21759bfc9ba61 P b63928177fa54227a5ff9929265ab9dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 56, Committed index: 56, Last appended: 1.56, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 57 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } }
I20240503 01:59:06.423677 2790 raft_consensus.cc:1261] T c10343409a2a423eb0b21759bfc9ba61 P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Refusing update from remote peer b63928177fa54227a5ff9929265ab9dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 56. Preceding OpId from leader: term: 1 index: 57. (index mismatch)
I20240503 01:59:06.424010 4438 consensus_queue.cc:1035] T c10343409a2a423eb0b21759bfc9ba61 P b63928177fa54227a5ff9929265ab9dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 57, Last known committed idx: 56, Time since last communication: 0.000s
I20240503 01:59:06.424258 3958 raft_consensus.cc:1261] T c10343409a2a423eb0b21759bfc9ba61 P 4125b1a70e864e129bd54bdc2720d7fa [term 1 FOLLOWER]: Refusing update from remote peer b63928177fa54227a5ff9929265ab9dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 56. Preceding OpId from leader: term: 1 index: 57. (index mismatch)
I20240503 01:59:06.424674 4438 consensus_queue.cc:1035] T c10343409a2a423eb0b21759bfc9ba61 P b63928177fa54227a5ff9929265ab9dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 57, Last known committed idx: 56, Time since last communication: 0.000s
I20240503 01:59:06.443276 2660 raft_consensus.cc:1261] T c10343409a2a423eb0b21759bfc9ba61 P b40a9a7b30d34a6db007ed8febd8a759 [term 1 FOLLOWER]: Refusing update from remote peer b63928177fa54227a5ff9929265ab9dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 56. Preceding OpId from leader: term: 1 index: 57. (index mismatch)
I20240503 01:59:06.444305 4439 consensus_queue.cc:1035] T c10343409a2a423eb0b21759bfc9ba61 P b63928177fa54227a5ff9929265ab9dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 57, Last known committed idx: 56, Time since last communication: 0.000s
I20240503 01:59:06.453624 2531 raft_consensus.cc:1261] T c10343409a2a423eb0b21759bfc9ba61 P a1a844ecd71145e4a362f4c42500fbcb [term 1 FOLLOWER]: Refusing update from remote peer b63928177fa54227a5ff9929265ab9dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 56. Preceding OpId from leader: term: 1 index: 57. (index mismatch)
I20240503 01:59:06.454969 4332 consensus_queue.cc:1035] T c10343409a2a423eb0b21759bfc9ba61 P b63928177fa54227a5ff9929265ab9dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 57, Last known committed idx: 56, Time since last communication: 0.000s
I20240503 01:59:06.456272 4332 raft_consensus.cc:2932] T c10343409a2a423eb0b21759bfc9ba61 P b63928177fa54227a5ff9929265ab9dd [term 1 LEADER]: Committing config change with OpId 1.57: config changed from index 56 to 57, VOTER 327aa26a8e434a609ebcb382aa665fce (127.0.99.133) evicted. New config: { opid_index: 57 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:06.463200 2531 raft_consensus.cc:1261] T a6e17a2c4ba0482e96d04eb9d3ba854a P a1a844ecd71145e4a362f4c42500fbcb [term 1 FOLLOWER]: Refusing update from remote peer 327aa26a8e434a609ebcb382aa665fce: Log matching property violated. Preceding OpId in replica: term: 1 index: 65. Preceding OpId from leader: term: 1 index: 66. (index mismatch)
I20240503 01:59:06.464140 2787 raft_consensus.cc:1261] T a6e17a2c4ba0482e96d04eb9d3ba854a P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Refusing update from remote peer 327aa26a8e434a609ebcb382aa665fce: Log matching property violated. Preceding OpId in replica: term: 1 index: 65. Preceding OpId from leader: term: 1 index: 66. (index mismatch)
I20240503 01:59:06.464092 2789 raft_consensus.cc:2932] T c10343409a2a423eb0b21759bfc9ba61 P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Committing config change with OpId 1.57: config changed from index 56 to 57, VOTER 327aa26a8e434a609ebcb382aa665fce (127.0.99.133) evicted. New config: { opid_index: 57 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:06.466562 4197 raft_consensus.cc:2932] T 5fd1010447524b6695abb98192dc54f1 P 327aa26a8e434a609ebcb382aa665fce [term 1 LEADER]: Committing config change with OpId 1.42: config changed from index 39 to 42, VOTER b40a9a7b30d34a6db007ed8febd8a759 (127.0.99.131) evicted. New config: { opid_index: 42 OBSOLETE_local: false peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } attrs { promote: false } } }
I20240503 01:59:06.469409 3054 raft_consensus.cc:2932] T 5fd1010447524b6695abb98192dc54f1 P 6568f69a78c44bb985162d2b7d0a9376 [term 1 FOLLOWER]: Committing config change with OpId 1.42: config changed from index 39 to 42, VOTER b40a9a7b30d34a6db007ed8febd8a759 (127.0.99.131) evicted. New config: { opid_index: 42 OBSOLETE_local: false peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } attrs { promote: false } } }
I20240503 01:59:06.471562 2243 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5fd1010447524b6695abb98192dc54f1 with cas_config_opid_index 39: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:06.474967 2253 catalog_manager.cc:5484] T c10343409a2a423eb0b21759bfc9ba61 P 52c196c15e3348b985c689652bc49918 reported cstate change: config changed from index 56 to 57, VOTER 327aa26a8e434a609ebcb382aa665fce (127.0.99.133) evicted. New cstate: current_term: 1 leader_uuid: "b63928177fa54227a5ff9929265ab9dd" committed_config { opid_index: 57 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:06.475069 4214 consensus_queue.cc:1035] T a6e17a2c4ba0482e96d04eb9d3ba854a P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 66, Last known committed idx: 65, Time since last communication: 0.000s
I20240503 01:59:06.476053 2245 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c10343409a2a423eb0b21759bfc9ba61 with cas_config_opid_index 56: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:06.476615 2250 catalog_manager.cc:5484] T 5fd1010447524b6695abb98192dc54f1 P 6568f69a78c44bb985162d2b7d0a9376 reported cstate change: config changed from index 39 to 42, VOTER b40a9a7b30d34a6db007ed8febd8a759 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "327aa26a8e434a609ebcb382aa665fce" committed_config { opid_index: 42 OBSOLETE_local: false peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } attrs { promote: false } } }
I20240503 01:59:06.477066 4039 consensus_queue.cc:1035] T a6e17a2c4ba0482e96d04eb9d3ba854a P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 66, Last known committed idx: 65, Time since last communication: 0.000s
I20240503 01:59:06.477545 2396 raft_consensus.cc:1261] T a6e17a2c4ba0482e96d04eb9d3ba854a P b63928177fa54227a5ff9929265ab9dd [term 1 LEARNER]: Refusing update from remote peer 327aa26a8e434a609ebcb382aa665fce: Log matching property violated. Preceding OpId in replica: term: 1 index: 65. Preceding OpId from leader: term: 1 index: 66. (index mismatch)
I20240503 01:59:06.479467 2641 tablet_service.cc:1514] Processing DeleteTablet for tablet 5fd1010447524b6695abb98192dc54f1 with delete_type TABLET_DATA_TOMBSTONED (TS b40a9a7b30d34a6db007ed8febd8a759 not found in new config with opid_index 42) from {username='slave'} at 127.0.0.1:40954
I20240503 01:59:06.480317 3958 raft_consensus.cc:2932] T c10343409a2a423eb0b21759bfc9ba61 P 4125b1a70e864e129bd54bdc2720d7fa [term 1 FOLLOWER]: Committing config change with OpId 1.57: config changed from index 56 to 57, VOTER 327aa26a8e434a609ebcb382aa665fce (127.0.99.133) evicted. New config: { opid_index: 57 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:06.484351 2904 tablet_service.cc:1514] Processing DeleteTablet for tablet c10343409a2a423eb0b21759bfc9ba61 with delete_type TABLET_DATA_TOMBSTONED (TS 327aa26a8e434a609ebcb382aa665fce not found in new config with opid_index 57) from {username='slave'} at 127.0.0.1:60782
I20240503 01:59:06.488179 2791 raft_consensus.cc:2932] T 5fd1010447524b6695abb98192dc54f1 P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Committing config change with OpId 1.42: config changed from index 39 to 42, VOTER b40a9a7b30d34a6db007ed8febd8a759 (127.0.99.131) evicted. New config: { opid_index: 42 OBSOLETE_local: false peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } attrs { promote: false } } }
I20240503 01:59:06.490104 2531 raft_consensus.cc:2932] T 5fd1010447524b6695abb98192dc54f1 P a1a844ecd71145e4a362f4c42500fbcb [term 1 FOLLOWER]: Committing config change with OpId 1.42: config changed from index 39 to 42, VOTER b40a9a7b30d34a6db007ed8febd8a759 (127.0.99.131) evicted. New config: { opid_index: 42 OBSOLETE_local: false peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } attrs { promote: false } } }
I20240503 01:59:06.492408 4458 tablet_replica.cc:330] T 5fd1010447524b6695abb98192dc54f1 P b40a9a7b30d34a6db007ed8febd8a759: stopping tablet replica
I20240503 01:59:06.493384 4458 raft_consensus.cc:2229] T 5fd1010447524b6695abb98192dc54f1 P b40a9a7b30d34a6db007ed8febd8a759 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:06.493549 4458 pending_rounds.cc:62] T 5fd1010447524b6695abb98192dc54f1 P b40a9a7b30d34a6db007ed8febd8a759: Trying to abort 1 pending ops.
I20240503 01:59:06.493635 4458 pending_rounds.cc:68] T 5fd1010447524b6695abb98192dc54f1 P b40a9a7b30d34a6db007ed8febd8a759: Aborting op as it isn't in flight: id { term: 1 index: 41 } timestamp: 7023417533600866304 op_type: WRITE_OP write_request { tablet_id: "5fd1010447524b6695abb98192dc54f1" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000 \014\255T\362\003\213\031\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""bd35d5beb2c7da55" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023417533565366272 authz_token { token_data: "\010\221\214\321\261\006\"3\n\005slave\022*\n f7b0d84dbe0541aaa2606d6e366719c8\020\001\030\001 \001(\001" signature: "<redacted>""\217jtB\3732\304\336xK^\n$\214\"\256\361\355\365/@\211\346\020k^2\312+\226F\356|\322C\220\335P\030\202\245Deb0\022\360\205\323\034\367\020\343\200}\236-O.UT\307\231\035" signing_key_seq_num: 0 } } request_id { client_id: "bb6121f1bc824d9ab40bab73a1bea25e" seq_no: 129 first_incomplete_seq_no: 129 attempt_no: 0 }
I20240503 01:59:06.494031 4458 raft_consensus.cc:2258] T 5fd1010447524b6695abb98192dc54f1 P b40a9a7b30d34a6db007ed8febd8a759 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:06.495412 4459 tablet_replica.cc:330] T c10343409a2a423eb0b21759bfc9ba61 P 327aa26a8e434a609ebcb382aa665fce: stopping tablet replica
I20240503 01:59:06.496089 4458 ts_tablet_manager.cc:1875] T 5fd1010447524b6695abb98192dc54f1 P b40a9a7b30d34a6db007ed8febd8a759: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:06.496558 4459 raft_consensus.cc:2229] T c10343409a2a423eb0b21759bfc9ba61 P 327aa26a8e434a609ebcb382aa665fce [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:06.498975 4459 raft_consensus.cc:2258] T c10343409a2a423eb0b21759bfc9ba61 P 327aa26a8e434a609ebcb382aa665fce [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:06.501444 3551 raft_consensus.cc:2932] T a6e17a2c4ba0482e96d04eb9d3ba854a P 327aa26a8e434a609ebcb382aa665fce [term 1 LEADER]: Committing config change with OpId 1.66: config changed from index 56 to 66, b63928177fa54227a5ff9929265ab9dd (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } attrs { replace: true } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { promote: false } } }
I20240503 01:59:06.503787 2662 raft_consensus.cc:2932] T c10343409a2a423eb0b21759bfc9ba61 P b40a9a7b30d34a6db007ed8febd8a759 [term 1 FOLLOWER]: Committing config change with OpId 1.57: config changed from index 56 to 57, VOTER 327aa26a8e434a609ebcb382aa665fce (127.0.99.133) evicted. New config: { opid_index: 57 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:06.505833 3055 raft_consensus.cc:2932] T a6e17a2c4ba0482e96d04eb9d3ba854a P 6568f69a78c44bb985162d2b7d0a9376 [term 1 FOLLOWER]: Committing config change with OpId 1.66: config changed from index 56 to 66, b63928177fa54227a5ff9929265ab9dd (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } attrs { replace: true } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { promote: false } } }
I20240503 01:59:06.507486 4459 ts_tablet_manager.cc:1875] T c10343409a2a423eb0b21759bfc9ba61 P 327aa26a8e434a609ebcb382aa665fce: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:06.506708 2660 raft_consensus.cc:2932] T a6e17a2c4ba0482e96d04eb9d3ba854a P b40a9a7b30d34a6db007ed8febd8a759 [term 1 FOLLOWER]: Committing config change with OpId 1.66: config changed from index 56 to 66, b63928177fa54227a5ff9929265ab9dd (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } attrs { replace: true } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { promote: false } } }
I20240503 01:59:06.513607 4216 consensus_queue.cc:1035] T a6e17a2c4ba0482e96d04eb9d3ba854a P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 66, Last known committed idx: 64, Time since last communication: 0.000s
I20240503 01:59:06.517001 2250 catalog_manager.cc:5484] T a6e17a2c4ba0482e96d04eb9d3ba854a P 6568f69a78c44bb985162d2b7d0a9376 reported cstate change: config changed from index 56 to 66, b63928177fa54227a5ff9929265ab9dd (127.0.99.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "327aa26a8e434a609ebcb382aa665fce" committed_config { opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } attrs { replace: true } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { promote: false } } }
I20240503 01:59:06.518399 2530 raft_consensus.cc:2932] T c10343409a2a423eb0b21759bfc9ba61 P a1a844ecd71145e4a362f4c42500fbcb [term 1 FOLLOWER]: Committing config change with OpId 1.57: config changed from index 56 to 57, VOTER 327aa26a8e434a609ebcb382aa665fce (127.0.99.133) evicted. New config: { opid_index: 57 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:06.531395 2789 raft_consensus.cc:2932] T a6e17a2c4ba0482e96d04eb9d3ba854a P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Committing config change with OpId 1.66: config changed from index 56 to 66, b63928177fa54227a5ff9929265ab9dd (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } attrs { replace: true } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { promote: false } } }
I20240503 01:59:06.539140 2530 raft_consensus.cc:2932] T a6e17a2c4ba0482e96d04eb9d3ba854a P a1a844ecd71145e4a362f4c42500fbcb [term 1 FOLLOWER]: Committing config change with OpId 1.66: config changed from index 56 to 66, b63928177fa54227a5ff9929265ab9dd (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } attrs { replace: true } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { promote: false } } }
I20240503 01:59:06.542914 4458 ts_tablet_manager.cc:1888] T 5fd1010447524b6695abb98192dc54f1 P b40a9a7b30d34a6db007ed8febd8a759: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.41
I20240503 01:59:06.544382 4458 log.cc:1196] T 5fd1010447524b6695abb98192dc54f1 P b40a9a7b30d34a6db007ed8febd8a759: Deleting WAL directory at /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/5fd1010447524b6695abb98192dc54f1
I20240503 01:59:06.545403 2242 catalog_manager.cc:4830] TS b40a9a7b30d34a6db007ed8febd8a759 (127.0.99.131:34079): tablet 5fd1010447524b6695abb98192dc54f1 (table rebalance_test_table_3 [id=f7b0d84dbe0541aaa2606d6e366719c8]) successfully deleted
I20240503 01:59:06.547569 4459 ts_tablet_manager.cc:1888] T c10343409a2a423eb0b21759bfc9ba61 P 327aa26a8e434a609ebcb382aa665fce: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.56
I20240503 01:59:06.547725 4459 log.cc:1196] T c10343409a2a423eb0b21759bfc9ba61 P 327aa26a8e434a609ebcb382aa665fce: Deleting WAL directory at /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/c10343409a2a423eb0b21759bfc9ba61
I20240503 01:59:06.548285 2243 catalog_manager.cc:4830] TS 327aa26a8e434a609ebcb382aa665fce (127.0.99.133:38853): tablet c10343409a2a423eb0b21759bfc9ba61 (table rebalance_test_table_1 [id=d0846ac337c64bf1ac218add4c796fff]) successfully deleted
I20240503 01:59:06.559173 2920 consensus_queue.cc:237] T 10ecacdfb9a44d068b0df24e991612ee P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 50, Committed index: 50, Last appended: 1.50, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 51 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } attrs { replace: true } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: true } }
I20240503 01:59:06.560879 2662 raft_consensus.cc:1261] T 10ecacdfb9a44d068b0df24e991612ee P b40a9a7b30d34a6db007ed8febd8a759 [term 1 FOLLOWER]: Refusing update from remote peer 327aa26a8e434a609ebcb382aa665fce: Log matching property violated. Preceding OpId in replica: term: 1 index: 50. Preceding OpId from leader: term: 1 index: 51. (index mismatch)
I20240503 01:59:06.561290 4214 consensus_queue.cc:1035] T 10ecacdfb9a44d068b0df24e991612ee P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 51, Last known committed idx: 50, Time since last communication: 0.000s
I20240503 01:59:06.561509 2789 raft_consensus.cc:1261] T 10ecacdfb9a44d068b0df24e991612ee P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Refusing update from remote peer 327aa26a8e434a609ebcb382aa665fce: Log matching property violated. Preceding OpId in replica: term: 1 index: 50. Preceding OpId from leader: term: 1 index: 51. (index mismatch)
I20240503 01:59:06.562044 2531 raft_consensus.cc:1261] T 10ecacdfb9a44d068b0df24e991612ee P a1a844ecd71145e4a362f4c42500fbcb [term 1 FOLLOWER]: Refusing update from remote peer 327aa26a8e434a609ebcb382aa665fce: Log matching property violated. Preceding OpId in replica: term: 1 index: 50. Preceding OpId from leader: term: 1 index: 51. (index mismatch)
W20240503 01:59:06.562644 2860 consensus_peers.cc:485] T 10ecacdfb9a44d068b0df24e991612ee P 327aa26a8e434a609ebcb382aa665fce -> Peer 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135:39501): Couldn't send request to peer 4125b1a70e864e129bd54bdc2720d7fa. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 10ecacdfb9a44d068b0df24e991612ee. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:06.562794 4216 consensus_queue.cc:1035] T 10ecacdfb9a44d068b0df24e991612ee P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 51, Last known committed idx: 50, Time since last communication: 0.000s
I20240503 01:59:06.563256 4216 consensus_queue.cc:1035] T 10ecacdfb9a44d068b0df24e991612ee P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 51, Last known committed idx: 50, Time since last communication: 0.000s
I20240503 01:59:06.567710 4039 raft_consensus.cc:2932] T 10ecacdfb9a44d068b0df24e991612ee P 327aa26a8e434a609ebcb382aa665fce [term 1 LEADER]: Committing config change with OpId 1.51: config changed from index -1 to 51, NON_VOTER 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) added. New config: { opid_index: 51 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } attrs { replace: true } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: true } } }
I20240503 01:59:06.570192 2660 raft_consensus.cc:2932] T 10ecacdfb9a44d068b0df24e991612ee P b40a9a7b30d34a6db007ed8febd8a759 [term 1 FOLLOWER]: Committing config change with OpId 1.51: config changed from index -1 to 51, NON_VOTER 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) added. New config: { opid_index: 51 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } attrs { replace: true } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: true } } }
I20240503 01:59:06.571844 4438 raft_consensus.cc:1050] T 19b3a90e13194997bde2b9a806f8f2b5 P b63928177fa54227a5ff9929265ab9dd: attempting to promote NON_VOTER 4125b1a70e864e129bd54bdc2720d7fa to VOTER
I20240503 01:59:06.574816 2394 raft_consensus.cc:2932] T a6e17a2c4ba0482e96d04eb9d3ba854a P b63928177fa54227a5ff9929265ab9dd [term 1 FOLLOWER]: Committing config change with OpId 1.66: config changed from index 56 to 66, b63928177fa54227a5ff9929265ab9dd (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } attrs { replace: true } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { promote: false } } }
I20240503 01:59:06.574887 2789 raft_consensus.cc:2932] T 10ecacdfb9a44d068b0df24e991612ee P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Committing config change with OpId 1.51: config changed from index -1 to 51, NON_VOTER 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) added. New config: { opid_index: 51 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } attrs { replace: true } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: true } } }
I20240503 01:59:06.579077 2251 catalog_manager.cc:5484] T 10ecacdfb9a44d068b0df24e991612ee P 52c196c15e3348b985c689652bc49918 reported cstate change: config changed from index -1 to 51, NON_VOTER 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "327aa26a8e434a609ebcb382aa665fce" committed_config { opid_index: 51 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } attrs { replace: true } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: true } } }
I20240503 01:59:06.581676 2530 raft_consensus.cc:2932] T 10ecacdfb9a44d068b0df24e991612ee P a1a844ecd71145e4a362f4c42500fbcb [term 1 FOLLOWER]: Committing config change with OpId 1.51: config changed from index -1 to 51, NON_VOTER 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) added. New config: { opid_index: 51 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } attrs { replace: true } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: true } } }
I20240503 01:59:06.585068 4438 consensus_queue.cc:237] T 19b3a90e13194997bde2b9a806f8f2b5 P b63928177fa54227a5ff9929265ab9dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 105, Committed index: 105, Last appended: 1.105, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } attrs { replace: true } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } }
I20240503 01:59:06.588536 3051 raft_consensus.cc:1261] T 19b3a90e13194997bde2b9a806f8f2b5 P 6568f69a78c44bb985162d2b7d0a9376 [term 1 FOLLOWER]: Refusing update from remote peer b63928177fa54227a5ff9929265ab9dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 105. Preceding OpId from leader: term: 1 index: 106. (index mismatch)
I20240503 01:59:06.588941 3802 consensus_queue.cc:1035] T 19b3a90e13194997bde2b9a806f8f2b5 P b63928177fa54227a5ff9929265ab9dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 106, Last known committed idx: 105, Time since last communication: 0.000s
I20240503 01:59:06.592123 2397 raft_consensus.cc:1261] T 10ecacdfb9a44d068b0df24e991612ee P b63928177fa54227a5ff9929265ab9dd [term 1 FOLLOWER]: Refusing update from remote peer 327aa26a8e434a609ebcb382aa665fce: Log matching property violated. Preceding OpId in replica: term: 1 index: 48. Preceding OpId from leader: term: 1 index: 51. (index mismatch)
I20240503 01:59:06.594192 2398 raft_consensus.cc:2932] T 5fd1010447524b6695abb98192dc54f1 P b63928177fa54227a5ff9929265ab9dd [term 1 FOLLOWER]: Committing config change with OpId 1.42: config changed from index 39 to 42, VOTER b40a9a7b30d34a6db007ed8febd8a759 (127.0.99.131) evicted. New config: { opid_index: 42 OBSOLETE_local: false peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } attrs { promote: false } } }
I20240503 01:59:06.600581 2662 raft_consensus.cc:1261] T 19b3a90e13194997bde2b9a806f8f2b5 P b40a9a7b30d34a6db007ed8febd8a759 [term 1 FOLLOWER]: Refusing update from remote peer b63928177fa54227a5ff9929265ab9dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 105. Preceding OpId from leader: term: 1 index: 106. (index mismatch)
I20240503 01:59:06.600667 4214 consensus_queue.cc:1035] T 10ecacdfb9a44d068b0df24e991612ee P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 51, Last known committed idx: 47, Time since last communication: 0.000s
I20240503 01:59:06.604440 3957 raft_consensus.cc:1261] T 19b3a90e13194997bde2b9a806f8f2b5 P 4125b1a70e864e129bd54bdc2720d7fa [term 1 LEARNER]: Refusing update from remote peer b63928177fa54227a5ff9929265ab9dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 105. Preceding OpId from leader: term: 1 index: 106. (index mismatch)
I20240503 01:59:06.605484 2787 raft_consensus.cc:1261] T 19b3a90e13194997bde2b9a806f8f2b5 P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Refusing update from remote peer b63928177fa54227a5ff9929265ab9dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 105. Preceding OpId from leader: term: 1 index: 106. (index mismatch)
I20240503 01:59:06.608830 2520 raft_consensus.cc:1261] T 19b3a90e13194997bde2b9a806f8f2b5 P a1a844ecd71145e4a362f4c42500fbcb [term 1 FOLLOWER]: Refusing update from remote peer b63928177fa54227a5ff9929265ab9dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 105. Preceding OpId from leader: term: 1 index: 106. (index mismatch)
I20240503 01:59:06.613114 4439 consensus_queue.cc:1035] T 19b3a90e13194997bde2b9a806f8f2b5 P b63928177fa54227a5ff9929265ab9dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 106, Last known committed idx: 104, Time since last communication: 0.000s
I20240503 01:59:06.671983 4439 consensus_queue.cc:1035] T 19b3a90e13194997bde2b9a806f8f2b5 P b63928177fa54227a5ff9929265ab9dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 106, Last known committed idx: 105, Time since last communication: 0.000s
I20240503 01:59:06.673454 4439 consensus_queue.cc:1035] T 19b3a90e13194997bde2b9a806f8f2b5 P b63928177fa54227a5ff9929265ab9dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 106, Last known committed idx: 105, Time since last communication: 0.000s
I20240503 01:59:06.687820 2397 raft_consensus.cc:2932] T 10ecacdfb9a44d068b0df24e991612ee P b63928177fa54227a5ff9929265ab9dd [term 1 FOLLOWER]: Committing config change with OpId 1.51: config changed from index -1 to 51, NON_VOTER 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) added. New config: { opid_index: 51 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } attrs { replace: true } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: true } } }
I20240503 01:59:06.693972 4341 consensus_queue.cc:1035] T 19b3a90e13194997bde2b9a806f8f2b5 P b63928177fa54227a5ff9929265ab9dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 106, Last known committed idx: 105, Time since last communication: 0.000s
I20240503 01:59:06.700163 4132 raft_consensus.cc:2932] T 19b3a90e13194997bde2b9a806f8f2b5 P b63928177fa54227a5ff9929265ab9dd [term 1 LEADER]: Committing config change with OpId 1.106: config changed from index 65 to 106, 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } attrs { replace: true } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:06.700821 2662 raft_consensus.cc:2932] T 19b3a90e13194997bde2b9a806f8f2b5 P b40a9a7b30d34a6db007ed8febd8a759 [term 1 FOLLOWER]: Committing config change with OpId 1.106: config changed from index 65 to 106, 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } attrs { replace: true } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:06.702836 3051 raft_consensus.cc:2932] T 19b3a90e13194997bde2b9a806f8f2b5 P 6568f69a78c44bb985162d2b7d0a9376 [term 1 FOLLOWER]: Committing config change with OpId 1.106: config changed from index 65 to 106, 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } attrs { replace: true } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:06.706490 2251 catalog_manager.cc:5484] T 19b3a90e13194997bde2b9a806f8f2b5 P 6568f69a78c44bb985162d2b7d0a9376 reported cstate change: config changed from index 65 to 106, 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b63928177fa54227a5ff9929265ab9dd" committed_config { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } attrs { replace: true } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:06.726328 2530 raft_consensus.cc:2932] T 19b3a90e13194997bde2b9a806f8f2b5 P a1a844ecd71145e4a362f4c42500fbcb [term 1 FOLLOWER]: Committing config change with OpId 1.106: config changed from index 65 to 106, 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } attrs { replace: true } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:06.731362 3955 raft_consensus.cc:2932] T 19b3a90e13194997bde2b9a806f8f2b5 P 4125b1a70e864e129bd54bdc2720d7fa [term 1 FOLLOWER]: Committing config change with OpId 1.106: config changed from index 65 to 106, 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } attrs { replace: true } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:06.733264 2789 raft_consensus.cc:2932] T 19b3a90e13194997bde2b9a806f8f2b5 P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Committing config change with OpId 1.106: config changed from index 65 to 106, 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } attrs { replace: true } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:06.747014 2394 consensus_queue.cc:237] T 19b3a90e13194997bde2b9a806f8f2b5 P b63928177fa54227a5ff9929265ab9dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 107, Committed index: 107, Last appended: 1.107, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } }
I20240503 01:59:06.751696 3955 raft_consensus.cc:1261] T 19b3a90e13194997bde2b9a806f8f2b5 P 4125b1a70e864e129bd54bdc2720d7fa [term 1 FOLLOWER]: Refusing update from remote peer b63928177fa54227a5ff9929265ab9dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 107. Preceding OpId from leader: term: 1 index: 108. (index mismatch)
I20240503 01:59:06.752097 2662 raft_consensus.cc:1261] T 19b3a90e13194997bde2b9a806f8f2b5 P b40a9a7b30d34a6db007ed8febd8a759 [term 1 FOLLOWER]: Refusing update from remote peer b63928177fa54227a5ff9929265ab9dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 107. Preceding OpId from leader: term: 1 index: 108. (index mismatch)
I20240503 01:59:06.752094 2530 raft_consensus.cc:1261] T 19b3a90e13194997bde2b9a806f8f2b5 P a1a844ecd71145e4a362f4c42500fbcb [term 1 FOLLOWER]: Refusing update from remote peer b63928177fa54227a5ff9929265ab9dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 107. Preceding OpId from leader: term: 1 index: 108. (index mismatch)
I20240503 01:59:06.752990 2791 raft_consensus.cc:1261] T 19b3a90e13194997bde2b9a806f8f2b5 P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Refusing update from remote peer b63928177fa54227a5ff9929265ab9dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 107. Preceding OpId from leader: term: 1 index: 108. (index mismatch)
I20240503 01:59:06.759150 3802 consensus_queue.cc:1035] T 19b3a90e13194997bde2b9a806f8f2b5 P b63928177fa54227a5ff9929265ab9dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 108, Last known committed idx: 107, Time since last communication: 0.000s
I20240503 01:59:06.760068 4332 consensus_queue.cc:1035] T 19b3a90e13194997bde2b9a806f8f2b5 P b63928177fa54227a5ff9929265ab9dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 108, Last known committed idx: 107, Time since last communication: 0.000s
I20240503 01:59:06.768507 4332 consensus_queue.cc:1035] T 19b3a90e13194997bde2b9a806f8f2b5 P b63928177fa54227a5ff9929265ab9dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 108, Last known committed idx: 107, Time since last communication: 0.000s
I20240503 01:59:06.768896 4341 raft_consensus.cc:2932] T 19b3a90e13194997bde2b9a806f8f2b5 P b63928177fa54227a5ff9929265ab9dd [term 1 LEADER]: Committing config change with OpId 1.108: config changed from index 106 to 108, VOTER 6568f69a78c44bb985162d2b7d0a9376 (127.0.99.134) evicted. New config: { opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:06.775246 2245 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 19b3a90e13194997bde2b9a806f8f2b5 with cas_config_opid_index 106: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:06.776989 3955 raft_consensus.cc:2932] T 19b3a90e13194997bde2b9a806f8f2b5 P 4125b1a70e864e129bd54bdc2720d7fa [term 1 FOLLOWER]: Committing config change with OpId 1.108: config changed from index 106 to 108, VOTER 6568f69a78c44bb985162d2b7d0a9376 (127.0.99.134) evicted. New config: { opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:06.779420 2789 raft_consensus.cc:2932] T 19b3a90e13194997bde2b9a806f8f2b5 P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Committing config change with OpId 1.108: config changed from index 106 to 108, VOTER 6568f69a78c44bb985162d2b7d0a9376 (127.0.99.134) evicted. New config: { opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:06.779788 2921 consensus_queue.cc:237] T a6e17a2c4ba0482e96d04eb9d3ba854a P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 68, Committed index: 68, Last appended: 1.69, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { promote: false } }
I20240503 01:59:06.781368 2662 raft_consensus.cc:1261] T a6e17a2c4ba0482e96d04eb9d3ba854a P b40a9a7b30d34a6db007ed8febd8a759 [term 1 FOLLOWER]: Refusing update from remote peer 327aa26a8e434a609ebcb382aa665fce: Log matching property violated. Preceding OpId in replica: term: 1 index: 69. Preceding OpId from leader: term: 1 index: 70. (index mismatch)
I20240503 01:59:06.781354 2530 raft_consensus.cc:2932] T 19b3a90e13194997bde2b9a806f8f2b5 P a1a844ecd71145e4a362f4c42500fbcb [term 1 FOLLOWER]: Committing config change with OpId 1.108: config changed from index 106 to 108, VOTER 6568f69a78c44bb985162d2b7d0a9376 (127.0.99.134) evicted. New config: { opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:06.781772 4098 consensus_queue.cc:1035] T a6e17a2c4ba0482e96d04eb9d3ba854a P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 70, Last known committed idx: 68, Time since last communication: 0.000s
I20240503 01:59:06.782145 2253 catalog_manager.cc:5484] T 19b3a90e13194997bde2b9a806f8f2b5 P 52c196c15e3348b985c689652bc49918 reported cstate change: config changed from index 106 to 108, VOTER 6568f69a78c44bb985162d2b7d0a9376 (127.0.99.134) evicted. New cstate: current_term: 1 leader_uuid: "b63928177fa54227a5ff9929265ab9dd" committed_config { opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:06.786010 3054 raft_consensus.cc:1261] T a6e17a2c4ba0482e96d04eb9d3ba854a P 6568f69a78c44bb985162d2b7d0a9376 [term 1 FOLLOWER]: Refusing update from remote peer 327aa26a8e434a609ebcb382aa665fce: Log matching property violated. Preceding OpId in replica: term: 1 index: 69. Preceding OpId from leader: term: 1 index: 70. (index mismatch)
I20240503 01:59:06.788908 4197 consensus_queue.cc:1035] T a6e17a2c4ba0482e96d04eb9d3ba854a P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 70, Last known committed idx: 68, Time since last communication: 0.000s
I20240503 01:59:06.791782 4216 raft_consensus.cc:2932] T a6e17a2c4ba0482e96d04eb9d3ba854a P 327aa26a8e434a609ebcb382aa665fce [term 1 LEADER]: Committing config change with OpId 1.70: config changed from index 66 to 70, VOTER 52c196c15e3348b985c689652bc49918 (127.0.99.132) evicted. New config: { opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { promote: false } } }
I20240503 01:59:06.796520 2243 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a6e17a2c4ba0482e96d04eb9d3ba854a with cas_config_opid_index 66: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:06.797221 3051 raft_consensus.cc:2932] T a6e17a2c4ba0482e96d04eb9d3ba854a P 6568f69a78c44bb985162d2b7d0a9376 [term 1 FOLLOWER]: Committing config change with OpId 1.70: config changed from index 66 to 70, VOTER 52c196c15e3348b985c689652bc49918 (127.0.99.132) evicted. New config: { opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { promote: false } } }
I20240503 01:59:06.797849 3955 raft_consensus.cc:1203] T d4d07c9621134780bc7ece7bb061d934 P 4125b1a70e864e129bd54bdc2720d7fa [term 1 LEARNER]: Deduplicated request from leader. Original: 1.54->[1.55-1.89] Dedup: 1.69->[1.70-1.89]
I20240503 01:59:06.802964 4341 consensus_queue.cc:1035] T 19b3a90e13194997bde2b9a806f8f2b5 P b63928177fa54227a5ff9929265ab9dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 108, Last known committed idx: 107, Time since last communication: 0.000s
I20240503 01:59:06.804344 2660 raft_consensus.cc:2932] T a6e17a2c4ba0482e96d04eb9d3ba854a P b40a9a7b30d34a6db007ed8febd8a759 [term 1 FOLLOWER]: Committing config change with OpId 1.70: config changed from index 66 to 70, VOTER 52c196c15e3348b985c689652bc49918 (127.0.99.132) evicted. New config: { opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { promote: false } } }
I20240503 01:59:06.805516 2530 raft_consensus.cc:1261] T a6e17a2c4ba0482e96d04eb9d3ba854a P a1a844ecd71145e4a362f4c42500fbcb [term 1 FOLLOWER]: Refusing update from remote peer 327aa26a8e434a609ebcb382aa665fce: Log matching property violated. Preceding OpId in replica: term: 1 index: 69. Preceding OpId from leader: term: 1 index: 70. (index mismatch)
I20240503 01:59:06.807147 2397 raft_consensus.cc:1261] T a6e17a2c4ba0482e96d04eb9d3ba854a P b63928177fa54227a5ff9929265ab9dd [term 1 FOLLOWER]: Refusing update from remote peer 327aa26a8e434a609ebcb382aa665fce: Log matching property violated. Preceding OpId in replica: term: 1 index: 69. Preceding OpId from leader: term: 1 index: 70. (index mismatch)
I20240503 01:59:06.808703 2250 catalog_manager.cc:5484] T a6e17a2c4ba0482e96d04eb9d3ba854a P 6568f69a78c44bb985162d2b7d0a9376 reported cstate change: config changed from index 66 to 70, VOTER 52c196c15e3348b985c689652bc49918 (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "327aa26a8e434a609ebcb382aa665fce" committed_config { opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { promote: false } } }
I20240503 01:59:06.810312 3035 tablet_service.cc:1514] Processing DeleteTablet for tablet 19b3a90e13194997bde2b9a806f8f2b5 with delete_type TABLET_DATA_TOMBSTONED (TS 6568f69a78c44bb985162d2b7d0a9376 not found in new config with opid_index 108) from {username='slave'} at 127.0.0.1:53118
I20240503 01:59:06.810432 4214 consensus_queue.cc:1035] T a6e17a2c4ba0482e96d04eb9d3ba854a P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 70, Last known committed idx: 68, Time since last communication: 0.000s
I20240503 01:59:06.812577 2660 raft_consensus.cc:2932] T 19b3a90e13194997bde2b9a806f8f2b5 P b40a9a7b30d34a6db007ed8febd8a759 [term 1 FOLLOWER]: Committing config change with OpId 1.108: config changed from index 106 to 108, VOTER 6568f69a78c44bb985162d2b7d0a9376 (127.0.99.134) evicted. New config: { opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:06.813139 2772 tablet_service.cc:1514] Processing DeleteTablet for tablet a6e17a2c4ba0482e96d04eb9d3ba854a with delete_type TABLET_DATA_TOMBSTONED (TS 52c196c15e3348b985c689652bc49918 not found in new config with opid_index 70) from {username='slave'} at 127.0.0.1:45406
I20240503 01:59:06.814401 4483 tablet_replica.cc:330] T 19b3a90e13194997bde2b9a806f8f2b5 P 6568f69a78c44bb985162d2b7d0a9376: stopping tablet replica
I20240503 01:59:06.814779 4483 raft_consensus.cc:2229] T 19b3a90e13194997bde2b9a806f8f2b5 P 6568f69a78c44bb985162d2b7d0a9376 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:06.815516 4483 raft_consensus.cc:2258] T 19b3a90e13194997bde2b9a806f8f2b5 P 6568f69a78c44bb985162d2b7d0a9376 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:06.817541 4483 ts_tablet_manager.cc:1875] T 19b3a90e13194997bde2b9a806f8f2b5 P 6568f69a78c44bb985162d2b7d0a9376: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:06.836192 4483 ts_tablet_manager.cc:1888] T 19b3a90e13194997bde2b9a806f8f2b5 P 6568f69a78c44bb985162d2b7d0a9376: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.107
I20240503 01:59:06.836427 4483 log.cc:1196] T 19b3a90e13194997bde2b9a806f8f2b5 P 6568f69a78c44bb985162d2b7d0a9376: Deleting WAL directory at /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/19b3a90e13194997bde2b9a806f8f2b5
I20240503 01:59:06.839602 2243 catalog_manager.cc:4830] TS 6568f69a78c44bb985162d2b7d0a9376 (127.0.99.134:43603): tablet 19b3a90e13194997bde2b9a806f8f2b5 (table rebalance_test_table_2 [id=18b5c04f80684606bc736d3ca6780b16]) successfully deleted
I20240503 01:59:06.841112 2396 raft_consensus.cc:2932] T a6e17a2c4ba0482e96d04eb9d3ba854a P b63928177fa54227a5ff9929265ab9dd [term 1 FOLLOWER]: Committing config change with OpId 1.70: config changed from index 66 to 70, VOTER 52c196c15e3348b985c689652bc49918 (127.0.99.132) evicted. New config: { opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { promote: false } } }
I20240503 01:59:06.865989 4214 consensus_queue.cc:1035] T a6e17a2c4ba0482e96d04eb9d3ba854a P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 70, Last known committed idx: 68, Time since last communication: 0.000s
I20240503 01:59:06.869864 4484 tablet_replica.cc:330] T a6e17a2c4ba0482e96d04eb9d3ba854a P 52c196c15e3348b985c689652bc49918: stopping tablet replica
I20240503 01:59:06.870231 4484 raft_consensus.cc:2229] T a6e17a2c4ba0482e96d04eb9d3ba854a P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:06.870508 4484 pending_rounds.cc:62] T a6e17a2c4ba0482e96d04eb9d3ba854a P 52c196c15e3348b985c689652bc49918: Trying to abort 1 pending ops.
I20240503 01:59:06.870605 4484 pending_rounds.cc:68] T a6e17a2c4ba0482e96d04eb9d3ba854a P 52c196c15e3348b985c689652bc49918: Aborting op as it isn't in flight: id { term: 1 index: 69 } timestamp: 7023417535442046976 op_type: WRITE_OP write_request { tablet_id: "a6e17a2c4ba0482e96d04eb9d3ba854a" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\231a2J\241\0053\363\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""eb05facf2b2a8434" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023417535189893120 authz_token { token_data: "\010\221\214\321\261\006\"3\n\005slave\022*\n d0846ac337c64bf1ac218add4c796fff\020\001\030\001 \001(\001" signature: "<redacted>""f\"w\200@\373>yk\204\026\017`\207;\333\274NS\371\203\003G\254\205\216\317jn9\304S&^\247\331\351\325\026\034\360\211\277\271k\365\212\336\224KO\223\005\375\245mo\223\2328U\tg\216" signing_key_seq_num: 0 } } request_id { client_id: "45556a5dc8664c03ba879df420f1330c" seq_no: 177 first_incomplete_seq_no: 177 attempt_no: 0 }
I20240503 01:59:06.872356 4484 raft_consensus.cc:2258] T a6e17a2c4ba0482e96d04eb9d3ba854a P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:06.873751 4484 ts_tablet_manager.cc:1875] T a6e17a2c4ba0482e96d04eb9d3ba854a P 52c196c15e3348b985c689652bc49918: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:06.880092 4484 ts_tablet_manager.cc:1888] T a6e17a2c4ba0482e96d04eb9d3ba854a P 52c196c15e3348b985c689652bc49918: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.69
I20240503 01:59:06.881162 2531 raft_consensus.cc:2932] T a6e17a2c4ba0482e96d04eb9d3ba854a P a1a844ecd71145e4a362f4c42500fbcb [term 1 FOLLOWER]: Committing config change with OpId 1.70: config changed from index 66 to 70, VOTER 52c196c15e3348b985c689652bc49918 (127.0.99.132) evicted. New config: { opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { promote: false } } }
I20240503 01:59:06.888229 4484 log.cc:1196] T a6e17a2c4ba0482e96d04eb9d3ba854a P 52c196c15e3348b985c689652bc49918: Deleting WAL directory at /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/a6e17a2c4ba0482e96d04eb9d3ba854a
I20240503 01:59:06.895713 2245 catalog_manager.cc:4830] TS 52c196c15e3348b985c689652bc49918 (127.0.99.132:42887): tablet a6e17a2c4ba0482e96d04eb9d3ba854a (table rebalance_test_table_1 [id=d0846ac337c64bf1ac218add4c796fff]) successfully deleted
I20240503 01:59:07.003309 4098 raft_consensus.cc:1050] T d4d07c9621134780bc7ece7bb061d934 P 327aa26a8e434a609ebcb382aa665fce: attempting to promote NON_VOTER 4125b1a70e864e129bd54bdc2720d7fa to VOTER
I20240503 01:59:07.003757 4098 consensus_queue.cc:237] T d4d07c9621134780bc7ece7bb061d934 P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 95, Committed index: 95, Last appended: 1.95, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 96 OBSOLETE_local: false peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } attrs { replace: true } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } }
I20240503 01:59:07.005445 3051 raft_consensus.cc:1261] T d4d07c9621134780bc7ece7bb061d934 P 6568f69a78c44bb985162d2b7d0a9376 [term 1 FOLLOWER]: Refusing update from remote peer 327aa26a8e434a609ebcb382aa665fce: Log matching property violated. Preceding OpId in replica: term: 1 index: 95. Preceding OpId from leader: term: 1 index: 96. (index mismatch)
I20240503 01:59:07.006608 4039 consensus_queue.cc:1035] T d4d07c9621134780bc7ece7bb061d934 P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 96, Last known committed idx: 95, Time since last communication: 0.000s
I20240503 01:59:07.009527 2662 raft_consensus.cc:1261] T d4d07c9621134780bc7ece7bb061d934 P b40a9a7b30d34a6db007ed8febd8a759 [term 1 FOLLOWER]: Refusing update from remote peer 327aa26a8e434a609ebcb382aa665fce: Log matching property violated. Preceding OpId in replica: term: 1 index: 95. Preceding OpId from leader: term: 1 index: 96. (index mismatch)
I20240503 01:59:07.010166 4039 consensus_queue.cc:1035] T d4d07c9621134780bc7ece7bb061d934 P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 96, Last known committed idx: 95, Time since last communication: 0.000s
I20240503 01:59:07.057018 2520 raft_consensus.cc:1261] T d4d07c9621134780bc7ece7bb061d934 P a1a844ecd71145e4a362f4c42500fbcb [term 1 FOLLOWER]: Refusing update from remote peer 327aa26a8e434a609ebcb382aa665fce: Log matching property violated. Preceding OpId in replica: term: 1 index: 93. Preceding OpId from leader: term: 1 index: 96. (index mismatch)
I20240503 01:59:07.057541 3957 raft_consensus.cc:1261] T d4d07c9621134780bc7ece7bb061d934 P 4125b1a70e864e129bd54bdc2720d7fa [term 1 LEARNER]: Refusing update from remote peer 327aa26a8e434a609ebcb382aa665fce: Log matching property violated. Preceding OpId in replica: term: 1 index: 95. Preceding OpId from leader: term: 1 index: 96. (index mismatch)
I20240503 01:59:07.066394 4506 ts_tablet_manager.cc:902] T 10ecacdfb9a44d068b0df24e991612ee P 4125b1a70e864e129bd54bdc2720d7fa: Initiating tablet copy from peer 327aa26a8e434a609ebcb382aa665fce (127.0.99.133:38853)
I20240503 01:59:07.068229 4506 tablet_copy_client.cc:308] T 10ecacdfb9a44d068b0df24e991612ee P 4125b1a70e864e129bd54bdc2720d7fa: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.133:38853
I20240503 01:59:07.068744 2944 tablet_copy_service.cc:140] P 327aa26a8e434a609ebcb382aa665fce: Received BeginTabletCopySession request for tablet 10ecacdfb9a44d068b0df24e991612ee from peer 4125b1a70e864e129bd54bdc2720d7fa ({username='slave'} at 127.0.99.135:39289)
I20240503 01:59:07.068851 2944 tablet_copy_service.cc:161] P 327aa26a8e434a609ebcb382aa665fce: Beginning new tablet copy session on tablet 10ecacdfb9a44d068b0df24e991612ee from peer 4125b1a70e864e129bd54bdc2720d7fa at {username='slave'} at 127.0.99.135:39289: session id = 4125b1a70e864e129bd54bdc2720d7fa-10ecacdfb9a44d068b0df24e991612ee
I20240503 01:59:07.069808 2944 tablet_copy_source_session.cc:205] T 10ecacdfb9a44d068b0df24e991612ee P 327aa26a8e434a609ebcb382aa665fce: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:07.073024 4506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10ecacdfb9a44d068b0df24e991612ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:07.075570 4506 tablet_copy_client.cc:783] T 10ecacdfb9a44d068b0df24e991612ee P 4125b1a70e864e129bd54bdc2720d7fa: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:07.075755 4506 tablet_copy_client.cc:647] T 10ecacdfb9a44d068b0df24e991612ee P 4125b1a70e864e129bd54bdc2720d7fa: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:07.076583 2398 raft_consensus.cc:1261] T d4d07c9621134780bc7ece7bb061d934 P b63928177fa54227a5ff9929265ab9dd [term 1 FOLLOWER]: Refusing update from remote peer 327aa26a8e434a609ebcb382aa665fce: Log matching property violated. Preceding OpId in replica: term: 1 index: 91. Preceding OpId from leader: term: 1 index: 96. (index mismatch)
I20240503 01:59:07.079269 4506 tablet_copy_client.cc:523] T 10ecacdfb9a44d068b0df24e991612ee P 4125b1a70e864e129bd54bdc2720d7fa: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:07.080677 4506 tablet_bootstrap.cc:492] T 10ecacdfb9a44d068b0df24e991612ee P 4125b1a70e864e129bd54bdc2720d7fa: Bootstrap starting.
I20240503 01:59:07.093513 4497 consensus_queue.cc:1035] T d4d07c9621134780bc7ece7bb061d934 P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 96, Last known committed idx: 95, Time since last communication: 0.000s
I20240503 01:59:07.098485 4497 consensus_queue.cc:1035] T d4d07c9621134780bc7ece7bb061d934 P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 96, Last known committed idx: 92, Time since last communication: 0.000s
I20240503 01:59:07.123473 4497 consensus_queue.cc:1035] T d4d07c9621134780bc7ece7bb061d934 P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 96, Last known committed idx: 91, Time since last communication: 0.000s
I20240503 01:59:07.128150 4214 raft_consensus.cc:2932] T d4d07c9621134780bc7ece7bb061d934 P 327aa26a8e434a609ebcb382aa665fce [term 1 LEADER]: Committing config change with OpId 1.96: config changed from index 55 to 96, 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 96 OBSOLETE_local: false peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } attrs { replace: true } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:07.130103 2660 raft_consensus.cc:2932] T d4d07c9621134780bc7ece7bb061d934 P b40a9a7b30d34a6db007ed8febd8a759 [term 1 FOLLOWER]: Committing config change with OpId 1.96: config changed from index 55 to 96, 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 96 OBSOLETE_local: false peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } attrs { replace: true } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:07.130717 3055 raft_consensus.cc:2932] T d4d07c9621134780bc7ece7bb061d934 P 6568f69a78c44bb985162d2b7d0a9376 [term 1 FOLLOWER]: Committing config change with OpId 1.96: config changed from index 55 to 96, 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 96 OBSOLETE_local: false peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } attrs { replace: true } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:07.133806 2251 catalog_manager.cc:5484] T d4d07c9621134780bc7ece7bb061d934 P b40a9a7b30d34a6db007ed8febd8a759 reported cstate change: config changed from index 55 to 96, 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "327aa26a8e434a609ebcb382aa665fce" committed_config { opid_index: 96 OBSOLETE_local: false peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } attrs { replace: true } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:07.138268 2524 raft_consensus.cc:2932] T d4d07c9621134780bc7ece7bb061d934 P a1a844ecd71145e4a362f4c42500fbcb [term 1 FOLLOWER]: Committing config change with OpId 1.96: config changed from index 55 to 96, 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 96 OBSOLETE_local: false peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } attrs { replace: true } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:07.139376 3957 raft_consensus.cc:2932] T d4d07c9621134780bc7ece7bb061d934 P 4125b1a70e864e129bd54bdc2720d7fa [term 1 FOLLOWER]: Committing config change with OpId 1.96: config changed from index 55 to 96, 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 96 OBSOLETE_local: false peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } attrs { replace: true } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:07.185132 2920 consensus_queue.cc:237] T d4d07c9621134780bc7ece7bb061d934 P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 97, Committed index: 97, Last appended: 1.98, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 99 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } }
I20240503 01:59:07.187438 2660 raft_consensus.cc:1261] T d4d07c9621134780bc7ece7bb061d934 P b40a9a7b30d34a6db007ed8febd8a759 [term 1 FOLLOWER]: Refusing update from remote peer 327aa26a8e434a609ebcb382aa665fce: Log matching property violated. Preceding OpId in replica: term: 1 index: 98. Preceding OpId from leader: term: 1 index: 99. (index mismatch)
I20240503 01:59:07.188779 3551 consensus_queue.cc:1035] T d4d07c9621134780bc7ece7bb061d934 P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 99, Last known committed idx: 97, Time since last communication: 0.000s
I20240503 01:59:07.189229 2530 raft_consensus.cc:1261] T d4d07c9621134780bc7ece7bb061d934 P a1a844ecd71145e4a362f4c42500fbcb [term 1 FOLLOWER]: Refusing update from remote peer 327aa26a8e434a609ebcb382aa665fce: Log matching property violated. Preceding OpId in replica: term: 1 index: 98. Preceding OpId from leader: term: 1 index: 99. (index mismatch)
I20240503 01:59:07.190574 3551 consensus_queue.cc:1035] T d4d07c9621134780bc7ece7bb061d934 P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 99, Last known committed idx: 97, Time since last communication: 0.000s
I20240503 01:59:07.191716 3957 raft_consensus.cc:1261] T d4d07c9621134780bc7ece7bb061d934 P 4125b1a70e864e129bd54bdc2720d7fa [term 1 FOLLOWER]: Refusing update from remote peer 327aa26a8e434a609ebcb382aa665fce: Log matching property violated. Preceding OpId in replica: term: 1 index: 98. Preceding OpId from leader: term: 1 index: 99. (index mismatch)
I20240503 01:59:07.198555 4497 raft_consensus.cc:2932] T d4d07c9621134780bc7ece7bb061d934 P 327aa26a8e434a609ebcb382aa665fce [term 1 LEADER]: Committing config change with OpId 1.99: config changed from index 96 to 99, VOTER 6568f69a78c44bb985162d2b7d0a9376 (127.0.99.134) evicted. New config: { opid_index: 99 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:07.200794 2243 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d4d07c9621134780bc7ece7bb061d934 with cas_config_opid_index 96: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:07.204001 4216 consensus_queue.cc:1035] T d4d07c9621134780bc7ece7bb061d934 P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 99, Last known committed idx: 97, Time since last communication: 0.000s
I20240503 01:59:07.207284 2251 catalog_manager.cc:5484] T d4d07c9621134780bc7ece7bb061d934 P 327aa26a8e434a609ebcb382aa665fce reported cstate change: config changed from index 96 to 99, VOTER 6568f69a78c44bb985162d2b7d0a9376 (127.0.99.134) evicted. New cstate: current_term: 1 leader_uuid: "327aa26a8e434a609ebcb382aa665fce" committed_config { opid_index: 99 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:07.204511 2660 raft_consensus.cc:2932] T d4d07c9621134780bc7ece7bb061d934 P b40a9a7b30d34a6db007ed8febd8a759 [term 1 FOLLOWER]: Committing config change with OpId 1.99: config changed from index 96 to 99, VOTER 6568f69a78c44bb985162d2b7d0a9376 (127.0.99.134) evicted. New config: { opid_index: 99 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:07.208488 2520 raft_consensus.cc:2932] T d4d07c9621134780bc7ece7bb061d934 P a1a844ecd71145e4a362f4c42500fbcb [term 1 FOLLOWER]: Committing config change with OpId 1.99: config changed from index 96 to 99, VOTER 6568f69a78c44bb985162d2b7d0a9376 (127.0.99.134) evicted. New config: { opid_index: 99 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:07.211786 2397 raft_consensus.cc:2932] T d4d07c9621134780bc7ece7bb061d934 P b63928177fa54227a5ff9929265ab9dd [term 1 FOLLOWER]: Committing config change with OpId 1.96: config changed from index 55 to 96, 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 96 OBSOLETE_local: false peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } attrs { replace: true } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:07.212635 3957 raft_consensus.cc:2932] T d4d07c9621134780bc7ece7bb061d934 P 4125b1a70e864e129bd54bdc2720d7fa [term 1 FOLLOWER]: Committing config change with OpId 1.99: config changed from index 96 to 99, VOTER 6568f69a78c44bb985162d2b7d0a9376 (127.0.99.134) evicted. New config: { opid_index: 99 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:07.221056 3035 tablet_service.cc:1514] Processing DeleteTablet for tablet d4d07c9621134780bc7ece7bb061d934 with delete_type TABLET_DATA_TOMBSTONED (TS 6568f69a78c44bb985162d2b7d0a9376 not found in new config with opid_index 99) from {username='slave'} at 127.0.0.1:53118
I20240503 01:59:07.221251 4483 tablet_replica.cc:330] T d4d07c9621134780bc7ece7bb061d934 P 6568f69a78c44bb985162d2b7d0a9376: stopping tablet replica
I20240503 01:59:07.221383 4483 raft_consensus.cc:2229] T d4d07c9621134780bc7ece7bb061d934 P 6568f69a78c44bb985162d2b7d0a9376 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:07.223627 2396 raft_consensus.cc:1261] T d4d07c9621134780bc7ece7bb061d934 P b63928177fa54227a5ff9929265ab9dd [term 1 FOLLOWER]: Refusing update from remote peer 327aa26a8e434a609ebcb382aa665fce: Log matching property violated. Preceding OpId in replica: term: 1 index: 98. Preceding OpId from leader: term: 1 index: 99. (index mismatch)
I20240503 01:59:07.229177 4483 pending_rounds.cc:62] T d4d07c9621134780bc7ece7bb061d934 P 6568f69a78c44bb985162d2b7d0a9376: Trying to abort 1 pending ops.
I20240503 01:59:07.229274 4483 pending_rounds.cc:68] T d4d07c9621134780bc7ece7bb061d934 P 6568f69a78c44bb985162d2b7d0a9376: Aborting op as it isn't in flight: id { term: 1 index: 98 } timestamp: 7023417537180450816 op_type: WRITE_OP write_request { tablet_id: "d4d07c9621134780bc7ece7bb061d934" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000k\233\376\034.\350(\032\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""6f80f261f551eeef" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023417537142333440 authz_token { token_data: "\010\221\214\321\261\006\"3\n\005slave\022*\n e8f9ec3f55d044f8b23964df20e79f65\020\001\030\001 \001(\001" signature: "<redacted>""\206\310\236u\"\221x&\215\277\353{P\205\036\241\347\356\265\023\345\241r\n\246\035F\317\331\211K?Xe\r\312\217s@\275\302X\023\020\3352@\263\243\337O\254\255e\361\252\226\25513\263\363Wt" signing_key_seq_num: 0 } } request_id { client_id: "68fd0adb63cc4930b1199ed9322feb03" seq_no: 267 first_incomplete_seq_no: 267 attempt_no: 0 }
I20240503 01:59:07.229614 4483 raft_consensus.cc:2258] T d4d07c9621134780bc7ece7bb061d934 P 6568f69a78c44bb985162d2b7d0a9376 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:07.231398 4483 ts_tablet_manager.cc:1875] T d4d07c9621134780bc7ece7bb061d934 P 6568f69a78c44bb985162d2b7d0a9376: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:07.233781 4483 ts_tablet_manager.cc:1888] T d4d07c9621134780bc7ece7bb061d934 P 6568f69a78c44bb985162d2b7d0a9376: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.98
I20240503 01:59:07.233846 4483 log.cc:1196] T d4d07c9621134780bc7ece7bb061d934 P 6568f69a78c44bb985162d2b7d0a9376: Deleting WAL directory at /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/d4d07c9621134780bc7ece7bb061d934
I20240503 01:59:07.235740 2243 catalog_manager.cc:4830] TS 6568f69a78c44bb985162d2b7d0a9376 (127.0.99.134:43603): tablet d4d07c9621134780bc7ece7bb061d934 (table rebalance_test_table_4 [id=e8f9ec3f55d044f8b23964df20e79f65]) successfully deleted
I20240503 01:59:07.236428 4216 consensus_queue.cc:1035] T d4d07c9621134780bc7ece7bb061d934 P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 99, Last known committed idx: 97, Time since last communication: 0.000s
I20240503 01:59:07.258857 4506 tablet_bootstrap.cc:492] T 10ecacdfb9a44d068b0df24e991612ee P 4125b1a70e864e129bd54bdc2720d7fa: Bootstrap replayed 1/1 log segments. Stats: ops{read=54 overwritten=0 applied=54 ignored=0} inserts{seen=726 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:07.259213 4506 tablet_bootstrap.cc:492] T 10ecacdfb9a44d068b0df24e991612ee P 4125b1a70e864e129bd54bdc2720d7fa: Bootstrap complete.
I20240503 01:59:07.259418 4506 ts_tablet_manager.cc:1367] T 10ecacdfb9a44d068b0df24e991612ee P 4125b1a70e864e129bd54bdc2720d7fa: Time spent bootstrapping tablet: real 0.179s user 0.057s sys 0.007s
I20240503 01:59:07.259869 4506 raft_consensus.cc:348] T 10ecacdfb9a44d068b0df24e991612ee P 4125b1a70e864e129bd54bdc2720d7fa [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 51 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } attrs { replace: true } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: true } }
I20240503 01:59:07.260015 4506 raft_consensus.cc:729] T 10ecacdfb9a44d068b0df24e991612ee P 4125b1a70e864e129bd54bdc2720d7fa [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4125b1a70e864e129bd54bdc2720d7fa, State: Initialized, Role: LEARNER
I20240503 01:59:07.260134 4506 consensus_queue.cc:260] T 10ecacdfb9a44d068b0df24e991612ee P 4125b1a70e864e129bd54bdc2720d7fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 54, Last appended: 1.54, Last appended by leader: 54, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 51 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } attrs { replace: true } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: true } }
I20240503 01:59:07.261054 4506 ts_tablet_manager.cc:1398] T 10ecacdfb9a44d068b0df24e991612ee P 4125b1a70e864e129bd54bdc2720d7fa: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240503 01:59:07.261973 2944 tablet_copy_service.cc:342] P 327aa26a8e434a609ebcb382aa665fce: Request end of tablet copy session 4125b1a70e864e129bd54bdc2720d7fa-10ecacdfb9a44d068b0df24e991612ee received from {username='slave'} at 127.0.99.135:39289
I20240503 01:59:07.262053 2944 tablet_copy_service.cc:433] P 327aa26a8e434a609ebcb382aa665fce: ending tablet copy session 4125b1a70e864e129bd54bdc2720d7fa-10ecacdfb9a44d068b0df24e991612ee on tablet 10ecacdfb9a44d068b0df24e991612ee with peer 4125b1a70e864e129bd54bdc2720d7fa
W20240503 01:59:07.283427 2336 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.132:42887, user_credentials={real_user=slave}} blocked reactor thread for 49172.9us
I20240503 01:59:07.298331 2397 raft_consensus.cc:2932] T d4d07c9621134780bc7ece7bb061d934 P b63928177fa54227a5ff9929265ab9dd [term 1 FOLLOWER]: Committing config change with OpId 1.99: config changed from index 96 to 99, VOTER 6568f69a78c44bb985162d2b7d0a9376 (127.0.99.134) evicted. New config: { opid_index: 99 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:07.592455 3957 raft_consensus.cc:1203] T 10ecacdfb9a44d068b0df24e991612ee P 4125b1a70e864e129bd54bdc2720d7fa [term 1 LEARNER]: Deduplicated request from leader. Original: 1.50->[1.51-1.61] Dedup: 1.54->[1.55-1.61]
I20240503 01:59:07.605604 3055 consensus_queue.cc:237] T c2911cf48f384bf09074f7bea2a4c173 P 6568f69a78c44bb985162d2b7d0a9376 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 121, Committed index: 121, Last appended: 1.122, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 123 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: true } }
I20240503 01:59:07.607987 2924 raft_consensus.cc:1261] T c2911cf48f384bf09074f7bea2a4c173 P 327aa26a8e434a609ebcb382aa665fce [term 1 FOLLOWER]: Refusing update from remote peer 6568f69a78c44bb985162d2b7d0a9376: Log matching property violated. Preceding OpId in replica: term: 1 index: 122. Preceding OpId from leader: term: 1 index: 123. (index mismatch)
I20240503 01:59:07.608856 4112 consensus_queue.cc:1035] T c2911cf48f384bf09074f7bea2a4c173 P 6568f69a78c44bb985162d2b7d0a9376 [LEADER]: Connected to new peer: Peer: permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 123, Last known committed idx: 121, Time since last communication: 0.000s
I20240503 01:59:07.611403 3870 raft_consensus.cc:1050] T 10ecacdfb9a44d068b0df24e991612ee P 327aa26a8e434a609ebcb382aa665fce: attempting to promote NON_VOTER 4125b1a70e864e129bd54bdc2720d7fa to VOTER
I20240503 01:59:07.611790 3870 consensus_queue.cc:237] T 10ecacdfb9a44d068b0df24e991612ee P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 61, Committed index: 61, Last appended: 1.61, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 62 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } attrs { replace: true } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } }
I20240503 01:59:07.614581 2530 raft_consensus.cc:1261] T 10ecacdfb9a44d068b0df24e991612ee P a1a844ecd71145e4a362f4c42500fbcb [term 1 FOLLOWER]: Refusing update from remote peer 327aa26a8e434a609ebcb382aa665fce: Log matching property violated. Preceding OpId in replica: term: 1 index: 61. Preceding OpId from leader: term: 1 index: 62. (index mismatch)
I20240503 01:59:07.614814 2789 raft_consensus.cc:1261] T 10ecacdfb9a44d068b0df24e991612ee P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Refusing update from remote peer 327aa26a8e434a609ebcb382aa665fce: Log matching property violated. Preceding OpId in replica: term: 1 index: 61. Preceding OpId from leader: term: 1 index: 62. (index mismatch)
I20240503 01:59:07.614938 3870 consensus_queue.cc:1035] T 10ecacdfb9a44d068b0df24e991612ee P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 62, Last known committed idx: 61, Time since last communication: 0.000s
I20240503 01:59:07.615399 2530 raft_consensus.cc:1261] T c2911cf48f384bf09074f7bea2a4c173 P a1a844ecd71145e4a362f4c42500fbcb [term 1 FOLLOWER]: Refusing update from remote peer 6568f69a78c44bb985162d2b7d0a9376: Log matching property violated. Preceding OpId in replica: term: 1 index: 120. Preceding OpId from leader: term: 1 index: 123. (index mismatch)
I20240503 01:59:07.615880 4529 consensus_queue.cc:1035] T 10ecacdfb9a44d068b0df24e991612ee P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 62, Last known committed idx: 60, Time since last communication: 0.000s
I20240503 01:59:07.616425 3957 raft_consensus.cc:1261] T 10ecacdfb9a44d068b0df24e991612ee P 4125b1a70e864e129bd54bdc2720d7fa [term 1 LEARNER]: Refusing update from remote peer 327aa26a8e434a609ebcb382aa665fce: Log matching property violated. Preceding OpId in replica: term: 1 index: 61. Preceding OpId from leader: term: 1 index: 62. (index mismatch)
I20240503 01:59:07.617398 3870 consensus_queue.cc:1035] T 10ecacdfb9a44d068b0df24e991612ee P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 62, Last known committed idx: 61, Time since last communication: 0.000s
I20240503 01:59:07.617558 2397 raft_consensus.cc:1261] T c2911cf48f384bf09074f7bea2a4c173 P b63928177fa54227a5ff9929265ab9dd [term 1 FOLLOWER]: Refusing update from remote peer 6568f69a78c44bb985162d2b7d0a9376: Log matching property violated. Preceding OpId in replica: term: 1 index: 120. Preceding OpId from leader: term: 1 index: 123. (index mismatch)
I20240503 01:59:07.620043 4112 consensus_queue.cc:1035] T c2911cf48f384bf09074f7bea2a4c173 P 6568f69a78c44bb985162d2b7d0a9376 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 123, Last known committed idx: 119, Time since last communication: 0.000s
I20240503 01:59:07.619999 4216 raft_consensus.cc:2932] T 10ecacdfb9a44d068b0df24e991612ee P 327aa26a8e434a609ebcb382aa665fce [term 1 LEADER]: Committing config change with OpId 1.62: config changed from index 51 to 62, 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 62 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } attrs { replace: true } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:07.620539 4112 consensus_queue.cc:1035] T c2911cf48f384bf09074f7bea2a4c173 P 6568f69a78c44bb985162d2b7d0a9376 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 123, Last known committed idx: 119, Time since last communication: 0.000s
I20240503 01:59:07.622351 2787 raft_consensus.cc:2932] T 10ecacdfb9a44d068b0df24e991612ee P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Committing config change with OpId 1.62: config changed from index 51 to 62, 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 62 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } attrs { replace: true } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:07.623332 2662 raft_consensus.cc:1261] T 10ecacdfb9a44d068b0df24e991612ee P b40a9a7b30d34a6db007ed8febd8a759 [term 1 FOLLOWER]: Refusing update from remote peer 327aa26a8e434a609ebcb382aa665fce: Log matching property violated. Preceding OpId in replica: term: 1 index: 61. Preceding OpId from leader: term: 1 index: 62. (index mismatch)
I20240503 01:59:07.624573 2660 raft_consensus.cc:1261] T c2911cf48f384bf09074f7bea2a4c173 P b40a9a7b30d34a6db007ed8febd8a759 [term 1 FOLLOWER]: Refusing update from remote peer 6568f69a78c44bb985162d2b7d0a9376: Log matching property violated. Preceding OpId in replica: term: 1 index: 121. Preceding OpId from leader: term: 1 index: 123. (index mismatch)
I20240503 01:59:07.624464 2525 raft_consensus.cc:2932] T 10ecacdfb9a44d068b0df24e991612ee P a1a844ecd71145e4a362f4c42500fbcb [term 1 FOLLOWER]: Committing config change with OpId 1.62: config changed from index 51 to 62, 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 62 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } attrs { replace: true } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:07.624716 2251 catalog_manager.cc:5484] T 10ecacdfb9a44d068b0df24e991612ee P 327aa26a8e434a609ebcb382aa665fce reported cstate change: config changed from index 51 to 62, 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "327aa26a8e434a609ebcb382aa665fce" committed_config { opid_index: 62 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:07.627238 4112 consensus_queue.cc:1035] T c2911cf48f384bf09074f7bea2a4c173 P 6568f69a78c44bb985162d2b7d0a9376 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 123, Last known committed idx: 121, Time since last communication: 0.000s
I20240503 01:59:07.628193 4498 consensus_queue.cc:1035] T 10ecacdfb9a44d068b0df24e991612ee P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 62, Last known committed idx: 61, Time since last communication: 0.000s
I20240503 01:59:07.629956 4487 raft_consensus.cc:2932] T c2911cf48f384bf09074f7bea2a4c173 P 6568f69a78c44bb985162d2b7d0a9376 [term 1 LEADER]: Committing config change with OpId 1.123: config changed from index -1 to 123, NON_VOTER 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) added. New config: { opid_index: 123 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: true } } }
I20240503 01:59:07.631960 3957 raft_consensus.cc:2932] T 10ecacdfb9a44d068b0df24e991612ee P 4125b1a70e864e129bd54bdc2720d7fa [term 1 FOLLOWER]: Committing config change with OpId 1.62: config changed from index 51 to 62, 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 62 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } attrs { replace: true } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:07.634779 2660 raft_consensus.cc:2932] T 10ecacdfb9a44d068b0df24e991612ee P b40a9a7b30d34a6db007ed8febd8a759 [term 1 FOLLOWER]: Committing config change with OpId 1.62: config changed from index 51 to 62, 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 62 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } attrs { replace: true } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:07.636384 2920 raft_consensus.cc:2932] T c2911cf48f384bf09074f7bea2a4c173 P 327aa26a8e434a609ebcb382aa665fce [term 1 FOLLOWER]: Committing config change with OpId 1.123: config changed from index -1 to 123, NON_VOTER 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) added. New config: { opid_index: 123 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: true } } }
I20240503 01:59:07.640129 2530 raft_consensus.cc:2932] T c2911cf48f384bf09074f7bea2a4c173 P a1a844ecd71145e4a362f4c42500fbcb [term 1 FOLLOWER]: Committing config change with OpId 1.123: config changed from index -1 to 123, NON_VOTER 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) added. New config: { opid_index: 123 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: true } } }
I20240503 01:59:07.640300 2250 catalog_manager.cc:5484] T c2911cf48f384bf09074f7bea2a4c173 P 6568f69a78c44bb985162d2b7d0a9376 reported cstate change: config changed from index -1 to 123, NON_VOTER 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "6568f69a78c44bb985162d2b7d0a9376" committed_config { opid_index: 123 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:07.642669 2398 raft_consensus.cc:1261] T 10ecacdfb9a44d068b0df24e991612ee P b63928177fa54227a5ff9929265ab9dd [term 1 FOLLOWER]: Refusing update from remote peer 327aa26a8e434a609ebcb382aa665fce: Log matching property violated. Preceding OpId in replica: term: 1 index: 61. Preceding OpId from leader: term: 1 index: 62. (index mismatch)
I20240503 01:59:07.644184 2662 raft_consensus.cc:2932] T c2911cf48f384bf09074f7bea2a4c173 P b40a9a7b30d34a6db007ed8febd8a759 [term 1 FOLLOWER]: Committing config change with OpId 1.123: config changed from index -1 to 123, NON_VOTER 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) added. New config: { opid_index: 123 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: true } } }
I20240503 01:59:07.646071 2396 raft_consensus.cc:2932] T c2911cf48f384bf09074f7bea2a4c173 P b63928177fa54227a5ff9929265ab9dd [term 1 FOLLOWER]: Committing config change with OpId 1.123: config changed from index -1 to 123, NON_VOTER 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) added. New config: { opid_index: 123 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: true } } }
W20240503 01:59:07.714326 2857 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.131:34079, user_credentials={real_user=slave}} blocked reactor thread for 66887.5us
W20240503 01:59:07.715054 2858 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.134:43603, user_credentials={real_user=slave}} blocked reactor thread for 67211.6us
W20240503 01:59:07.715106 2860 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.135:39501, user_credentials={real_user=slave}} blocked reactor thread for 67477.5us
I20240503 01:59:07.716429 4216 consensus_queue.cc:1035] T 10ecacdfb9a44d068b0df24e991612ee P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 62, Last known committed idx: 61, Time since last communication: 0.000s
W20240503 01:59:07.722887 2991 consensus_peers.cc:485] T c2911cf48f384bf09074f7bea2a4c173 P 6568f69a78c44bb985162d2b7d0a9376 -> Peer 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135:39501): Couldn't send request to peer 4125b1a70e864e129bd54bdc2720d7fa. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c2911cf48f384bf09074f7bea2a4c173. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:07.740016 3054 consensus_queue.cc:237] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 6568f69a78c44bb985162d2b7d0a9376 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 63, Committed index: 63, Last appended: 1.63, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 64 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { replace: true } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: true } }
I20240503 01:59:07.747701 2791 raft_consensus.cc:1261] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Refusing update from remote peer 6568f69a78c44bb985162d2b7d0a9376: Log matching property violated. Preceding OpId in replica: term: 1 index: 63. Preceding OpId from leader: term: 1 index: 64. (index mismatch)
I20240503 01:59:07.748363 2921 raft_consensus.cc:1261] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 327aa26a8e434a609ebcb382aa665fce [term 1 FOLLOWER]: Refusing update from remote peer 6568f69a78c44bb985162d2b7d0a9376: Log matching property violated. Preceding OpId in replica: term: 1 index: 63. Preceding OpId from leader: term: 1 index: 64. (index mismatch)
I20240503 01:59:07.748862 2399 raft_consensus.cc:1261] T 8682bff56e604cc3b7e9b07c8ba3fca5 P b63928177fa54227a5ff9929265ab9dd [term 1 FOLLOWER]: Refusing update from remote peer 6568f69a78c44bb985162d2b7d0a9376: Log matching property violated. Preceding OpId in replica: term: 1 index: 63. Preceding OpId from leader: term: 1 index: 64. (index mismatch)
I20240503 01:59:07.751777 3726 consensus_queue.cc:1035] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 6568f69a78c44bb985162d2b7d0a9376 [LEADER]: Connected to new peer: Peer: permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 64, Last known committed idx: 63, Time since last communication: 0.000s
I20240503 01:59:07.760867 2662 raft_consensus.cc:1261] T 8682bff56e604cc3b7e9b07c8ba3fca5 P b40a9a7b30d34a6db007ed8febd8a759 [term 1 FOLLOWER]: Refusing update from remote peer 6568f69a78c44bb985162d2b7d0a9376: Log matching property violated. Preceding OpId in replica: term: 1 index: 63. Preceding OpId from leader: term: 1 index: 64. (index mismatch)
I20240503 01:59:07.760890 4488 consensus_queue.cc:1035] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 6568f69a78c44bb985162d2b7d0a9376 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 64, Last known committed idx: 63, Time since last communication: 0.000s
I20240503 01:59:07.761633 4488 consensus_queue.cc:1035] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 6568f69a78c44bb985162d2b7d0a9376 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 64, Last known committed idx: 63, Time since last communication: 0.000s
I20240503 01:59:07.762369 4488 consensus_queue.cc:1035] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 6568f69a78c44bb985162d2b7d0a9376 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 64, Last known committed idx: 63, Time since last communication: 0.000s
W20240503 01:59:07.763322 2991 consensus_peers.cc:485] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 6568f69a78c44bb985162d2b7d0a9376 -> Peer 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135:39501): Couldn't send request to peer 4125b1a70e864e129bd54bdc2720d7fa. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8682bff56e604cc3b7e9b07c8ba3fca5. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:07.766180 3726 raft_consensus.cc:2932] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 6568f69a78c44bb985162d2b7d0a9376 [term 1 LEADER]: Committing config change with OpId 1.64: config changed from index -1 to 64, NON_VOTER 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) added. New config: { opid_index: 64 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { replace: true } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: true } } }
I20240503 01:59:07.771127 2398 raft_consensus.cc:2932] T 10ecacdfb9a44d068b0df24e991612ee P b63928177fa54227a5ff9929265ab9dd [term 1 FOLLOWER]: Committing config change with OpId 1.62: config changed from index 51 to 62, 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 62 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } attrs { replace: true } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:07.770977 2661 raft_consensus.cc:2932] T 8682bff56e604cc3b7e9b07c8ba3fca5 P b40a9a7b30d34a6db007ed8febd8a759 [term 1 FOLLOWER]: Committing config change with OpId 1.64: config changed from index -1 to 64, NON_VOTER 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) added. New config: { opid_index: 64 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { replace: true } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: true } } }
I20240503 01:59:07.777307 2920 raft_consensus.cc:2932] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 327aa26a8e434a609ebcb382aa665fce [term 1 FOLLOWER]: Committing config change with OpId 1.64: config changed from index -1 to 64, NON_VOTER 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) added. New config: { opid_index: 64 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { replace: true } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: true } } }
I20240503 01:59:07.784193 2251 catalog_manager.cc:5484] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 6568f69a78c44bb985162d2b7d0a9376 reported cstate change: config changed from index -1 to 64, NON_VOTER 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "6568f69a78c44bb985162d2b7d0a9376" committed_config { opid_index: 64 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:07.789649 2787 raft_consensus.cc:2932] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Committing config change with OpId 1.64: config changed from index -1 to 64, NON_VOTER 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) added. New config: { opid_index: 64 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { replace: true } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: true } } }
I20240503 01:59:07.818439 2397 raft_consensus.cc:2932] T 8682bff56e604cc3b7e9b07c8ba3fca5 P b63928177fa54227a5ff9929265ab9dd [term 1 FOLLOWER]: Committing config change with OpId 1.64: config changed from index -1 to 64, NON_VOTER 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) added. New config: { opid_index: 64 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { replace: true } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: true } } }
I20240503 01:59:07.830472 2921 consensus_queue.cc:237] T 10ecacdfb9a44d068b0df24e991612ee P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 64, Committed index: 64, Last appended: 1.65, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } }
I20240503 01:59:07.833374 3958 raft_consensus.cc:1261] T 10ecacdfb9a44d068b0df24e991612ee P 4125b1a70e864e129bd54bdc2720d7fa [term 1 FOLLOWER]: Refusing update from remote peer 327aa26a8e434a609ebcb382aa665fce: Log matching property violated. Preceding OpId in replica: term: 1 index: 65. Preceding OpId from leader: term: 1 index: 66. (index mismatch)
I20240503 01:59:07.833711 2395 raft_consensus.cc:1261] T 10ecacdfb9a44d068b0df24e991612ee P b63928177fa54227a5ff9929265ab9dd [term 1 FOLLOWER]: Refusing update from remote peer 327aa26a8e434a609ebcb382aa665fce: Log matching property violated. Preceding OpId in replica: term: 1 index: 63. Preceding OpId from leader: term: 1 index: 66. (index mismatch)
I20240503 01:59:07.838188 4214 consensus_queue.cc:1035] T 10ecacdfb9a44d068b0df24e991612ee P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 66, Last known committed idx: 64, Time since last communication: 0.000s
I20240503 01:59:07.842360 4214 consensus_queue.cc:1035] T 10ecacdfb9a44d068b0df24e991612ee P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 66, Last known committed idx: 62, Time since last communication: 0.000s
I20240503 01:59:07.850351 4214 raft_consensus.cc:2932] T 10ecacdfb9a44d068b0df24e991612ee P 327aa26a8e434a609ebcb382aa665fce [term 1 LEADER]: Committing config change with OpId 1.66: config changed from index 62 to 66, VOTER 52c196c15e3348b985c689652bc49918 (127.0.99.132) evicted. New config: { opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:07.852102 2243 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 10ecacdfb9a44d068b0df24e991612ee with cas_config_opid_index 62: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:07.852878 2398 raft_consensus.cc:1203] T 10ecacdfb9a44d068b0df24e991612ee P b63928177fa54227a5ff9929265ab9dd [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.63->[1.64-1.65] Dedup: 1.65->[]
I20240503 01:59:07.853266 2525 raft_consensus.cc:1261] T 10ecacdfb9a44d068b0df24e991612ee P a1a844ecd71145e4a362f4c42500fbcb [term 1 FOLLOWER]: Refusing update from remote peer 327aa26a8e434a609ebcb382aa665fce: Log matching property violated. Preceding OpId in replica: term: 1 index: 65. Preceding OpId from leader: term: 1 index: 66. (index mismatch)
I20240503 01:59:07.855144 2661 raft_consensus.cc:1261] T 10ecacdfb9a44d068b0df24e991612ee P b40a9a7b30d34a6db007ed8febd8a759 [term 1 FOLLOWER]: Refusing update from remote peer 327aa26a8e434a609ebcb382aa665fce: Log matching property violated. Preceding OpId in replica: term: 1 index: 65. Preceding OpId from leader: term: 1 index: 66. (index mismatch)
I20240503 01:59:07.856113 3870 consensus_queue.cc:1035] T 10ecacdfb9a44d068b0df24e991612ee P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 66, Last known committed idx: 64, Time since last communication: 0.000s
I20240503 01:59:07.861241 2661 raft_consensus.cc:2932] T 10ecacdfb9a44d068b0df24e991612ee P b40a9a7b30d34a6db007ed8febd8a759 [term 1 FOLLOWER]: Committing config change with OpId 1.66: config changed from index 62 to 66, VOTER 52c196c15e3348b985c689652bc49918 (127.0.99.132) evicted. New config: { opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:07.864964 3955 raft_consensus.cc:2932] T 10ecacdfb9a44d068b0df24e991612ee P 4125b1a70e864e129bd54bdc2720d7fa [term 1 FOLLOWER]: Committing config change with OpId 1.66: config changed from index 62 to 66, VOTER 52c196c15e3348b985c689652bc49918 (127.0.99.132) evicted. New config: { opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:07.865334 2253 catalog_manager.cc:5484] T 10ecacdfb9a44d068b0df24e991612ee P 327aa26a8e434a609ebcb382aa665fce reported cstate change: config changed from index 62 to 66, VOTER 52c196c15e3348b985c689652bc49918 (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "327aa26a8e434a609ebcb382aa665fce" committed_config { opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:07.868443 2772 tablet_service.cc:1514] Processing DeleteTablet for tablet 10ecacdfb9a44d068b0df24e991612ee with delete_type TABLET_DATA_TOMBSTONED (TS 52c196c15e3348b985c689652bc49918 not found in new config with opid_index 66) from {username='slave'} at 127.0.0.1:45406
I20240503 01:59:07.869171 4527 consensus_queue.cc:1035] T 10ecacdfb9a44d068b0df24e991612ee P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 66, Last known committed idx: 64, Time since last communication: 0.000s
I20240503 01:59:07.871547 2520 raft_consensus.cc:2932] T 10ecacdfb9a44d068b0df24e991612ee P a1a844ecd71145e4a362f4c42500fbcb [term 1 FOLLOWER]: Committing config change with OpId 1.66: config changed from index 62 to 66, VOTER 52c196c15e3348b985c689652bc49918 (127.0.99.132) evicted. New config: { opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:07.875577 4555 tablet_replica.cc:330] T 10ecacdfb9a44d068b0df24e991612ee P 52c196c15e3348b985c689652bc49918: stopping tablet replica
I20240503 01:59:07.875989 4555 raft_consensus.cc:2229] T 10ecacdfb9a44d068b0df24e991612ee P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:07.877791 4555 pending_rounds.cc:62] T 10ecacdfb9a44d068b0df24e991612ee P 52c196c15e3348b985c689652bc49918: Trying to abort 1 pending ops.
I20240503 01:59:07.882756 4555 pending_rounds.cc:68] T 10ecacdfb9a44d068b0df24e991612ee P 52c196c15e3348b985c689652bc49918: Aborting op as it isn't in flight: id { term: 1 index: 65 } timestamp: 7023417539847815168 op_type: WRITE_OP write_request { tablet_id: "10ecacdfb9a44d068b0df24e991612ee" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000]]\225j\005\005\207\227\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""61428e73b12ca3cf" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023417539764547584 authz_token { token_data: "\010\221\214\321\261\006\"3\n\005slave\022*\n f7b0d84dbe0541aaa2606d6e366719c8\020\001\030\001 \001(\001" signature: "<redacted>""\217jtB\3732\304\336xK^\n$\214\"\256\361\355\365/@\211\346\020k^2\312+\226F\356|\322C\220\335P\030\202\245Deb0\022\360\205\323\034\367\020\343\200}\236-O.UT\307\231\035" signing_key_seq_num: 0 } } request_id { client_id: "bb6121f1bc824d9ab40bab73a1bea25e" seq_no: 175 first_incomplete_seq_no: 175 attempt_no: 0 }
I20240503 01:59:07.883199 4555 raft_consensus.cc:2258] T 10ecacdfb9a44d068b0df24e991612ee P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:07.883991 4555 ts_tablet_manager.cc:1875] T 10ecacdfb9a44d068b0df24e991612ee P 52c196c15e3348b985c689652bc49918: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:07.884604 2398 raft_consensus.cc:2932] T 10ecacdfb9a44d068b0df24e991612ee P b63928177fa54227a5ff9929265ab9dd [term 1 FOLLOWER]: Committing config change with OpId 1.66: config changed from index 62 to 66, VOTER 52c196c15e3348b985c689652bc49918 (127.0.99.132) evicted. New config: { opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:07.887869 4555 ts_tablet_manager.cc:1888] T 10ecacdfb9a44d068b0df24e991612ee P 52c196c15e3348b985c689652bc49918: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.65
I20240503 01:59:07.892282 4555 log.cc:1196] T 10ecacdfb9a44d068b0df24e991612ee P 52c196c15e3348b985c689652bc49918: Deleting WAL directory at /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/10ecacdfb9a44d068b0df24e991612ee
I20240503 01:59:07.892916 2245 catalog_manager.cc:4830] TS 52c196c15e3348b985c689652bc49918 (127.0.99.132:42887): tablet 10ecacdfb9a44d068b0df24e991612ee (table rebalance_test_table_3 [id=f7b0d84dbe0541aaa2606d6e366719c8]) successfully deleted
I20240503 01:59:07.906694 2524 consensus_queue.cc:237] T 43008abe74604666b330d78ec0f51c7c P a1a844ecd71145e4a362f4c42500fbcb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 133, Committed index: 133, Last appended: 1.133, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 134 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { replace: true } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: true } }
I20240503 01:59:07.936764 2397 raft_consensus.cc:1261] T 43008abe74604666b330d78ec0f51c7c P b63928177fa54227a5ff9929265ab9dd [term 1 FOLLOWER]: Refusing update from remote peer a1a844ecd71145e4a362f4c42500fbcb: Log matching property violated. Preceding OpId in replica: term: 1 index: 133. Preceding OpId from leader: term: 1 index: 135. (index mismatch)
I20240503 01:59:07.937094 3051 raft_consensus.cc:1261] T 43008abe74604666b330d78ec0f51c7c P 6568f69a78c44bb985162d2b7d0a9376 [term 1 FOLLOWER]: Refusing update from remote peer a1a844ecd71145e4a362f4c42500fbcb: Log matching property violated. Preceding OpId in replica: term: 1 index: 133. Preceding OpId from leader: term: 1 index: 135. (index mismatch)
I20240503 01:59:07.937660 4030 consensus_queue.cc:1035] T 43008abe74604666b330d78ec0f51c7c P a1a844ecd71145e4a362f4c42500fbcb [LEADER]: Connected to new peer: Peer: permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 134, Last known committed idx: 133, Time since last communication: 0.000s
I20240503 01:59:07.940368 2659 raft_consensus.cc:1261] T 43008abe74604666b330d78ec0f51c7c P b40a9a7b30d34a6db007ed8febd8a759 [term 1 FOLLOWER]: Refusing update from remote peer a1a844ecd71145e4a362f4c42500fbcb: Log matching property violated. Preceding OpId in replica: term: 1 index: 133. Preceding OpId from leader: term: 1 index: 135. (index mismatch)
I20240503 01:59:07.940817 4030 consensus_queue.cc:1035] T 43008abe74604666b330d78ec0f51c7c P a1a844ecd71145e4a362f4c42500fbcb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 134, Last known committed idx: 133, Time since last communication: 0.000s
I20240503 01:59:07.941298 2921 raft_consensus.cc:1261] T 43008abe74604666b330d78ec0f51c7c P 327aa26a8e434a609ebcb382aa665fce [term 1 FOLLOWER]: Refusing update from remote peer a1a844ecd71145e4a362f4c42500fbcb: Log matching property violated. Preceding OpId in replica: term: 1 index: 133. Preceding OpId from leader: term: 1 index: 135. (index mismatch)
I20240503 01:59:07.942585 4030 consensus_queue.cc:1035] T 43008abe74604666b330d78ec0f51c7c P a1a844ecd71145e4a362f4c42500fbcb [LEADER]: Connected to new peer: Peer: permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 134, Last known committed idx: 132, Time since last communication: 0.000s
I20240503 01:59:07.945225 4030 raft_consensus.cc:2932] T 43008abe74604666b330d78ec0f51c7c P a1a844ecd71145e4a362f4c42500fbcb [term 1 LEADER]: Committing config change with OpId 1.134: config changed from index -1 to 134, NON_VOTER 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) added. New config: { opid_index: 134 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { replace: true } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: true } } }
I20240503 01:59:07.946527 4333 consensus_queue.cc:1035] T 43008abe74604666b330d78ec0f51c7c P a1a844ecd71145e4a362f4c42500fbcb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 134, Last known committed idx: 133, Time since last communication: 0.000s
I20240503 01:59:07.949057 3055 raft_consensus.cc:2932] T 43008abe74604666b330d78ec0f51c7c P 6568f69a78c44bb985162d2b7d0a9376 [term 1 FOLLOWER]: Committing config change with OpId 1.134: config changed from index -1 to 134, NON_VOTER 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) added. New config: { opid_index: 134 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { replace: true } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: true } } }
I20240503 01:59:07.951973 2659 raft_consensus.cc:2932] T 43008abe74604666b330d78ec0f51c7c P b40a9a7b30d34a6db007ed8febd8a759 [term 1 FOLLOWER]: Committing config change with OpId 1.134: config changed from index -1 to 134, NON_VOTER 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) added. New config: { opid_index: 134 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { replace: true } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: true } } }
I20240503 01:59:07.952325 2253 catalog_manager.cc:5484] T 43008abe74604666b330d78ec0f51c7c P 6568f69a78c44bb985162d2b7d0a9376 reported cstate change: config changed from index -1 to 134, NON_VOTER 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "a1a844ecd71145e4a362f4c42500fbcb" committed_config { opid_index: 134 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { replace: true } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: true } } }
I20240503 01:59:07.959578 2394 raft_consensus.cc:2932] T 43008abe74604666b330d78ec0f51c7c P b63928177fa54227a5ff9929265ab9dd [term 1 FOLLOWER]: Committing config change with OpId 1.134: config changed from index -1 to 134, NON_VOTER 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) added. New config: { opid_index: 134 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { replace: true } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: true } } }
I20240503 01:59:07.968312 2921 raft_consensus.cc:2932] T 43008abe74604666b330d78ec0f51c7c P 327aa26a8e434a609ebcb382aa665fce [term 1 FOLLOWER]: Committing config change with OpId 1.134: config changed from index -1 to 134, NON_VOTER 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) added. New config: { opid_index: 134 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { replace: true } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: true } } }
I20240503 01:59:08.227360 4571 ts_tablet_manager.cc:902] T c2911cf48f384bf09074f7bea2a4c173 P 4125b1a70e864e129bd54bdc2720d7fa: Initiating tablet copy from peer 6568f69a78c44bb985162d2b7d0a9376 (127.0.99.134:43603)
I20240503 01:59:08.234203 4571 tablet_copy_client.cc:308] T c2911cf48f384bf09074f7bea2a4c173 P 4125b1a70e864e129bd54bdc2720d7fa: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.134:43603
I20240503 01:59:08.275604 3075 tablet_copy_service.cc:140] P 6568f69a78c44bb985162d2b7d0a9376: Received BeginTabletCopySession request for tablet c2911cf48f384bf09074f7bea2a4c173 from peer 4125b1a70e864e129bd54bdc2720d7fa ({username='slave'} at 127.0.99.135:33083)
I20240503 01:59:08.275769 3075 tablet_copy_service.cc:161] P 6568f69a78c44bb985162d2b7d0a9376: Beginning new tablet copy session on tablet c2911cf48f384bf09074f7bea2a4c173 from peer 4125b1a70e864e129bd54bdc2720d7fa at {username='slave'} at 127.0.99.135:33083: session id = 4125b1a70e864e129bd54bdc2720d7fa-c2911cf48f384bf09074f7bea2a4c173
I20240503 01:59:08.279035 3075 tablet_copy_source_session.cc:205] T c2911cf48f384bf09074f7bea2a4c173 P 6568f69a78c44bb985162d2b7d0a9376: Tablet Copy: opened 0 blocks and 1 log segments
W20240503 01:59:08.279692 2467 consensus_peers.cc:485] T 43008abe74604666b330d78ec0f51c7c P a1a844ecd71145e4a362f4c42500fbcb -> Peer 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135:39501): Couldn't send request to peer 4125b1a70e864e129bd54bdc2720d7fa. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 43008abe74604666b330d78ec0f51c7c. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:08.281008 4571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2911cf48f384bf09074f7bea2a4c173. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:08.288491 4571 tablet_copy_client.cc:783] T c2911cf48f384bf09074f7bea2a4c173 P 4125b1a70e864e129bd54bdc2720d7fa: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:08.288710 4571 tablet_copy_client.cc:647] T c2911cf48f384bf09074f7bea2a4c173 P 4125b1a70e864e129bd54bdc2720d7fa: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:08.292910 4571 tablet_copy_client.cc:523] T c2911cf48f384bf09074f7bea2a4c173 P 4125b1a70e864e129bd54bdc2720d7fa: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:08.294401 4571 tablet_bootstrap.cc:492] T c2911cf48f384bf09074f7bea2a4c173 P 4125b1a70e864e129bd54bdc2720d7fa: Bootstrap starting.
I20240503 01:59:08.359656 4582 ts_tablet_manager.cc:902] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 4125b1a70e864e129bd54bdc2720d7fa: Initiating tablet copy from peer 6568f69a78c44bb985162d2b7d0a9376 (127.0.99.134:43603)
I20240503 01:59:08.366930 4582 tablet_copy_client.cc:308] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 4125b1a70e864e129bd54bdc2720d7fa: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.134:43603
I20240503 01:59:08.367424 3075 tablet_copy_service.cc:140] P 6568f69a78c44bb985162d2b7d0a9376: Received BeginTabletCopySession request for tablet 8682bff56e604cc3b7e9b07c8ba3fca5 from peer 4125b1a70e864e129bd54bdc2720d7fa ({username='slave'} at 127.0.99.135:33083)
I20240503 01:59:08.367538 3075 tablet_copy_service.cc:161] P 6568f69a78c44bb985162d2b7d0a9376: Beginning new tablet copy session on tablet 8682bff56e604cc3b7e9b07c8ba3fca5 from peer 4125b1a70e864e129bd54bdc2720d7fa at {username='slave'} at 127.0.99.135:33083: session id = 4125b1a70e864e129bd54bdc2720d7fa-8682bff56e604cc3b7e9b07c8ba3fca5
I20240503 01:59:08.368546 3075 tablet_copy_source_session.cc:205] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 6568f69a78c44bb985162d2b7d0a9376: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:08.369632 4582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8682bff56e604cc3b7e9b07c8ba3fca5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:08.373649 4582 tablet_copy_client.cc:783] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 4125b1a70e864e129bd54bdc2720d7fa: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:08.374163 4582 tablet_copy_client.cc:647] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 4125b1a70e864e129bd54bdc2720d7fa: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:08.377629 4582 tablet_copy_client.cc:523] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 4125b1a70e864e129bd54bdc2720d7fa: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:08.379505 4582 tablet_bootstrap.cc:492] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 4125b1a70e864e129bd54bdc2720d7fa: Bootstrap starting.
I20240503 01:59:08.456549 4592 ts_tablet_manager.cc:902] T 43008abe74604666b330d78ec0f51c7c P 4125b1a70e864e129bd54bdc2720d7fa: Initiating tablet copy from peer a1a844ecd71145e4a362f4c42500fbcb (127.0.99.130:46033)
I20240503 01:59:08.457295 4592 tablet_copy_client.cc:308] T 43008abe74604666b330d78ec0f51c7c P 4125b1a70e864e129bd54bdc2720d7fa: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:46033
I20240503 01:59:08.463214 2551 tablet_copy_service.cc:140] P a1a844ecd71145e4a362f4c42500fbcb: Received BeginTabletCopySession request for tablet 43008abe74604666b330d78ec0f51c7c from peer 4125b1a70e864e129bd54bdc2720d7fa ({username='slave'} at 127.0.99.135:46707)
I20240503 01:59:08.463335 2551 tablet_copy_service.cc:161] P a1a844ecd71145e4a362f4c42500fbcb: Beginning new tablet copy session on tablet 43008abe74604666b330d78ec0f51c7c from peer 4125b1a70e864e129bd54bdc2720d7fa at {username='slave'} at 127.0.99.135:46707: session id = 4125b1a70e864e129bd54bdc2720d7fa-43008abe74604666b330d78ec0f51c7c
I20240503 01:59:08.464797 2551 tablet_copy_source_session.cc:205] T 43008abe74604666b330d78ec0f51c7c P a1a844ecd71145e4a362f4c42500fbcb: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:08.470837 4592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43008abe74604666b330d78ec0f51c7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:08.511245 4592 tablet_copy_client.cc:783] T 43008abe74604666b330d78ec0f51c7c P 4125b1a70e864e129bd54bdc2720d7fa: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:08.511559 4592 tablet_copy_client.cc:647] T 43008abe74604666b330d78ec0f51c7c P 4125b1a70e864e129bd54bdc2720d7fa: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:08.515197 4592 tablet_copy_client.cc:523] T 43008abe74604666b330d78ec0f51c7c P 4125b1a70e864e129bd54bdc2720d7fa: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:08.517761 4592 tablet_bootstrap.cc:492] T 43008abe74604666b330d78ec0f51c7c P 4125b1a70e864e129bd54bdc2720d7fa: Bootstrap starting.
I20240503 01:59:08.770984 2661 consensus_queue.cc:237] T 0b669a47f1d14ce98ccab24479d39304 P b40a9a7b30d34a6db007ed8febd8a759 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 118, Committed index: 118, Last appended: 1.118, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 119 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { replace: true } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: true } }
I20240503 01:59:08.773252 2791 raft_consensus.cc:1261] T 0b669a47f1d14ce98ccab24479d39304 P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Refusing update from remote peer b40a9a7b30d34a6db007ed8febd8a759: Log matching property violated. Preceding OpId in replica: term: 1 index: 118. Preceding OpId from leader: term: 1 index: 119. (index mismatch)
W20240503 01:59:08.773397 2598 consensus_peers.cc:485] T 0b669a47f1d14ce98ccab24479d39304 P b40a9a7b30d34a6db007ed8febd8a759 -> Peer 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135:39501): Couldn't send request to peer 4125b1a70e864e129bd54bdc2720d7fa. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0b669a47f1d14ce98ccab24479d39304. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:08.773705 4327 consensus_queue.cc:1035] T 0b669a47f1d14ce98ccab24479d39304 P b40a9a7b30d34a6db007ed8febd8a759 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 119, Last known committed idx: 118, Time since last communication: 0.000s
I20240503 01:59:08.774415 3055 raft_consensus.cc:1261] T 0b669a47f1d14ce98ccab24479d39304 P 6568f69a78c44bb985162d2b7d0a9376 [term 1 FOLLOWER]: Refusing update from remote peer b40a9a7b30d34a6db007ed8febd8a759: Log matching property violated. Preceding OpId in replica: term: 1 index: 118. Preceding OpId from leader: term: 1 index: 119. (index mismatch)
I20240503 01:59:08.774914 4327 consensus_queue.cc:1035] T 0b669a47f1d14ce98ccab24479d39304 P b40a9a7b30d34a6db007ed8febd8a759 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 119, Last known committed idx: 118, Time since last communication: 0.000s
I20240503 01:59:08.775261 2525 raft_consensus.cc:1261] T 0b669a47f1d14ce98ccab24479d39304 P a1a844ecd71145e4a362f4c42500fbcb [term 1 FOLLOWER]: Refusing update from remote peer b40a9a7b30d34a6db007ed8febd8a759: Log matching property violated. Preceding OpId in replica: term: 1 index: 118. Preceding OpId from leader: term: 1 index: 119. (index mismatch)
I20240503 01:59:08.775671 3545 consensus_queue.cc:1035] T 0b669a47f1d14ce98ccab24479d39304 P b40a9a7b30d34a6db007ed8febd8a759 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 119, Last known committed idx: 118, Time since last communication: 0.000s
I20240503 01:59:08.779474 4551 raft_consensus.cc:2932] T 0b669a47f1d14ce98ccab24479d39304 P b40a9a7b30d34a6db007ed8febd8a759 [term 1 LEADER]: Committing config change with OpId 1.119: config changed from index -1 to 119, NON_VOTER 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) added. New config: { opid_index: 119 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { replace: true } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: true } } }
I20240503 01:59:08.781929 2791 raft_consensus.cc:2932] T 0b669a47f1d14ce98ccab24479d39304 P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Committing config change with OpId 1.119: config changed from index -1 to 119, NON_VOTER 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) added. New config: { opid_index: 119 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { replace: true } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: true } } }
I20240503 01:59:08.782274 2525 raft_consensus.cc:2932] T 0b669a47f1d14ce98ccab24479d39304 P a1a844ecd71145e4a362f4c42500fbcb [term 1 FOLLOWER]: Committing config change with OpId 1.119: config changed from index -1 to 119, NON_VOTER 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) added. New config: { opid_index: 119 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { replace: true } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: true } } }
I20240503 01:59:08.785624 2253 catalog_manager.cc:5484] T 0b669a47f1d14ce98ccab24479d39304 P b40a9a7b30d34a6db007ed8febd8a759 reported cstate change: config changed from index -1 to 119, NON_VOTER 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "b40a9a7b30d34a6db007ed8febd8a759" committed_config { opid_index: 119 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:08.787173 3051 raft_consensus.cc:2932] T 0b669a47f1d14ce98ccab24479d39304 P 6568f69a78c44bb985162d2b7d0a9376 [term 1 FOLLOWER]: Committing config change with OpId 1.119: config changed from index -1 to 119, NON_VOTER 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) added. New config: { opid_index: 119 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { replace: true } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: true } } }
I20240503 01:59:08.814682 4571 tablet_bootstrap.cc:492] T c2911cf48f384bf09074f7bea2a4c173 P 4125b1a70e864e129bd54bdc2720d7fa: Bootstrap replayed 1/1 log segments. Stats: ops{read=132 overwritten=0 applied=131 ignored=0} inserts{seen=762 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 01:59:08.815094 4571 tablet_bootstrap.cc:492] T c2911cf48f384bf09074f7bea2a4c173 P 4125b1a70e864e129bd54bdc2720d7fa: Bootstrap complete.
I20240503 01:59:08.815306 4571 ts_tablet_manager.cc:1367] T c2911cf48f384bf09074f7bea2a4c173 P 4125b1a70e864e129bd54bdc2720d7fa: Time spent bootstrapping tablet: real 0.521s user 0.112s sys 0.009s
I20240503 01:59:08.816115 4571 raft_consensus.cc:348] T c2911cf48f384bf09074f7bea2a4c173 P 4125b1a70e864e129bd54bdc2720d7fa [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 123 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: true } }
I20240503 01:59:08.816767 4571 raft_consensus.cc:729] T c2911cf48f384bf09074f7bea2a4c173 P 4125b1a70e864e129bd54bdc2720d7fa [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4125b1a70e864e129bd54bdc2720d7fa, State: Initialized, Role: LEARNER
I20240503 01:59:08.816900 4571 consensus_queue.cc:260] T c2911cf48f384bf09074f7bea2a4c173 P 4125b1a70e864e129bd54bdc2720d7fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 131, Last appended: 1.132, Last appended by leader: 132, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 123 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: true } }
I20240503 01:59:08.817266 4571 ts_tablet_manager.cc:1398] T c2911cf48f384bf09074f7bea2a4c173 P 4125b1a70e864e129bd54bdc2720d7fa: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240503 01:59:08.818161 3075 tablet_copy_service.cc:342] P 6568f69a78c44bb985162d2b7d0a9376: Request end of tablet copy session 4125b1a70e864e129bd54bdc2720d7fa-c2911cf48f384bf09074f7bea2a4c173 received from {username='slave'} at 127.0.99.135:33083
I20240503 01:59:08.818250 3075 tablet_copy_service.cc:433] P 6568f69a78c44bb985162d2b7d0a9376: ending tablet copy session 4125b1a70e864e129bd54bdc2720d7fa-c2911cf48f384bf09074f7bea2a4c173 on tablet c2911cf48f384bf09074f7bea2a4c173 with peer 4125b1a70e864e129bd54bdc2720d7fa
W20240503 01:59:08.821914 2336 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.135:39501, user_credentials={real_user=slave}} blocked reactor thread for 148501us
I20240503 01:59:08.828195 2398 raft_consensus.cc:1261] T 0b669a47f1d14ce98ccab24479d39304 P b63928177fa54227a5ff9929265ab9dd [term 1 FOLLOWER]: Refusing update from remote peer b40a9a7b30d34a6db007ed8febd8a759: Log matching property violated. Preceding OpId in replica: term: 1 index: 117. Preceding OpId from leader: term: 1 index: 119. (index mismatch)
I20240503 01:59:08.844086 4562 consensus_queue.cc:1035] T 0b669a47f1d14ce98ccab24479d39304 P b40a9a7b30d34a6db007ed8febd8a759 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 119, Last known committed idx: 116, Time since last communication: 0.000s
I20240503 01:59:08.847306 2393 raft_consensus.cc:2932] T 0b669a47f1d14ce98ccab24479d39304 P b63928177fa54227a5ff9929265ab9dd [term 1 FOLLOWER]: Committing config change with OpId 1.119: config changed from index -1 to 119, NON_VOTER 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) added. New config: { opid_index: 119 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { replace: true } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: true } } }
I20240503 01:59:08.848943 4592 tablet_bootstrap.cc:492] T 43008abe74604666b330d78ec0f51c7c P 4125b1a70e864e129bd54bdc2720d7fa: Bootstrap replayed 1/1 log segments. Stats: ops{read=145 overwritten=0 applied=145 ignored=0} inserts{seen=776 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:08.849269 4592 tablet_bootstrap.cc:492] T 43008abe74604666b330d78ec0f51c7c P 4125b1a70e864e129bd54bdc2720d7fa: Bootstrap complete.
I20240503 01:59:08.849541 4592 ts_tablet_manager.cc:1367] T 43008abe74604666b330d78ec0f51c7c P 4125b1a70e864e129bd54bdc2720d7fa: Time spent bootstrapping tablet: real 0.332s user 0.110s sys 0.013s
I20240503 01:59:08.850051 4592 raft_consensus.cc:348] T 43008abe74604666b330d78ec0f51c7c P 4125b1a70e864e129bd54bdc2720d7fa [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 134 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { replace: true } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: true } }
I20240503 01:59:08.850189 4592 raft_consensus.cc:729] T 43008abe74604666b330d78ec0f51c7c P 4125b1a70e864e129bd54bdc2720d7fa [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4125b1a70e864e129bd54bdc2720d7fa, State: Initialized, Role: LEARNER
I20240503 01:59:08.850306 4592 consensus_queue.cc:260] T 43008abe74604666b330d78ec0f51c7c P 4125b1a70e864e129bd54bdc2720d7fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 145, Last appended: 1.145, Last appended by leader: 145, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 134 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { replace: true } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: true } }
I20240503 01:59:08.850836 4592 ts_tablet_manager.cc:1398] T 43008abe74604666b330d78ec0f51c7c P 4125b1a70e864e129bd54bdc2720d7fa: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 01:59:08.851661 2551 tablet_copy_service.cc:342] P a1a844ecd71145e4a362f4c42500fbcb: Request end of tablet copy session 4125b1a70e864e129bd54bdc2720d7fa-43008abe74604666b330d78ec0f51c7c received from {username='slave'} at 127.0.99.135:46707
I20240503 01:59:08.851758 2551 tablet_copy_service.cc:433] P a1a844ecd71145e4a362f4c42500fbcb: ending tablet copy session 4125b1a70e864e129bd54bdc2720d7fa-43008abe74604666b330d78ec0f51c7c on tablet 43008abe74604666b330d78ec0f51c7c with peer 4125b1a70e864e129bd54bdc2720d7fa
I20240503 01:59:08.856328 3955 raft_consensus.cc:1203] T 43008abe74604666b330d78ec0f51c7c P 4125b1a70e864e129bd54bdc2720d7fa [term 1 LEARNER]: Deduplicated request from leader. Original: 1.133->[1.134-1.155] Dedup: 1.145->[1.146-1.155]
I20240503 01:59:09.018273 4582 tablet_bootstrap.cc:492] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 4125b1a70e864e129bd54bdc2720d7fa: Bootstrap replayed 1/1 log segments. Stats: ops{read=71 overwritten=0 applied=71 ignored=0} inserts{seen=702 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:09.018632 4582 tablet_bootstrap.cc:492] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 4125b1a70e864e129bd54bdc2720d7fa: Bootstrap complete.
I20240503 01:59:09.018846 4582 ts_tablet_manager.cc:1367] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 4125b1a70e864e129bd54bdc2720d7fa: Time spent bootstrapping tablet: real 0.639s user 0.066s sys 0.007s
I20240503 01:59:09.019343 4582 raft_consensus.cc:348] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 4125b1a70e864e129bd54bdc2720d7fa [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 64 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { replace: true } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: true } }
I20240503 01:59:09.019483 4582 raft_consensus.cc:729] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 4125b1a70e864e129bd54bdc2720d7fa [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4125b1a70e864e129bd54bdc2720d7fa, State: Initialized, Role: LEARNER
I20240503 01:59:09.019587 4582 consensus_queue.cc:260] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 4125b1a70e864e129bd54bdc2720d7fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 71, Last appended: 1.71, Last appended by leader: 71, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 64 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { replace: true } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: true } }
I20240503 01:59:09.020015 4582 ts_tablet_manager.cc:1398] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 4125b1a70e864e129bd54bdc2720d7fa: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:09.020442 3075 tablet_copy_service.cc:342] P 6568f69a78c44bb985162d2b7d0a9376: Request end of tablet copy session 4125b1a70e864e129bd54bdc2720d7fa-8682bff56e604cc3b7e9b07c8ba3fca5 received from {username='slave'} at 127.0.99.135:33083
I20240503 01:59:09.020537 3075 tablet_copy_service.cc:433] P 6568f69a78c44bb985162d2b7d0a9376: ending tablet copy session 4125b1a70e864e129bd54bdc2720d7fa-8682bff56e604cc3b7e9b07c8ba3fca5 on tablet 8682bff56e604cc3b7e9b07c8ba3fca5 with peer 4125b1a70e864e129bd54bdc2720d7fa
I20240503 01:59:09.049757 4028 raft_consensus.cc:1050] T 43008abe74604666b330d78ec0f51c7c P a1a844ecd71145e4a362f4c42500fbcb: attempting to promote NON_VOTER 4125b1a70e864e129bd54bdc2720d7fa to VOTER
I20240503 01:59:09.050230 4028 consensus_queue.cc:237] T 43008abe74604666b330d78ec0f51c7c P a1a844ecd71145e4a362f4c42500fbcb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 156, Committed index: 156, Last appended: 1.156, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 157 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { replace: true } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } }
I20240503 01:59:09.053153 2661 raft_consensus.cc:1261] T 43008abe74604666b330d78ec0f51c7c P b40a9a7b30d34a6db007ed8febd8a759 [term 1 FOLLOWER]: Refusing update from remote peer a1a844ecd71145e4a362f4c42500fbcb: Log matching property violated. Preceding OpId in replica: term: 1 index: 156. Preceding OpId from leader: term: 1 index: 157. (index mismatch)
I20240503 01:59:09.053331 3958 raft_consensus.cc:1261] T 43008abe74604666b330d78ec0f51c7c P 4125b1a70e864e129bd54bdc2720d7fa [term 1 LEARNER]: Refusing update from remote peer a1a844ecd71145e4a362f4c42500fbcb: Log matching property violated. Preceding OpId in replica: term: 1 index: 156. Preceding OpId from leader: term: 1 index: 157. (index mismatch)
I20240503 01:59:09.053544 4612 consensus_queue.cc:1035] T 43008abe74604666b330d78ec0f51c7c P a1a844ecd71145e4a362f4c42500fbcb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 157, Last known committed idx: 156, Time since last communication: 0.000s
I20240503 01:59:09.055153 4612 consensus_queue.cc:1035] T 43008abe74604666b330d78ec0f51c7c P a1a844ecd71145e4a362f4c42500fbcb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 157, Last known committed idx: 155, Time since last communication: 0.000s
I20240503 01:59:09.056002 2397 raft_consensus.cc:1261] T 43008abe74604666b330d78ec0f51c7c P b63928177fa54227a5ff9929265ab9dd [term 1 FOLLOWER]: Refusing update from remote peer a1a844ecd71145e4a362f4c42500fbcb: Log matching property violated. Preceding OpId in replica: term: 1 index: 156. Preceding OpId from leader: term: 1 index: 157. (index mismatch)
I20240503 01:59:09.056378 4612 consensus_queue.cc:1035] T 43008abe74604666b330d78ec0f51c7c P a1a844ecd71145e4a362f4c42500fbcb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 157, Last known committed idx: 156, Time since last communication: 0.000s
I20240503 01:59:09.057395 3051 raft_consensus.cc:1261] T 43008abe74604666b330d78ec0f51c7c P 6568f69a78c44bb985162d2b7d0a9376 [term 1 FOLLOWER]: Refusing update from remote peer a1a844ecd71145e4a362f4c42500fbcb: Log matching property violated. Preceding OpId in replica: term: 1 index: 156. Preceding OpId from leader: term: 1 index: 157. (index mismatch)
I20240503 01:59:09.057942 4612 consensus_queue.cc:1035] T 43008abe74604666b330d78ec0f51c7c P a1a844ecd71145e4a362f4c42500fbcb [LEADER]: Connected to new peer: Peer: permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 157, Last known committed idx: 155, Time since last communication: 0.000s
I20240503 01:59:09.059847 3624 raft_consensus.cc:2932] T 43008abe74604666b330d78ec0f51c7c P a1a844ecd71145e4a362f4c42500fbcb [term 1 LEADER]: Committing config change with OpId 1.157: config changed from index 134 to 157, 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 157 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { replace: true } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:09.062492 3051 raft_consensus.cc:2932] T 43008abe74604666b330d78ec0f51c7c P 6568f69a78c44bb985162d2b7d0a9376 [term 1 FOLLOWER]: Committing config change with OpId 1.157: config changed from index 134 to 157, 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 157 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { replace: true } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:09.066053 2253 catalog_manager.cc:5484] T 43008abe74604666b330d78ec0f51c7c P a1a844ecd71145e4a362f4c42500fbcb reported cstate change: config changed from index 134 to 157, 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a1a844ecd71145e4a362f4c42500fbcb" committed_config { opid_index: 157 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:09.066877 3955 raft_consensus.cc:2932] T 43008abe74604666b330d78ec0f51c7c P 4125b1a70e864e129bd54bdc2720d7fa [term 1 FOLLOWER]: Committing config change with OpId 1.157: config changed from index 134 to 157, 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 157 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { replace: true } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:09.069967 2661 raft_consensus.cc:2932] T 43008abe74604666b330d78ec0f51c7c P b40a9a7b30d34a6db007ed8febd8a759 [term 1 FOLLOWER]: Committing config change with OpId 1.157: config changed from index 134 to 157, 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 157 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { replace: true } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:09.071789 2397 raft_consensus.cc:2932] T 43008abe74604666b330d78ec0f51c7c P b63928177fa54227a5ff9929265ab9dd [term 1 FOLLOWER]: Committing config change with OpId 1.157: config changed from index 134 to 157, 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 157 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { replace: true } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:09.079452 2524 consensus_queue.cc:237] T 43008abe74604666b330d78ec0f51c7c P a1a844ecd71145e4a362f4c42500fbcb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 157, Committed index: 157, Last appended: 1.158, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 159 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } }
I20240503 01:59:09.083556 2396 raft_consensus.cc:1261] T 43008abe74604666b330d78ec0f51c7c P b63928177fa54227a5ff9929265ab9dd [term 1 FOLLOWER]: Refusing update from remote peer a1a844ecd71145e4a362f4c42500fbcb: Log matching property violated. Preceding OpId in replica: term: 1 index: 158. Preceding OpId from leader: term: 1 index: 159. (index mismatch)
I20240503 01:59:09.083824 3052 raft_consensus.cc:1261] T 43008abe74604666b330d78ec0f51c7c P 6568f69a78c44bb985162d2b7d0a9376 [term 1 FOLLOWER]: Refusing update from remote peer a1a844ecd71145e4a362f4c42500fbcb: Log matching property violated. Preceding OpId in replica: term: 1 index: 158. Preceding OpId from leader: term: 1 index: 159. (index mismatch)
I20240503 01:59:09.084070 4333 consensus_queue.cc:1035] T 43008abe74604666b330d78ec0f51c7c P a1a844ecd71145e4a362f4c42500fbcb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 159, Last known committed idx: 157, Time since last communication: 0.000s
I20240503 01:59:09.084215 4612 consensus_queue.cc:1035] T 43008abe74604666b330d78ec0f51c7c P a1a844ecd71145e4a362f4c42500fbcb [LEADER]: Connected to new peer: Peer: permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 159, Last known committed idx: 157, Time since last communication: 0.000s
I20240503 01:59:09.084338 3955 raft_consensus.cc:1261] T 43008abe74604666b330d78ec0f51c7c P 4125b1a70e864e129bd54bdc2720d7fa [term 1 FOLLOWER]: Refusing update from remote peer a1a844ecd71145e4a362f4c42500fbcb: Log matching property violated. Preceding OpId in replica: term: 1 index: 157. Preceding OpId from leader: term: 1 index: 159. (index mismatch)
I20240503 01:59:09.085001 4612 consensus_queue.cc:1035] T 43008abe74604666b330d78ec0f51c7c P a1a844ecd71145e4a362f4c42500fbcb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 159, Last known committed idx: 157, Time since last communication: 0.000s
I20240503 01:59:09.087218 2662 raft_consensus.cc:1261] T 43008abe74604666b330d78ec0f51c7c P b40a9a7b30d34a6db007ed8febd8a759 [term 1 FOLLOWER]: Refusing update from remote peer a1a844ecd71145e4a362f4c42500fbcb: Log matching property violated. Preceding OpId in replica: term: 1 index: 158. Preceding OpId from leader: term: 1 index: 159. (index mismatch)
I20240503 01:59:09.087846 4612 raft_consensus.cc:2932] T 43008abe74604666b330d78ec0f51c7c P a1a844ecd71145e4a362f4c42500fbcb [term 1 LEADER]: Committing config change with OpId 1.159: config changed from index 157 to 159, VOTER 327aa26a8e434a609ebcb382aa665fce (127.0.99.133) evicted. New config: { opid_index: 159 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:09.088299 4333 consensus_queue.cc:1035] T 43008abe74604666b330d78ec0f51c7c P a1a844ecd71145e4a362f4c42500fbcb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 159, Last known committed idx: 157, Time since last communication: 0.000s
I20240503 01:59:09.090813 2245 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 43008abe74604666b330d78ec0f51c7c with cas_config_opid_index 157: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:09.091572 2253 catalog_manager.cc:5484] T 43008abe74604666b330d78ec0f51c7c P a1a844ecd71145e4a362f4c42500fbcb reported cstate change: config changed from index 157 to 159, VOTER 327aa26a8e434a609ebcb382aa665fce (127.0.99.133) evicted. New cstate: current_term: 1 leader_uuid: "a1a844ecd71145e4a362f4c42500fbcb" committed_config { opid_index: 159 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:09.091578 3052 raft_consensus.cc:2932] T 43008abe74604666b330d78ec0f51c7c P 6568f69a78c44bb985162d2b7d0a9376 [term 1 FOLLOWER]: Committing config change with OpId 1.159: config changed from index 157 to 159, VOTER 327aa26a8e434a609ebcb382aa665fce (127.0.99.133) evicted. New config: { opid_index: 159 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:09.095466 2904 tablet_service.cc:1514] Processing DeleteTablet for tablet 43008abe74604666b330d78ec0f51c7c with delete_type TABLET_DATA_TOMBSTONED (TS 327aa26a8e434a609ebcb382aa665fce not found in new config with opid_index 159) from {username='slave'} at 127.0.0.1:60782
I20240503 01:59:09.095700 3955 raft_consensus.cc:2932] T 43008abe74604666b330d78ec0f51c7c P 4125b1a70e864e129bd54bdc2720d7fa [term 1 FOLLOWER]: Committing config change with OpId 1.159: config changed from index 157 to 159, VOTER 327aa26a8e434a609ebcb382aa665fce (127.0.99.133) evicted. New config: { opid_index: 159 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:09.097927 2924 raft_consensus.cc:1261] T 43008abe74604666b330d78ec0f51c7c P 327aa26a8e434a609ebcb382aa665fce [term 1 FOLLOWER]: Refusing update from remote peer a1a844ecd71145e4a362f4c42500fbcb: Log matching property violated. Preceding OpId in replica: term: 1 index: 156. Preceding OpId from leader: term: 1 index: 157. (index mismatch)
I20240503 01:59:09.106287 2397 raft_consensus.cc:2932] T 43008abe74604666b330d78ec0f51c7c P b63928177fa54227a5ff9929265ab9dd [term 1 FOLLOWER]: Committing config change with OpId 1.159: config changed from index 157 to 159, VOTER 327aa26a8e434a609ebcb382aa665fce (127.0.99.133) evicted. New config: { opid_index: 159 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:09.112552 4650 tablet_replica.cc:330] T 43008abe74604666b330d78ec0f51c7c P 327aa26a8e434a609ebcb382aa665fce: stopping tablet replica
I20240503 01:59:09.112778 4650 raft_consensus.cc:2229] T 43008abe74604666b330d78ec0f51c7c P 327aa26a8e434a609ebcb382aa665fce [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:09.112924 4650 pending_rounds.cc:62] T 43008abe74604666b330d78ec0f51c7c P 327aa26a8e434a609ebcb382aa665fce: Trying to abort 1 pending ops.
I20240503 01:59:09.113027 4650 pending_rounds.cc:68] T 43008abe74604666b330d78ec0f51c7c P 327aa26a8e434a609ebcb382aa665fce: Aborting op as it isn't in flight: id { term: 1 index: 156 } timestamp: 7023417544884789248 op_type: WRITE_OP write_request { tablet_id: "43008abe74604666b330d78ec0f51c7c" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000r\363?\007\001#\234\362\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""136a42114a2ee818" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023417544865746944 authz_token { token_data: "\010\221\214\321\261\006\"3\n\005slave\022*\n 2de29504ce33484ba12dd9985ab742a9\020\001\030\001 \001(\001" signature: "<redacted>""\024\264\251Q\231\371%s\361I\035\220\355\332\234\352\004T\024\300\332=\004\331\302\020\3459\254\251d\211\204\316n\345\223M\020r\224\362,1G\324\264\250\357[Y\310\211t\356\002e9\323\364\341_\2447" signing_key_seq_num: 0 } } request_id { client_id: "f06eac613c694c3aaf48086938797abc" seq_no: 468 first_incomplete_seq_no: 468 attempt_no: 0 }
I20240503 01:59:09.113495 4650 raft_consensus.cc:2258] T 43008abe74604666b330d78ec0f51c7c P 327aa26a8e434a609ebcb382aa665fce [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:09.114228 4650 ts_tablet_manager.cc:1875] T 43008abe74604666b330d78ec0f51c7c P 327aa26a8e434a609ebcb382aa665fce: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:09.121346 2659 raft_consensus.cc:2932] T 43008abe74604666b330d78ec0f51c7c P b40a9a7b30d34a6db007ed8febd8a759 [term 1 FOLLOWER]: Committing config change with OpId 1.159: config changed from index 157 to 159, VOTER 327aa26a8e434a609ebcb382aa665fce (127.0.99.133) evicted. New config: { opid_index: 159 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:09.125699 4650 ts_tablet_manager.cc:1888] T 43008abe74604666b330d78ec0f51c7c P 327aa26a8e434a609ebcb382aa665fce: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.156
I20240503 01:59:09.125836 4650 log.cc:1196] T 43008abe74604666b330d78ec0f51c7c P 327aa26a8e434a609ebcb382aa665fce: Deleting WAL directory at /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/43008abe74604666b330d78ec0f51c7c
I20240503 01:59:09.126750 2243 catalog_manager.cc:4830] TS 327aa26a8e434a609ebcb382aa665fce (127.0.99.133:38853): tablet 43008abe74604666b330d78ec0f51c7c (table rebalance_test_table_0 [id=2de29504ce33484ba12dd9985ab742a9]) successfully deleted
I20240503 01:59:09.190780 3955 raft_consensus.cc:1203] T c2911cf48f384bf09074f7bea2a4c173 P 4125b1a70e864e129bd54bdc2720d7fa [term 1 LEARNER]: Deduplicated request from leader. Original: 1.122->[1.123-1.138] Dedup: 1.132->[1.133-1.138]
I20240503 01:59:09.242475 4380 raft_consensus.cc:1050] T c2911cf48f384bf09074f7bea2a4c173 P 6568f69a78c44bb985162d2b7d0a9376: attempting to promote NON_VOTER 4125b1a70e864e129bd54bdc2720d7fa to VOTER
I20240503 01:59:09.243032 4380 consensus_queue.cc:237] T c2911cf48f384bf09074f7bea2a4c173 P 6568f69a78c44bb985162d2b7d0a9376 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 138, Committed index: 138, Last appended: 1.139, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 140 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } }
I20240503 01:59:09.245148 3956 raft_consensus.cc:1261] T c2911cf48f384bf09074f7bea2a4c173 P 4125b1a70e864e129bd54bdc2720d7fa [term 1 LEARNER]: Refusing update from remote peer 6568f69a78c44bb985162d2b7d0a9376: Log matching property violated. Preceding OpId in replica: term: 1 index: 139. Preceding OpId from leader: term: 1 index: 140. (index mismatch)
I20240503 01:59:09.245366 2921 raft_consensus.cc:1261] T c2911cf48f384bf09074f7bea2a4c173 P 327aa26a8e434a609ebcb382aa665fce [term 1 FOLLOWER]: Refusing update from remote peer 6568f69a78c44bb985162d2b7d0a9376: Log matching property violated. Preceding OpId in replica: term: 1 index: 139. Preceding OpId from leader: term: 1 index: 140. (index mismatch)
I20240503 01:59:09.245756 4112 consensus_queue.cc:1035] T c2911cf48f384bf09074f7bea2a4c173 P 6568f69a78c44bb985162d2b7d0a9376 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 140, Last known committed idx: 138, Time since last communication: 0.000s
I20240503 01:59:09.245918 2524 raft_consensus.cc:1261] T c2911cf48f384bf09074f7bea2a4c173 P a1a844ecd71145e4a362f4c42500fbcb [term 1 FOLLOWER]: Refusing update from remote peer 6568f69a78c44bb985162d2b7d0a9376: Log matching property violated. Preceding OpId in replica: term: 1 index: 139. Preceding OpId from leader: term: 1 index: 140. (index mismatch)
I20240503 01:59:09.246651 4112 consensus_queue.cc:1035] T c2911cf48f384bf09074f7bea2a4c173 P 6568f69a78c44bb985162d2b7d0a9376 [LEADER]: Connected to new peer: Peer: permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 140, Last known committed idx: 138, Time since last communication: 0.000s
I20240503 01:59:09.247126 4582 ts_tablet_manager.cc:902] T 0b669a47f1d14ce98ccab24479d39304 P 4125b1a70e864e129bd54bdc2720d7fa: Initiating tablet copy from peer b40a9a7b30d34a6db007ed8febd8a759 (127.0.99.131:34079)
I20240503 01:59:09.247602 4582 tablet_copy_client.cc:308] T 0b669a47f1d14ce98ccab24479d39304 P 4125b1a70e864e129bd54bdc2720d7fa: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:34079
I20240503 01:59:09.248189 2682 tablet_copy_service.cc:140] P b40a9a7b30d34a6db007ed8febd8a759: Received BeginTabletCopySession request for tablet 0b669a47f1d14ce98ccab24479d39304 from peer 4125b1a70e864e129bd54bdc2720d7fa ({username='slave'} at 127.0.99.135:51755)
I20240503 01:59:09.248297 2682 tablet_copy_service.cc:161] P b40a9a7b30d34a6db007ed8febd8a759: Beginning new tablet copy session on tablet 0b669a47f1d14ce98ccab24479d39304 from peer 4125b1a70e864e129bd54bdc2720d7fa at {username='slave'} at 127.0.99.135:51755: session id = 4125b1a70e864e129bd54bdc2720d7fa-0b669a47f1d14ce98ccab24479d39304
I20240503 01:59:09.248183 4380 consensus_queue.cc:1035] T c2911cf48f384bf09074f7bea2a4c173 P 6568f69a78c44bb985162d2b7d0a9376 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 140, Last known committed idx: 138, Time since last communication: 0.000s
I20240503 01:59:09.249485 2682 tablet_copy_source_session.cc:205] T 0b669a47f1d14ce98ccab24479d39304 P b40a9a7b30d34a6db007ed8febd8a759: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:09.251435 4582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b669a47f1d14ce98ccab24479d39304. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:09.253588 4112 raft_consensus.cc:2932] T c2911cf48f384bf09074f7bea2a4c173 P 6568f69a78c44bb985162d2b7d0a9376 [term 1 LEADER]: Committing config change with OpId 1.140: config changed from index 123 to 140, 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 140 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:09.254149 4582 tablet_copy_client.cc:783] T 0b669a47f1d14ce98ccab24479d39304 P 4125b1a70e864e129bd54bdc2720d7fa: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:09.254386 4582 tablet_copy_client.cc:647] T 0b669a47f1d14ce98ccab24479d39304 P 4125b1a70e864e129bd54bdc2720d7fa: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:09.255895 3956 raft_consensus.cc:2932] T c2911cf48f384bf09074f7bea2a4c173 P 4125b1a70e864e129bd54bdc2720d7fa [term 1 FOLLOWER]: Committing config change with OpId 1.140: config changed from index 123 to 140, 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 140 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:09.258854 2253 catalog_manager.cc:5484] T c2911cf48f384bf09074f7bea2a4c173 P 4125b1a70e864e129bd54bdc2720d7fa reported cstate change: config changed from index 123 to 140, 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6568f69a78c44bb985162d2b7d0a9376" committed_config { opid_index: 140 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:09.263913 2921 raft_consensus.cc:2932] T c2911cf48f384bf09074f7bea2a4c173 P 327aa26a8e434a609ebcb382aa665fce [term 1 FOLLOWER]: Committing config change with OpId 1.140: config changed from index 123 to 140, 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 140 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:09.265264 2520 raft_consensus.cc:2932] T c2911cf48f384bf09074f7bea2a4c173 P a1a844ecd71145e4a362f4c42500fbcb [term 1 FOLLOWER]: Committing config change with OpId 1.140: config changed from index 123 to 140, 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 140 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:09.296469 2661 raft_consensus.cc:1261] T c2911cf48f384bf09074f7bea2a4c173 P b40a9a7b30d34a6db007ed8febd8a759 [term 1 FOLLOWER]: Refusing update from remote peer 6568f69a78c44bb985162d2b7d0a9376: Log matching property violated. Preceding OpId in replica: term: 1 index: 139. Preceding OpId from leader: term: 1 index: 140. (index mismatch)
I20240503 01:59:09.317154 4380 consensus_queue.cc:1035] T c2911cf48f384bf09074f7bea2a4c173 P 6568f69a78c44bb985162d2b7d0a9376 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 140, Last known committed idx: 138, Time since last communication: 0.000s
I20240503 01:59:09.322706 2661 raft_consensus.cc:2932] T c2911cf48f384bf09074f7bea2a4c173 P b40a9a7b30d34a6db007ed8febd8a759 [term 1 FOLLOWER]: Committing config change with OpId 1.140: config changed from index 123 to 140, 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 140 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:09.333259 4582 tablet_copy_client.cc:523] T 0b669a47f1d14ce98ccab24479d39304 P 4125b1a70e864e129bd54bdc2720d7fa: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:09.334933 4582 tablet_bootstrap.cc:492] T 0b669a47f1d14ce98ccab24479d39304 P 4125b1a70e864e129bd54bdc2720d7fa: Bootstrap starting.
I20240503 01:59:09.372211 2397 raft_consensus.cc:1261] T c2911cf48f384bf09074f7bea2a4c173 P b63928177fa54227a5ff9929265ab9dd [term 1 FOLLOWER]: Refusing update from remote peer 6568f69a78c44bb985162d2b7d0a9376: Log matching property violated. Preceding OpId in replica: term: 1 index: 138. Preceding OpId from leader: term: 1 index: 140. (index mismatch)
I20240503 01:59:09.401732 3726 consensus_queue.cc:1035] T c2911cf48f384bf09074f7bea2a4c173 P 6568f69a78c44bb985162d2b7d0a9376 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 140, Last known committed idx: 138, Time since last communication: 0.000s
I20240503 01:59:09.432072 2393 raft_consensus.cc:2932] T c2911cf48f384bf09074f7bea2a4c173 P b63928177fa54227a5ff9929265ab9dd [term 1 FOLLOWER]: Committing config change with OpId 1.140: config changed from index 123 to 140, 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 140 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:09.477324 3054 consensus_queue.cc:237] T c2911cf48f384bf09074f7bea2a4c173 P 6568f69a78c44bb985162d2b7d0a9376 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 141, Committed index: 141, Last appended: 1.142, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 143 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } }
I20240503 01:59:09.479197 2661 raft_consensus.cc:1261] T c2911cf48f384bf09074f7bea2a4c173 P b40a9a7b30d34a6db007ed8febd8a759 [term 1 FOLLOWER]: Refusing update from remote peer 6568f69a78c44bb985162d2b7d0a9376: Log matching property violated. Preceding OpId in replica: term: 1 index: 142. Preceding OpId from leader: term: 1 index: 143. (index mismatch)
I20240503 01:59:09.481204 4651 consensus_queue.cc:1035] T c2911cf48f384bf09074f7bea2a4c173 P 6568f69a78c44bb985162d2b7d0a9376 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 143, Last known committed idx: 141, Time since last communication: 0.000s
I20240503 01:59:09.488675 3955 raft_consensus.cc:1203] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 4125b1a70e864e129bd54bdc2720d7fa [term 1 LEARNER]: Deduplicated request from leader. Original: 1.63->[1.64-1.92] Dedup: 1.71->[1.72-1.92]
I20240503 01:59:09.496804 3958 raft_consensus.cc:1261] T c2911cf48f384bf09074f7bea2a4c173 P 4125b1a70e864e129bd54bdc2720d7fa [term 1 FOLLOWER]: Refusing update from remote peer 6568f69a78c44bb985162d2b7d0a9376: Log matching property violated. Preceding OpId in replica: term: 1 index: 142. Preceding OpId from leader: term: 1 index: 143. (index mismatch)
I20240503 01:59:09.501852 3726 consensus_queue.cc:1035] T c2911cf48f384bf09074f7bea2a4c173 P 6568f69a78c44bb985162d2b7d0a9376 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 143, Last known committed idx: 141, Time since last communication: 0.000s
I20240503 01:59:09.505126 4651 raft_consensus.cc:2932] T c2911cf48f384bf09074f7bea2a4c173 P 6568f69a78c44bb985162d2b7d0a9376 [term 1 LEADER]: Committing config change with OpId 1.143: config changed from index 140 to 143, VOTER a1a844ecd71145e4a362f4c42500fbcb (127.0.99.130) evicted. New config: { opid_index: 143 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:09.508423 2661 raft_consensus.cc:2932] T c2911cf48f384bf09074f7bea2a4c173 P b40a9a7b30d34a6db007ed8febd8a759 [term 1 FOLLOWER]: Committing config change with OpId 1.143: config changed from index 140 to 143, VOTER a1a844ecd71145e4a362f4c42500fbcb (127.0.99.130) evicted. New config: { opid_index: 143 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:09.509104 2243 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c2911cf48f384bf09074f7bea2a4c173 with cas_config_opid_index 140: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:09.509742 3958 raft_consensus.cc:2932] T c2911cf48f384bf09074f7bea2a4c173 P 4125b1a70e864e129bd54bdc2720d7fa [term 1 FOLLOWER]: Committing config change with OpId 1.143: config changed from index 140 to 143, VOTER a1a844ecd71145e4a362f4c42500fbcb (127.0.99.130) evicted. New config: { opid_index: 143 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:09.510879 2253 catalog_manager.cc:5484] T c2911cf48f384bf09074f7bea2a4c173 P b40a9a7b30d34a6db007ed8febd8a759 reported cstate change: config changed from index 140 to 143, VOTER a1a844ecd71145e4a362f4c42500fbcb (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "6568f69a78c44bb985162d2b7d0a9376" committed_config { opid_index: 143 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:09.522922 2510 tablet_service.cc:1514] Processing DeleteTablet for tablet c2911cf48f384bf09074f7bea2a4c173 with delete_type TABLET_DATA_TOMBSTONED (TS a1a844ecd71145e4a362f4c42500fbcb not found in new config with opid_index 143) from {username='slave'} at 127.0.0.1:59126
I20240503 01:59:09.541071 2923 raft_consensus.cc:1261] T c2911cf48f384bf09074f7bea2a4c173 P 327aa26a8e434a609ebcb382aa665fce [term 1 FOLLOWER]: Refusing update from remote peer 6568f69a78c44bb985162d2b7d0a9376: Log matching property violated. Preceding OpId in replica: term: 1 index: 142. Preceding OpId from leader: term: 1 index: 143. (index mismatch)
I20240503 01:59:09.542204 4380 consensus_queue.cc:1035] T c2911cf48f384bf09074f7bea2a4c173 P 6568f69a78c44bb985162d2b7d0a9376 [LEADER]: Connected to new peer: Peer: permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 143, Last known committed idx: 141, Time since last communication: 0.000s
I20240503 01:59:09.556777 2397 raft_consensus.cc:1261] T c2911cf48f384bf09074f7bea2a4c173 P b63928177fa54227a5ff9929265ab9dd [term 1 FOLLOWER]: Refusing update from remote peer 6568f69a78c44bb985162d2b7d0a9376: Log matching property violated. Preceding OpId in replica: term: 1 index: 142. Preceding OpId from leader: term: 1 index: 143. (index mismatch)
I20240503 01:59:09.568714 4380 consensus_queue.cc:1035] T c2911cf48f384bf09074f7bea2a4c173 P 6568f69a78c44bb985162d2b7d0a9376 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 143, Last known committed idx: 141, Time since last communication: 0.000s
I20240503 01:59:09.587456 2921 raft_consensus.cc:2932] T c2911cf48f384bf09074f7bea2a4c173 P 327aa26a8e434a609ebcb382aa665fce [term 1 FOLLOWER]: Committing config change with OpId 1.143: config changed from index 140 to 143, VOTER a1a844ecd71145e4a362f4c42500fbcb (127.0.99.130) evicted. New config: { opid_index: 143 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:09.596334 4651 raft_consensus.cc:1050] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 6568f69a78c44bb985162d2b7d0a9376: attempting to promote NON_VOTER 4125b1a70e864e129bd54bdc2720d7fa to VOTER
I20240503 01:59:09.597082 4651 consensus_queue.cc:237] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 6568f69a78c44bb985162d2b7d0a9376 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 92, Committed index: 92, Last appended: 1.93, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { replace: true } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } }
I20240503 01:59:09.601140 2661 raft_consensus.cc:1261] T 8682bff56e604cc3b7e9b07c8ba3fca5 P b40a9a7b30d34a6db007ed8febd8a759 [term 1 FOLLOWER]: Refusing update from remote peer 6568f69a78c44bb985162d2b7d0a9376: Log matching property violated. Preceding OpId in replica: term: 1 index: 93. Preceding OpId from leader: term: 1 index: 94. (index mismatch)
I20240503 01:59:09.601727 4488 consensus_queue.cc:1035] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 6568f69a78c44bb985162d2b7d0a9376 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 94, Last known committed idx: 92, Time since last communication: 0.000s
I20240503 01:59:09.604311 3956 raft_consensus.cc:1261] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 4125b1a70e864e129bd54bdc2720d7fa [term 1 LEARNER]: Refusing update from remote peer 6568f69a78c44bb985162d2b7d0a9376: Log matching property violated. Preceding OpId in replica: term: 1 index: 93. Preceding OpId from leader: term: 1 index: 94. (index mismatch)
I20240503 01:59:09.604808 4112 consensus_queue.cc:1035] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 6568f69a78c44bb985162d2b7d0a9376 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 94, Last known committed idx: 92, Time since last communication: 0.000s
I20240503 01:59:09.604936 2787 raft_consensus.cc:1261] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Refusing update from remote peer 6568f69a78c44bb985162d2b7d0a9376: Log matching property violated. Preceding OpId in replica: term: 1 index: 93. Preceding OpId from leader: term: 1 index: 94. (index mismatch)
I20240503 01:59:09.605324 4380 consensus_queue.cc:1035] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 6568f69a78c44bb985162d2b7d0a9376 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 94, Last known committed idx: 92, Time since last communication: 0.000s
I20240503 01:59:09.608204 4488 raft_consensus.cc:2932] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 6568f69a78c44bb985162d2b7d0a9376 [term 1 LEADER]: Committing config change with OpId 1.94: config changed from index 64 to 94, 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { replace: true } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:09.621204 2659 raft_consensus.cc:2932] T 8682bff56e604cc3b7e9b07c8ba3fca5 P b40a9a7b30d34a6db007ed8febd8a759 [term 1 FOLLOWER]: Committing config change with OpId 1.94: config changed from index 64 to 94, 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { replace: true } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:09.621908 2787 raft_consensus.cc:2932] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Committing config change with OpId 1.94: config changed from index 64 to 94, 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { replace: true } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:09.622323 3956 raft_consensus.cc:2932] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 4125b1a70e864e129bd54bdc2720d7fa [term 1 FOLLOWER]: Committing config change with OpId 1.94: config changed from index 64 to 94, 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { replace: true } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:09.625427 2250 catalog_manager.cc:5484] T 8682bff56e604cc3b7e9b07c8ba3fca5 P b40a9a7b30d34a6db007ed8febd8a759 reported cstate change: config changed from index 64 to 94, 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6568f69a78c44bb985162d2b7d0a9376" committed_config { opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { replace: true } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:09.661675 4685 tablet_replica.cc:330] T c2911cf48f384bf09074f7bea2a4c173 P a1a844ecd71145e4a362f4c42500fbcb: stopping tablet replica
I20240503 01:59:09.661818 4685 raft_consensus.cc:2229] T c2911cf48f384bf09074f7bea2a4c173 P a1a844ecd71145e4a362f4c42500fbcb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:09.661926 4685 pending_rounds.cc:62] T c2911cf48f384bf09074f7bea2a4c173 P a1a844ecd71145e4a362f4c42500fbcb: Trying to abort 1 pending ops.
I20240503 01:59:09.661948 4685 pending_rounds.cc:68] T c2911cf48f384bf09074f7bea2a4c173 P a1a844ecd71145e4a362f4c42500fbcb: Aborting op as it isn't in flight: id { term: 1 index: 142 } timestamp: 7023417546526687232 op_type: WRITE_OP write_request { tablet_id: "c2911cf48f384bf09074f7bea2a4c173" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\351V,4]x\021\347\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""3c0208a1d3098343" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023417546334654464 authz_token { token_data: "\010\221\214\321\261\006\"3\n\005slave\022*\n 18b5c04f80684606bc736d3ca6780b16\020\001\030\001 \001(\001" signature: "<redacted>""u\355x\260\217(\214\267\257&\351A\001\332R\342\027\006\006\365s\177\344\373\033\301\033\355\331Z\264\2564\023\005,!\224\225\027#,\314\001_\2507\2025W3N\317\225@\351\020\341\251Ig\251\226\234" signing_key_seq_num: 0 } } request_id { client_id: "aca3b665abf249fbb6dcc5c0e82250db" seq_no: 385 first_incomplete_seq_no: 385 attempt_no: 0 }
I20240503 01:59:09.662515 4685 raft_consensus.cc:2258] T c2911cf48f384bf09074f7bea2a4c173 P a1a844ecd71145e4a362f4c42500fbcb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:09.692703 2924 raft_consensus.cc:1261] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 327aa26a8e434a609ebcb382aa665fce [term 1 FOLLOWER]: Refusing update from remote peer 6568f69a78c44bb985162d2b7d0a9376: Log matching property violated. Preceding OpId in replica: term: 1 index: 92. Preceding OpId from leader: term: 1 index: 94. (index mismatch)
I20240503 01:59:09.693408 2393 raft_consensus.cc:2932] T c2911cf48f384bf09074f7bea2a4c173 P b63928177fa54227a5ff9929265ab9dd [term 1 FOLLOWER]: Committing config change with OpId 1.143: config changed from index 140 to 143, VOTER a1a844ecd71145e4a362f4c42500fbcb (127.0.99.130) evicted. New config: { opid_index: 143 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:09.698258 2397 raft_consensus.cc:1261] T 8682bff56e604cc3b7e9b07c8ba3fca5 P b63928177fa54227a5ff9929265ab9dd [term 1 FOLLOWER]: Refusing update from remote peer 6568f69a78c44bb985162d2b7d0a9376: Log matching property violated. Preceding OpId in replica: term: 1 index: 93. Preceding OpId from leader: term: 1 index: 94. (index mismatch)
I20240503 01:59:09.706802 4685 ts_tablet_manager.cc:1875] T c2911cf48f384bf09074f7bea2a4c173 P a1a844ecd71145e4a362f4c42500fbcb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:09.709066 4685 ts_tablet_manager.cc:1888] T c2911cf48f384bf09074f7bea2a4c173 P a1a844ecd71145e4a362f4c42500fbcb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.142
I20240503 01:59:09.709152 4685 log.cc:1196] T c2911cf48f384bf09074f7bea2a4c173 P a1a844ecd71145e4a362f4c42500fbcb: Deleting WAL directory at /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/c2911cf48f384bf09074f7bea2a4c173
I20240503 01:59:09.709805 2245 catalog_manager.cc:4830] TS a1a844ecd71145e4a362f4c42500fbcb (127.0.99.130:46033): tablet c2911cf48f384bf09074f7bea2a4c173 (table rebalance_test_table_2 [id=18b5c04f80684606bc736d3ca6780b16]) successfully deleted
I20240503 01:59:09.713523 3726 consensus_queue.cc:1035] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 6568f69a78c44bb985162d2b7d0a9376 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 94, Last known committed idx: 92, Time since last communication: 0.000s
I20240503 01:59:09.718487 2399 raft_consensus.cc:2932] T 8682bff56e604cc3b7e9b07c8ba3fca5 P b63928177fa54227a5ff9929265ab9dd [term 1 FOLLOWER]: Committing config change with OpId 1.94: config changed from index 64 to 94, 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { replace: true } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:09.716807 3726 consensus_queue.cc:1035] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 6568f69a78c44bb985162d2b7d0a9376 [LEADER]: Connected to new peer: Peer: permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 94, Last known committed idx: 92, Time since last communication: 0.003s
I20240503 01:59:09.751456 4582 tablet_bootstrap.cc:492] T 0b669a47f1d14ce98ccab24479d39304 P 4125b1a70e864e129bd54bdc2720d7fa: Bootstrap replayed 1/1 log segments. Stats: ops{read=127 overwritten=0 applied=126 ignored=0} inserts{seen=811 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 01:59:09.751819 4582 tablet_bootstrap.cc:492] T 0b669a47f1d14ce98ccab24479d39304 P 4125b1a70e864e129bd54bdc2720d7fa: Bootstrap complete.
I20240503 01:59:09.752050 4582 ts_tablet_manager.cc:1367] T 0b669a47f1d14ce98ccab24479d39304 P 4125b1a70e864e129bd54bdc2720d7fa: Time spent bootstrapping tablet: real 0.417s user 0.105s sys 0.011s
I20240503 01:59:09.752251 2920 raft_consensus.cc:1203] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 327aa26a8e434a609ebcb382aa665fce [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.92->[1.93-1.94] Dedup: 1.93->[1.94-1.94]
I20240503 01:59:09.752640 2920 raft_consensus.cc:2932] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 327aa26a8e434a609ebcb382aa665fce [term 1 FOLLOWER]: Committing config change with OpId 1.94: config changed from index 64 to 94, 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { replace: true } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:09.752787 4582 raft_consensus.cc:348] T 0b669a47f1d14ce98ccab24479d39304 P 4125b1a70e864e129bd54bdc2720d7fa [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 119 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { replace: true } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: true } }
I20240503 01:59:09.753075 4582 raft_consensus.cc:729] T 0b669a47f1d14ce98ccab24479d39304 P 4125b1a70e864e129bd54bdc2720d7fa [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4125b1a70e864e129bd54bdc2720d7fa, State: Initialized, Role: LEARNER
I20240503 01:59:09.753217 4582 consensus_queue.cc:260] T 0b669a47f1d14ce98ccab24479d39304 P 4125b1a70e864e129bd54bdc2720d7fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 126, Last appended: 1.127, Last appended by leader: 127, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 119 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { replace: true } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: true } }
I20240503 01:59:09.753599 4582 ts_tablet_manager.cc:1398] T 0b669a47f1d14ce98ccab24479d39304 P 4125b1a70e864e129bd54bdc2720d7fa: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 01:59:09.755825 2682 tablet_copy_service.cc:342] P b40a9a7b30d34a6db007ed8febd8a759: Request end of tablet copy session 4125b1a70e864e129bd54bdc2720d7fa-0b669a47f1d14ce98ccab24479d39304 received from {username='slave'} at 127.0.99.135:51755
I20240503 01:59:09.755903 2682 tablet_copy_service.cc:433] P b40a9a7b30d34a6db007ed8febd8a759: ending tablet copy session 4125b1a70e864e129bd54bdc2720d7fa-0b669a47f1d14ce98ccab24479d39304 on tablet 0b669a47f1d14ce98ccab24479d39304 with peer 4125b1a70e864e129bd54bdc2720d7fa
I20240503 01:59:09.790675 3054 consensus_queue.cc:237] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 6568f69a78c44bb985162d2b7d0a9376 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 95, Committed index: 95, Last appended: 1.95, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 96 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } }
I20240503 01:59:09.795928 2659 raft_consensus.cc:1261] T 8682bff56e604cc3b7e9b07c8ba3fca5 P b40a9a7b30d34a6db007ed8febd8a759 [term 1 FOLLOWER]: Refusing update from remote peer 6568f69a78c44bb985162d2b7d0a9376: Log matching property violated. Preceding OpId in replica: term: 1 index: 95. Preceding OpId from leader: term: 1 index: 96. (index mismatch)
I20240503 01:59:09.796308 4651 consensus_queue.cc:1035] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 6568f69a78c44bb985162d2b7d0a9376 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 96, Last known committed idx: 95, Time since last communication: 0.000s
I20240503 01:59:09.800794 2789 raft_consensus.cc:1261] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Refusing update from remote peer 6568f69a78c44bb985162d2b7d0a9376: Log matching property violated. Preceding OpId in replica: term: 1 index: 95. Preceding OpId from leader: term: 1 index: 96. (index mismatch)
I20240503 01:59:09.800943 3955 raft_consensus.cc:1261] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 4125b1a70e864e129bd54bdc2720d7fa [term 1 FOLLOWER]: Refusing update from remote peer 6568f69a78c44bb985162d2b7d0a9376: Log matching property violated. Preceding OpId in replica: term: 1 index: 95. Preceding OpId from leader: term: 1 index: 96. (index mismatch)
I20240503 01:59:09.803714 4486 consensus_queue.cc:1035] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 6568f69a78c44bb985162d2b7d0a9376 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 96, Last known committed idx: 95, Time since last communication: 0.000s
I20240503 01:59:09.803983 4380 consensus_queue.cc:1035] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 6568f69a78c44bb985162d2b7d0a9376 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 96, Last known committed idx: 95, Time since last communication: 0.000s
I20240503 01:59:09.805100 2920 raft_consensus.cc:1261] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 327aa26a8e434a609ebcb382aa665fce [term 1 FOLLOWER]: Refusing update from remote peer 6568f69a78c44bb985162d2b7d0a9376: Log matching property violated. Preceding OpId in replica: term: 1 index: 95. Preceding OpId from leader: term: 1 index: 96. (index mismatch)
I20240503 01:59:09.811636 4486 raft_consensus.cc:2932] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 6568f69a78c44bb985162d2b7d0a9376 [term 1 LEADER]: Committing config change with OpId 1.96: config changed from index 94 to 96, VOTER b63928177fa54227a5ff9929265ab9dd (127.0.99.129) evicted. New config: { opid_index: 96 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:09.813342 4380 consensus_queue.cc:1035] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 6568f69a78c44bb985162d2b7d0a9376 [LEADER]: Connected to new peer: Peer: permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 96, Last known committed idx: 95, Time since last communication: 0.000s
I20240503 01:59:09.817981 2243 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8682bff56e604cc3b7e9b07c8ba3fca5 with cas_config_opid_index 94: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:09.818513 3955 raft_consensus.cc:2932] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 4125b1a70e864e129bd54bdc2720d7fa [term 1 FOLLOWER]: Committing config change with OpId 1.96: config changed from index 94 to 96, VOTER b63928177fa54227a5ff9929265ab9dd (127.0.99.129) evicted. New config: { opid_index: 96 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:09.823462 2923 raft_consensus.cc:2932] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 327aa26a8e434a609ebcb382aa665fce [term 1 FOLLOWER]: Committing config change with OpId 1.96: config changed from index 94 to 96, VOTER b63928177fa54227a5ff9929265ab9dd (127.0.99.129) evicted. New config: { opid_index: 96 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:09.825134 3956 raft_consensus.cc:1203] T 0b669a47f1d14ce98ccab24479d39304 P 4125b1a70e864e129bd54bdc2720d7fa [term 1 LEARNER]: Deduplicated request from leader. Original: 1.118->[1.119-1.135] Dedup: 1.127->[1.128-1.135]
I20240503 01:59:09.826277 2661 raft_consensus.cc:2932] T 8682bff56e604cc3b7e9b07c8ba3fca5 P b40a9a7b30d34a6db007ed8febd8a759 [term 1 FOLLOWER]: Committing config change with OpId 1.96: config changed from index 94 to 96, VOTER b63928177fa54227a5ff9929265ab9dd (127.0.99.129) evicted. New config: { opid_index: 96 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:09.829469 2251 catalog_manager.cc:5484] T 8682bff56e604cc3b7e9b07c8ba3fca5 P b40a9a7b30d34a6db007ed8febd8a759 reported cstate change: config changed from index 94 to 96, VOTER b63928177fa54227a5ff9929265ab9dd (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "6568f69a78c44bb985162d2b7d0a9376" committed_config { opid_index: 96 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:09.835033 2378 tablet_service.cc:1514] Processing DeleteTablet for tablet 8682bff56e604cc3b7e9b07c8ba3fca5 with delete_type TABLET_DATA_TOMBSTONED (TS b63928177fa54227a5ff9929265ab9dd not found in new config with opid_index 96) from {username='slave'} at 127.0.0.1:44104
I20240503 01:59:09.857733 2793 raft_consensus.cc:2932] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Committing config change with OpId 1.96: config changed from index 94 to 96, VOTER b63928177fa54227a5ff9929265ab9dd (127.0.99.129) evicted. New config: { opid_index: 96 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:09.920990 4699 tablet_replica.cc:330] T 8682bff56e604cc3b7e9b07c8ba3fca5 P b63928177fa54227a5ff9929265ab9dd: stopping tablet replica
I20240503 01:59:09.929934 4699 raft_consensus.cc:2229] T 8682bff56e604cc3b7e9b07c8ba3fca5 P b63928177fa54227a5ff9929265ab9dd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:09.930109 4699 raft_consensus.cc:2258] T 8682bff56e604cc3b7e9b07c8ba3fca5 P b63928177fa54227a5ff9929265ab9dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:09.938206 4699 ts_tablet_manager.cc:1875] T 8682bff56e604cc3b7e9b07c8ba3fca5 P b63928177fa54227a5ff9929265ab9dd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:09.960805 4667 raft_consensus.cc:1050] T 0b669a47f1d14ce98ccab24479d39304 P b40a9a7b30d34a6db007ed8febd8a759: attempting to promote NON_VOTER 4125b1a70e864e129bd54bdc2720d7fa to VOTER
I20240503 01:59:09.961304 4667 consensus_queue.cc:237] T 0b669a47f1d14ce98ccab24479d39304 P b40a9a7b30d34a6db007ed8febd8a759 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 139, Committed index: 139, Last appended: 1.139, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 140 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { replace: true } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } }
I20240503 01:59:09.963179 4699 ts_tablet_manager.cc:1888] T 8682bff56e604cc3b7e9b07c8ba3fca5 P b63928177fa54227a5ff9929265ab9dd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.95
I20240503 01:59:09.963320 4699 log.cc:1196] T 8682bff56e604cc3b7e9b07c8ba3fca5 P b63928177fa54227a5ff9929265ab9dd: Deleting WAL directory at /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/8682bff56e604cc3b7e9b07c8ba3fca5
I20240503 01:59:09.965965 2787 raft_consensus.cc:1261] T 0b669a47f1d14ce98ccab24479d39304 P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Refusing update from remote peer b40a9a7b30d34a6db007ed8febd8a759: Log matching property violated. Preceding OpId in replica: term: 1 index: 139. Preceding OpId from leader: term: 1 index: 140. (index mismatch)
I20240503 01:59:09.966156 3954 raft_consensus.cc:1261] T 0b669a47f1d14ce98ccab24479d39304 P 4125b1a70e864e129bd54bdc2720d7fa [term 1 LEARNER]: Refusing update from remote peer b40a9a7b30d34a6db007ed8febd8a759: Log matching property violated. Preceding OpId in replica: term: 1 index: 139. Preceding OpId from leader: term: 1 index: 140. (index mismatch)
I20240503 01:59:09.966423 4562 consensus_queue.cc:1035] T 0b669a47f1d14ce98ccab24479d39304 P b40a9a7b30d34a6db007ed8febd8a759 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 140, Last known committed idx: 139, Time since last communication: 0.000s
I20240503 01:59:09.966725 4562 consensus_queue.cc:1035] T 0b669a47f1d14ce98ccab24479d39304 P b40a9a7b30d34a6db007ed8febd8a759 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 140, Last known committed idx: 139, Time since last communication: 0.000s
I20240503 01:59:09.967540 3051 raft_consensus.cc:1261] T 0b669a47f1d14ce98ccab24479d39304 P 6568f69a78c44bb985162d2b7d0a9376 [term 1 FOLLOWER]: Refusing update from remote peer b40a9a7b30d34a6db007ed8febd8a759: Log matching property violated. Preceding OpId in replica: term: 1 index: 139. Preceding OpId from leader: term: 1 index: 140. (index mismatch)
I20240503 01:59:09.967641 2520 raft_consensus.cc:1261] T 0b669a47f1d14ce98ccab24479d39304 P a1a844ecd71145e4a362f4c42500fbcb [term 1 FOLLOWER]: Refusing update from remote peer b40a9a7b30d34a6db007ed8febd8a759: Log matching property violated. Preceding OpId in replica: term: 1 index: 139. Preceding OpId from leader: term: 1 index: 140. (index mismatch)
I20240503 01:59:09.967907 4327 consensus_queue.cc:1035] T 0b669a47f1d14ce98ccab24479d39304 P b40a9a7b30d34a6db007ed8febd8a759 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 140, Last known committed idx: 139, Time since last communication: 0.000s
I20240503 01:59:09.969664 4562 consensus_queue.cc:1035] T 0b669a47f1d14ce98ccab24479d39304 P b40a9a7b30d34a6db007ed8febd8a759 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 140, Last known committed idx: 139, Time since last communication: 0.000s
I20240503 01:59:09.972929 4562 raft_consensus.cc:2932] T 0b669a47f1d14ce98ccab24479d39304 P b40a9a7b30d34a6db007ed8febd8a759 [term 1 LEADER]: Committing config change with OpId 1.140: config changed from index 119 to 140, 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 140 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { replace: true } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:09.978343 2250 catalog_manager.cc:5484] T 0b669a47f1d14ce98ccab24479d39304 P b40a9a7b30d34a6db007ed8febd8a759 reported cstate change: config changed from index 119 to 140, 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b40a9a7b30d34a6db007ed8febd8a759" committed_config { opid_index: 140 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:09.978670 3054 raft_consensus.cc:2932] T 0b669a47f1d14ce98ccab24479d39304 P 6568f69a78c44bb985162d2b7d0a9376 [term 1 FOLLOWER]: Committing config change with OpId 1.140: config changed from index 119 to 140, 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 140 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { replace: true } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:09.978760 2793 raft_consensus.cc:2932] T 0b669a47f1d14ce98ccab24479d39304 P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Committing config change with OpId 1.140: config changed from index 119 to 140, 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 140 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { replace: true } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:09.979399 2531 raft_consensus.cc:2932] T 0b669a47f1d14ce98ccab24479d39304 P a1a844ecd71145e4a362f4c42500fbcb [term 1 FOLLOWER]: Committing config change with OpId 1.140: config changed from index 119 to 140, 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 140 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { replace: true } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:09.985951 2245 catalog_manager.cc:4830] TS b63928177fa54227a5ff9929265ab9dd (127.0.99.129:36425): tablet 8682bff56e604cc3b7e9b07c8ba3fca5 (table rebalance_test_table_3 [id=f7b0d84dbe0541aaa2606d6e366719c8]) successfully deleted
I20240503 01:59:09.988396 3955 raft_consensus.cc:2932] T 0b669a47f1d14ce98ccab24479d39304 P 4125b1a70e864e129bd54bdc2720d7fa [term 1 FOLLOWER]: Committing config change with OpId 1.140: config changed from index 119 to 140, 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 140 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { replace: true } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:09.988957 2393 raft_consensus.cc:1261] T 0b669a47f1d14ce98ccab24479d39304 P b63928177fa54227a5ff9929265ab9dd [term 1 FOLLOWER]: Refusing update from remote peer b40a9a7b30d34a6db007ed8febd8a759: Log matching property violated. Preceding OpId in replica: term: 1 index: 139. Preceding OpId from leader: term: 1 index: 140. (index mismatch)
I20240503 01:59:09.992045 4551 consensus_queue.cc:1035] T 0b669a47f1d14ce98ccab24479d39304 P b40a9a7b30d34a6db007ed8febd8a759 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 140, Last known committed idx: 139, Time since last communication: 0.000s
I20240503 01:59:09.994138 2393 raft_consensus.cc:2932] T 0b669a47f1d14ce98ccab24479d39304 P b63928177fa54227a5ff9929265ab9dd [term 1 FOLLOWER]: Committing config change with OpId 1.140: config changed from index 119 to 140, 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 140 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { replace: true } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:09.998169 2661 consensus_queue.cc:237] T 0b669a47f1d14ce98ccab24479d39304 P b40a9a7b30d34a6db007ed8febd8a759 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 140, Committed index: 140, Last appended: 1.140, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 141 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } }
I20240503 01:59:10.000445 3052 raft_consensus.cc:1261] T 0b669a47f1d14ce98ccab24479d39304 P 6568f69a78c44bb985162d2b7d0a9376 [term 1 FOLLOWER]: Refusing update from remote peer b40a9a7b30d34a6db007ed8febd8a759: Log matching property violated. Preceding OpId in replica: term: 1 index: 140. Preceding OpId from leader: term: 1 index: 141. (index mismatch)
I20240503 01:59:10.000844 4667 consensus_queue.cc:1035] T 0b669a47f1d14ce98ccab24479d39304 P b40a9a7b30d34a6db007ed8febd8a759 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 141, Last known committed idx: 140, Time since last communication: 0.000s
I20240503 01:59:10.001930 3954 raft_consensus.cc:1261] T 0b669a47f1d14ce98ccab24479d39304 P 4125b1a70e864e129bd54bdc2720d7fa [term 1 FOLLOWER]: Refusing update from remote peer b40a9a7b30d34a6db007ed8febd8a759: Log matching property violated. Preceding OpId in replica: term: 1 index: 140. Preceding OpId from leader: term: 1 index: 141. (index mismatch)
I20240503 01:59:10.002607 2525 raft_consensus.cc:1261] T 0b669a47f1d14ce98ccab24479d39304 P a1a844ecd71145e4a362f4c42500fbcb [term 1 FOLLOWER]: Refusing update from remote peer b40a9a7b30d34a6db007ed8febd8a759: Log matching property violated. Preceding OpId in replica: term: 1 index: 140. Preceding OpId from leader: term: 1 index: 141. (index mismatch)
I20240503 01:59:10.003268 4251 consensus_queue.cc:1035] T 0b669a47f1d14ce98ccab24479d39304 P b40a9a7b30d34a6db007ed8febd8a759 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 141, Last known committed idx: 140, Time since last communication: 0.000s
I20240503 01:59:10.004271 4551 consensus_queue.cc:1035] T 0b669a47f1d14ce98ccab24479d39304 P b40a9a7b30d34a6db007ed8febd8a759 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 141, Last known committed idx: 140, Time since last communication: 0.000s
I20240503 01:59:10.007176 4667 raft_consensus.cc:2932] T 0b669a47f1d14ce98ccab24479d39304 P b40a9a7b30d34a6db007ed8febd8a759 [term 1 LEADER]: Committing config change with OpId 1.141: config changed from index 140 to 141, VOTER b63928177fa54227a5ff9929265ab9dd (127.0.99.129) evicted. New config: { opid_index: 141 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:10.008320 2787 raft_consensus.cc:1261] T 0b669a47f1d14ce98ccab24479d39304 P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Refusing update from remote peer b40a9a7b30d34a6db007ed8febd8a759: Log matching property violated. Preceding OpId in replica: term: 1 index: 140. Preceding OpId from leader: term: 1 index: 141. (index mismatch)
I20240503 01:59:10.010118 2242 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0b669a47f1d14ce98ccab24479d39304 with cas_config_opid_index 140: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:10.010078 3052 raft_consensus.cc:2932] T 0b669a47f1d14ce98ccab24479d39304 P 6568f69a78c44bb985162d2b7d0a9376 [term 1 FOLLOWER]: Committing config change with OpId 1.141: config changed from index 140 to 141, VOTER b63928177fa54227a5ff9929265ab9dd (127.0.99.129) evicted. New config: { opid_index: 141 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:10.011921 4667 consensus_queue.cc:1035] T 0b669a47f1d14ce98ccab24479d39304 P b40a9a7b30d34a6db007ed8febd8a759 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 141, Last known committed idx: 140, Time since last communication: 0.000s
I20240503 01:59:10.013552 2787 raft_consensus.cc:2932] T 0b669a47f1d14ce98ccab24479d39304 P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Committing config change with OpId 1.141: config changed from index 140 to 141, VOTER b63928177fa54227a5ff9929265ab9dd (127.0.99.129) evicted. New config: { opid_index: 141 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:10.014726 2250 catalog_manager.cc:5484] T 0b669a47f1d14ce98ccab24479d39304 P 6568f69a78c44bb985162d2b7d0a9376 reported cstate change: config changed from index 140 to 141, VOTER b63928177fa54227a5ff9929265ab9dd (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "b40a9a7b30d34a6db007ed8febd8a759" committed_config { opid_index: 141 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:10.019436 3954 raft_consensus.cc:2932] T 0b669a47f1d14ce98ccab24479d39304 P 4125b1a70e864e129bd54bdc2720d7fa [term 1 FOLLOWER]: Committing config change with OpId 1.141: config changed from index 140 to 141, VOTER b63928177fa54227a5ff9929265ab9dd (127.0.99.129) evicted. New config: { opid_index: 141 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:10.026147 2378 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b669a47f1d14ce98ccab24479d39304 with delete_type TABLET_DATA_TOMBSTONED (TS b63928177fa54227a5ff9929265ab9dd not found in new config with opid_index 141) from {username='slave'} at 127.0.0.1:44104
I20240503 01:59:10.026371 4699 tablet_replica.cc:330] T 0b669a47f1d14ce98ccab24479d39304 P b63928177fa54227a5ff9929265ab9dd: stopping tablet replica
I20240503 01:59:10.026512 4699 raft_consensus.cc:2229] T 0b669a47f1d14ce98ccab24479d39304 P b63928177fa54227a5ff9929265ab9dd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:10.026666 4699 raft_consensus.cc:2258] T 0b669a47f1d14ce98ccab24479d39304 P b63928177fa54227a5ff9929265ab9dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:10.027803 4699 ts_tablet_manager.cc:1875] T 0b669a47f1d14ce98ccab24479d39304 P b63928177fa54227a5ff9929265ab9dd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:10.030001 4699 ts_tablet_manager.cc:1888] T 0b669a47f1d14ce98ccab24479d39304 P b63928177fa54227a5ff9929265ab9dd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.140
I20240503 01:59:10.030066 4699 log.cc:1196] T 0b669a47f1d14ce98ccab24479d39304 P b63928177fa54227a5ff9929265ab9dd: Deleting WAL directory at /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/0b669a47f1d14ce98ccab24479d39304
I20240503 01:59:10.034806 2520 raft_consensus.cc:2932] T 0b669a47f1d14ce98ccab24479d39304 P a1a844ecd71145e4a362f4c42500fbcb [term 1 FOLLOWER]: Committing config change with OpId 1.141: config changed from index 140 to 141, VOTER b63928177fa54227a5ff9929265ab9dd (127.0.99.129) evicted. New config: { opid_index: 141 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } }
I20240503 01:59:10.035315 2245 catalog_manager.cc:4830] TS b63928177fa54227a5ff9929265ab9dd (127.0.99.129:36425): tablet 0b669a47f1d14ce98ccab24479d39304 (table rebalance_test_table_4 [id=e8f9ec3f55d044f8b23964df20e79f65]) successfully deleted
I20240503 01:59:10.653049 2904 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 01:59:10.717216 2772 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:10.724830 3938 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:10.764092 2641 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:10.779357 2378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:10.850795 2510 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:10.859601 3035 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 01:59:11.213923 2991 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.129:36425, user_credentials={real_user=slave}} blocked reactor thread for 45765.4us
I20240503 01:59:11.829326 2904 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 01:59:11.851562 2510 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:11.871285 3035 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 01:59:11.921051 2772 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:11.921972 2378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:11.944972 2641 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:11.978579 3938 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 01:59:12.133147 2991 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.135:39501, user_credentials={real_user=slave}} blocked reactor thread for 44340us
W20240503 01:59:13.098023 2860 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.129:36425, user_credentials={real_user=slave}} blocked reactor thread for 134626us
I20240503 01:59:13.395377 2510 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:13.399295 2641 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:13.420605 2378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:13.452908 3035 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 01:59:13.455178 2772 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:13.465801 2904 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 01:59:13.476507 3938 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 01:59:14.573706 2991 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.135:39501, user_credentials={real_user=slave}} blocked reactor thread for 44657.8us
I20240503 01:59:14.664232 3035 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 01:59:14.691365 3938 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:14.733112 2378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:14.735410 2904 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 01:59:14.764814 2772 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:14.767441 2510 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 01:59:14.871443 2860 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.135:39501, user_credentials={real_user=slave}} blocked reactor thread for 44948.7us
I20240503 01:59:14.880960 2641 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 01:59:15.186429 2860 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.129:36425, user_credentials={real_user=slave}} blocked reactor thread for 83030.3us
I20240503 01:59:15.371349 3035 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 01:59:15.372136 3938 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:15.372413 2904 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 01:59:15.372743 2772 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:15.380193 2641 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:15.393292 2510 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:15.393352 2378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
ab13918717ec4184908d11fa45b8012a | 127.0.99.190:38489 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.99.148:35375 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
327aa26a8e434a609ebcb382aa665fce | 127.0.99.133:38853 | HEALTHY | <none> | 5 | 0
4125b1a70e864e129bd54bdc2720d7fa | 127.0.99.135:39501 | HEALTHY | <none> | 0 | 0
52c196c15e3348b985c689652bc49918 | 127.0.99.132:42887 | HEALTHY | <none> | 0 | 0
6568f69a78c44bb985162d2b7d0a9376 | 127.0.99.134:43603 | HEALTHY | <none> | 4 | 0
a1a844ecd71145e4a362f4c42500fbcb | 127.0.99.130:46033 | HEALTHY | <none> | 1 | 0
b40a9a7b30d34a6db007ed8febd8a759 | 127.0.99.131:34079 | HEALTHY | <none> | 3 | 0
b63928177fa54227a5ff9929265ab9dd | 127.0.99.129:36425 | HEALTHY | <none> | 3 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 7
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 7 server(s) checked
local_ip_for_outbound_sockets | 127.0.99.129 | experimental | 127.0.99.129:36425
local_ip_for_outbound_sockets | 127.0.99.130 | experimental | 127.0.99.130:46033
local_ip_for_outbound_sockets | 127.0.99.131 | experimental | 127.0.99.131:34079
local_ip_for_outbound_sockets | 127.0.99.132 | experimental | 127.0.99.132:42887
local_ip_for_outbound_sockets | 127.0.99.133 | experimental | 127.0.99.133:38853
local_ip_for_outbound_sockets | 127.0.99.134 | experimental | 127.0.99.134:43603
local_ip_for_outbound_sockets | 127.0.99.135 | experimental | 127.0.99.135:39501
never_fsync | true | unsafe,advanced | all 7 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 7 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 7 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 7 server(s) checked
server_dump_info_format | pb | hidden | all 7 server(s) checked
server_dump_info_path | /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.99.129:36425
server_dump_info_path | /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.0.99.130:46033
server_dump_info_path | /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.99.131:34079
server_dump_info_path | /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.0.99.132:42887
server_dump_info_path | /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.0.99.133:38853
server_dump_info_path | /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.0.99.134:43603
server_dump_info_path | /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.0.99.135:39501
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.99.148:35375 | all 7 server(s) checked
time_source | builtin | all 7 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 8 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 5 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 5 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 5 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 5 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 5 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 5 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 11
First Quartile | 11
Median | 11
Third Quartile | 12
Maximum | 12
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 7
Tables | 6
Tablets | 16
Replicas | 80
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240503 01:59:15.432078 398 log_verifier.cc:126] Checking tablet 0b669a47f1d14ce98ccab24479d39304
I20240503 01:59:15.596107 398 log_verifier.cc:177] Verified matching terms for 220 ops in tablet 0b669a47f1d14ce98ccab24479d39304
I20240503 01:59:15.596331 398 log_verifier.cc:126] Checking tablet 10ecacdfb9a44d068b0df24e991612ee
I20240503 01:59:15.743011 398 log_verifier.cc:177] Verified matching terms for 148 ops in tablet 10ecacdfb9a44d068b0df24e991612ee
I20240503 01:59:15.743171 398 log_verifier.cc:126] Checking tablet 19b3a90e13194997bde2b9a806f8f2b5
I20240503 01:59:15.912181 398 log_verifier.cc:177] Verified matching terms for 240 ops in tablet 19b3a90e13194997bde2b9a806f8f2b5
I20240503 01:59:15.912412 398 log_verifier.cc:126] Checking tablet 240749b0904f4bb2a4c79b7713473411
I20240503 01:59:15.999109 398 log_verifier.cc:177] Verified matching terms for 4 ops in tablet 240749b0904f4bb2a4c79b7713473411
I20240503 01:59:15.999222 398 log_verifier.cc:126] Checking tablet 43008abe74604666b330d78ec0f51c7c
I20240503 01:59:16.184830 398 log_verifier.cc:177] Verified matching terms for 278 ops in tablet 43008abe74604666b330d78ec0f51c7c
I20240503 01:59:16.185081 398 log_verifier.cc:126] Checking tablet 568a05972208414c90f7f4e2ea2c10a9
I20240503 01:59:16.372983 398 log_verifier.cc:177] Verified matching terms for 293 ops in tablet 568a05972208414c90f7f4e2ea2c10a9
I20240503 01:59:16.373245 398 log_verifier.cc:126] Checking tablet 5fd1010447524b6695abb98192dc54f1
I20240503 01:59:16.508065 398 log_verifier.cc:177] Verified matching terms for 154 ops in tablet 5fd1010447524b6695abb98192dc54f1
I20240503 01:59:16.508190 398 log_verifier.cc:126] Checking tablet 8682bff56e604cc3b7e9b07c8ba3fca5
I20240503 01:59:16.649160 398 log_verifier.cc:177] Verified matching terms for 155 ops in tablet 8682bff56e604cc3b7e9b07c8ba3fca5
I20240503 01:59:16.649297 398 log_verifier.cc:126] Checking tablet 8f4c3f70f2714cc299bc0d595f838c85
I20240503 01:59:16.807142 398 log_verifier.cc:177] Verified matching terms for 217 ops in tablet 8f4c3f70f2714cc299bc0d595f838c85
I20240503 01:59:16.807299 398 log_verifier.cc:126] Checking tablet 9d84cbdf02c14873b8838be296977aaf
I20240503 01:59:16.950399 398 log_verifier.cc:177] Verified matching terms for 176 ops in tablet 9d84cbdf02c14873b8838be296977aaf
I20240503 01:59:16.950549 398 log_verifier.cc:126] Checking tablet a6e17a2c4ba0482e96d04eb9d3ba854a
I20240503 01:59:17.099644 398 log_verifier.cc:177] Verified matching terms for 199 ops in tablet a6e17a2c4ba0482e96d04eb9d3ba854a
I20240503 01:59:17.099781 398 log_verifier.cc:126] Checking tablet c10343409a2a423eb0b21759bfc9ba61
I20240503 01:59:17.240797 398 log_verifier.cc:177] Verified matching terms for 179 ops in tablet c10343409a2a423eb0b21759bfc9ba61
I20240503 01:59:17.240938 398 log_verifier.cc:126] Checking tablet c2911cf48f384bf09074f7bea2a4c173
I20240503 01:59:17.403172 398 log_verifier.cc:177] Verified matching terms for 248 ops in tablet c2911cf48f384bf09074f7bea2a4c173
I20240503 01:59:17.403426 398 log_verifier.cc:126] Checking tablet c9787dcf6d674611a2f06c143a291bb5
I20240503 01:59:17.577220 398 log_verifier.cc:177] Verified matching terms for 262 ops in tablet c9787dcf6d674611a2f06c143a291bb5
I20240503 01:59:17.577416 398 log_verifier.cc:126] Checking tablet d4d07c9621134780bc7ece7bb061d934
I20240503 01:59:17.734721 398 log_verifier.cc:177] Verified matching terms for 216 ops in tablet d4d07c9621134780bc7ece7bb061d934
I20240503 01:59:17.734964 398 log_verifier.cc:126] Checking tablet f40f77238d1e4a34a3f7bea387c6406c
I20240503 01:59:17.885396 398 log_verifier.cc:177] Verified matching terms for 206 ops in tablet f40f77238d1e4a34a3f7bea387c6406c
I20240503 01:59:17.887502 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task42rE7p/build/debug/bin/kudu
/tmp/dist-test-task42rE7p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.136:0
--local_ip_for_outbound_sockets=127.0.99.136
--webserver_interface=127.0.99.136
--webserver_port=0
--tserver_master_addrs=127.0.99.190:38489
--builtin_ntp_servers=127.0.99.148:35375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 01:59:18.015427 4985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:18.015666 4985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:59:18.015712 4985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:18.018828 4985 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:59:18.018909 4985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:59:18.019006 4985 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.136
I20240503 01:59:18.022931 4985 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.136:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.0.99.136
--webserver_port=0
--tserver_master_addrs=127.0.99.190:38489
--never_fsync=true
--heap_profile_path=/tmp/kudu.4985
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.136
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:59:18.023970 4985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:59:18.025264 4985 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:18.035300 4990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:18.036500 4991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:18.037361 4993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:18.038520 4985 server_base.cc:1034] running on GCE node
I20240503 01:59:18.038991 4985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:59:18.039642 4985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:59:18.040820 4985 hybrid_clock.cc:648] HybridClock initialized: now 1714701558040769 us; error 56 us; skew 500 ppm
I20240503 01:59:18.042740 4985 webserver.cc:460] Webserver started at http://127.0.99.136:45491/ using document root <none> and password file <none>
I20240503 01:59:18.043279 4985 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:18.043340 4985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:18.043525 4985 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:59:18.045239 4985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "c4a1b28608a6413caaf5980d39b99223"
format_stamp: "Formatted at 2024-05-03 01:59:18 on dist-test-slave-dtz3"
I20240503 01:59:18.045707 4985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "c4a1b28608a6413caaf5980d39b99223"
format_stamp: "Formatted at 2024-05-03 01:59:18 on dist-test-slave-dtz3"
I20240503 01:59:18.049432 4985 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240503 01:59:18.052250 4999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:18.053714 4985 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 01:59:18.053843 4985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "c4a1b28608a6413caaf5980d39b99223"
format_stamp: "Formatted at 2024-05-03 01:59:18 on dist-test-slave-dtz3"
I20240503 01:59:18.054013 4985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:18.080775 4985 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:18.081677 4985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:59:18.081851 4985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:18.082609 4985 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:18.083751 4985 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:59:18.083849 4985 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:18.083933 4985 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:59:18.083997 4985 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:18.097923 4985 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.136:33113
I20240503 01:59:18.098169 5112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.136:33113 every 8 connection(s)
I20240503 01:59:18.099014 4985 server_base.cc:1166] Dumped server information to /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 01:59:18.104818 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task42rE7p/build/debug/bin/kudu as pid 4985
I20240503 01:59:18.104993 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240503 01:59:18.110354 5113 heartbeater.cc:347] Connected to a master server at 127.0.99.190:38489
I20240503 01:59:18.110724 5113 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:18.111616 5113 heartbeater.cc:511] Master 127.0.99.190:38489 requested a full tablet report, sending...
I20240503 01:59:18.112943 2251 ts_manager.cc:194] Registered new tserver with Master: c4a1b28608a6413caaf5980d39b99223 (127.0.99.136:33113)
I20240503 01:59:18.113512 2251 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.136:46301
I20240503 01:59:18.348111 3938 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:18.348351 2904 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 01:59:18.352212 3035 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 01:59:18.352285 2641 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:18.375921 2510 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:18.383052 2772 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:18.392227 2378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:18.404451 5047 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:18.513547 2904 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 01:59:18.518766 2772 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:18.521603 3938 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:18.526387 5047 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:18.526403 2510 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:18.526903 2378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:18.543521 2641 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:18.562764 3035 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 01:59:18.607620 2660 consensus_queue.cc:237] T 0b669a47f1d14ce98ccab24479d39304 P b40a9a7b30d34a6db007ed8febd8a759 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 220, Committed index: 220, Last appended: 1.220, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 221 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } attrs { replace: true } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: true } }
I20240503 01:59:18.610354 3957 raft_consensus.cc:1261] T 0b669a47f1d14ce98ccab24479d39304 P 4125b1a70e864e129bd54bdc2720d7fa [term 1 FOLLOWER]: Refusing update from remote peer b40a9a7b30d34a6db007ed8febd8a759: Log matching property violated. Preceding OpId in replica: term: 1 index: 220. Preceding OpId from leader: term: 1 index: 221. (index mismatch)
I20240503 01:59:18.610347 2787 raft_consensus.cc:1261] T 0b669a47f1d14ce98ccab24479d39304 P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Refusing update from remote peer b40a9a7b30d34a6db007ed8febd8a759: Log matching property violated. Preceding OpId in replica: term: 1 index: 220. Preceding OpId from leader: term: 1 index: 221. (index mismatch)
I20240503 01:59:18.610850 2525 raft_consensus.cc:1261] T 0b669a47f1d14ce98ccab24479d39304 P a1a844ecd71145e4a362f4c42500fbcb [term 1 FOLLOWER]: Refusing update from remote peer b40a9a7b30d34a6db007ed8febd8a759: Log matching property violated. Preceding OpId in replica: term: 1 index: 220. Preceding OpId from leader: term: 1 index: 221. (index mismatch)
I20240503 01:59:18.610970 5208 consensus_queue.cc:1035] T 0b669a47f1d14ce98ccab24479d39304 P b40a9a7b30d34a6db007ed8febd8a759 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 221, Last known committed idx: 220, Time since last communication: 0.000s
I20240503 01:59:18.611495 5210 consensus_queue.cc:1035] T 0b669a47f1d14ce98ccab24479d39304 P b40a9a7b30d34a6db007ed8febd8a759 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 221, Last known committed idx: 220, Time since last communication: 0.000s
I20240503 01:59:18.613571 5208 consensus_queue.cc:1035] T 0b669a47f1d14ce98ccab24479d39304 P b40a9a7b30d34a6db007ed8febd8a759 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 221, Last known committed idx: 220, Time since last communication: 0.000s
I20240503 01:59:18.613644 3054 raft_consensus.cc:1261] T 0b669a47f1d14ce98ccab24479d39304 P 6568f69a78c44bb985162d2b7d0a9376 [term 1 FOLLOWER]: Refusing update from remote peer b40a9a7b30d34a6db007ed8febd8a759: Log matching property violated. Preceding OpId in replica: term: 1 index: 220. Preceding OpId from leader: term: 1 index: 221. (index mismatch)
I20240503 01:59:18.614598 5210 consensus_queue.cc:1035] T 0b669a47f1d14ce98ccab24479d39304 P b40a9a7b30d34a6db007ed8febd8a759 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 221, Last known committed idx: 220, Time since last communication: 0.000s
W20240503 01:59:18.615262 2595 consensus_peers.cc:485] T 0b669a47f1d14ce98ccab24479d39304 P b40a9a7b30d34a6db007ed8febd8a759 -> Peer c4a1b28608a6413caaf5980d39b99223 (127.0.99.136:33113): Couldn't send request to peer c4a1b28608a6413caaf5980d39b99223. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0b669a47f1d14ce98ccab24479d39304. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:18.616294 5210 raft_consensus.cc:2932] T 0b669a47f1d14ce98ccab24479d39304 P b40a9a7b30d34a6db007ed8febd8a759 [term 1 LEADER]: Committing config change with OpId 1.221: config changed from index 141 to 221, NON_VOTER c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) added. New config: { opid_index: 221 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } attrs { replace: true } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: true } } }
I20240503 01:59:18.616894 3054 raft_consensus.cc:2932] T 0b669a47f1d14ce98ccab24479d39304 P 6568f69a78c44bb985162d2b7d0a9376 [term 1 FOLLOWER]: Committing config change with OpId 1.221: config changed from index 141 to 221, NON_VOTER c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) added. New config: { opid_index: 221 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } attrs { replace: true } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: true } } }
I20240503 01:59:18.616997 2523 raft_consensus.cc:2932] T 0b669a47f1d14ce98ccab24479d39304 P a1a844ecd71145e4a362f4c42500fbcb [term 1 FOLLOWER]: Committing config change with OpId 1.221: config changed from index 141 to 221, NON_VOTER c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) added. New config: { opid_index: 221 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } attrs { replace: true } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: true } } }
I20240503 01:59:18.617110 3957 raft_consensus.cc:2932] T 0b669a47f1d14ce98ccab24479d39304 P 4125b1a70e864e129bd54bdc2720d7fa [term 1 FOLLOWER]: Committing config change with OpId 1.221: config changed from index 141 to 221, NON_VOTER c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) added. New config: { opid_index: 221 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } attrs { replace: true } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: true } } }
I20240503 01:59:18.620709 2251 catalog_manager.cc:5484] T 0b669a47f1d14ce98ccab24479d39304 P b40a9a7b30d34a6db007ed8febd8a759 reported cstate change: config changed from index 141 to 221, NON_VOTER c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "b40a9a7b30d34a6db007ed8febd8a759" committed_config { opid_index: 221 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:18.620795 2787 raft_consensus.cc:2932] T 0b669a47f1d14ce98ccab24479d39304 P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Committing config change with OpId 1.221: config changed from index 141 to 221, NON_VOTER c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) added. New config: { opid_index: 221 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } attrs { replace: true } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: true } } }
I20240503 01:59:18.630316 3054 consensus_queue.cc:237] T c2911cf48f384bf09074f7bea2a4c173 P 6568f69a78c44bb985162d2b7d0a9376 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 248, Committed index: 248, Last appended: 1.248, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 249 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: true } }
I20240503 01:59:18.632236 3957 raft_consensus.cc:1261] T c2911cf48f384bf09074f7bea2a4c173 P 4125b1a70e864e129bd54bdc2720d7fa [term 1 FOLLOWER]: Refusing update from remote peer 6568f69a78c44bb985162d2b7d0a9376: Log matching property violated. Preceding OpId in replica: term: 1 index: 248. Preceding OpId from leader: term: 1 index: 249. (index mismatch)
I20240503 01:59:18.632643 2398 raft_consensus.cc:1261] T c2911cf48f384bf09074f7bea2a4c173 P b63928177fa54227a5ff9929265ab9dd [term 1 FOLLOWER]: Refusing update from remote peer 6568f69a78c44bb985162d2b7d0a9376: Log matching property violated. Preceding OpId in replica: term: 1 index: 248. Preceding OpId from leader: term: 1 index: 249. (index mismatch)
I20240503 01:59:18.632769 2660 raft_consensus.cc:1261] T c2911cf48f384bf09074f7bea2a4c173 P b40a9a7b30d34a6db007ed8febd8a759 [term 1 FOLLOWER]: Refusing update from remote peer 6568f69a78c44bb985162d2b7d0a9376: Log matching property violated. Preceding OpId in replica: term: 1 index: 248. Preceding OpId from leader: term: 1 index: 249. (index mismatch)
I20240503 01:59:18.633136 5229 consensus_queue.cc:1035] T c2911cf48f384bf09074f7bea2a4c173 P 6568f69a78c44bb985162d2b7d0a9376 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 249, Last known committed idx: 248, Time since last communication: 0.000s
I20240503 01:59:18.633415 2924 raft_consensus.cc:1261] T c2911cf48f384bf09074f7bea2a4c173 P 327aa26a8e434a609ebcb382aa665fce [term 1 FOLLOWER]: Refusing update from remote peer 6568f69a78c44bb985162d2b7d0a9376: Log matching property violated. Preceding OpId in replica: term: 1 index: 248. Preceding OpId from leader: term: 1 index: 249. (index mismatch)
I20240503 01:59:18.633531 4983 consensus_queue.cc:1035] T c2911cf48f384bf09074f7bea2a4c173 P 6568f69a78c44bb985162d2b7d0a9376 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 249, Last known committed idx: 248, Time since last communication: 0.000s
I20240503 01:59:18.633657 5228 consensus_queue.cc:1035] T c2911cf48f384bf09074f7bea2a4c173 P 6568f69a78c44bb985162d2b7d0a9376 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 249, Last known committed idx: 248, Time since last communication: 0.000s
I20240503 01:59:18.633940 4983 consensus_queue.cc:1035] T c2911cf48f384bf09074f7bea2a4c173 P 6568f69a78c44bb985162d2b7d0a9376 [LEADER]: Connected to new peer: Peer: permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 249, Last known committed idx: 248, Time since last communication: 0.000s
I20240503 01:59:18.636466 5229 raft_consensus.cc:2932] T c2911cf48f384bf09074f7bea2a4c173 P 6568f69a78c44bb985162d2b7d0a9376 [term 1 LEADER]: Committing config change with OpId 1.249: config changed from index 143 to 249, NON_VOTER c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) added. New config: { opid_index: 249 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: true } } }
I20240503 01:59:18.636966 2398 raft_consensus.cc:2932] T c2911cf48f384bf09074f7bea2a4c173 P b63928177fa54227a5ff9929265ab9dd [term 1 FOLLOWER]: Committing config change with OpId 1.249: config changed from index 143 to 249, NON_VOTER c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) added. New config: { opid_index: 249 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: true } } }
I20240503 01:59:18.637087 2660 raft_consensus.cc:2932] T c2911cf48f384bf09074f7bea2a4c173 P b40a9a7b30d34a6db007ed8febd8a759 [term 1 FOLLOWER]: Committing config change with OpId 1.249: config changed from index 143 to 249, NON_VOTER c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) added. New config: { opid_index: 249 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: true } } }
I20240503 01:59:18.639840 2924 raft_consensus.cc:2932] T c2911cf48f384bf09074f7bea2a4c173 P 327aa26a8e434a609ebcb382aa665fce [term 1 FOLLOWER]: Committing config change with OpId 1.249: config changed from index 143 to 249, NON_VOTER c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) added. New config: { opid_index: 249 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: true } } }
I20240503 01:59:18.640518 2252 catalog_manager.cc:5484] T c2911cf48f384bf09074f7bea2a4c173 P b40a9a7b30d34a6db007ed8febd8a759 reported cstate change: config changed from index 143 to 249, NON_VOTER c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "6568f69a78c44bb985162d2b7d0a9376" committed_config { opid_index: 249 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: true } } }
W20240503 01:59:18.641752 2988 consensus_peers.cc:485] T c2911cf48f384bf09074f7bea2a4c173 P 6568f69a78c44bb985162d2b7d0a9376 -> Peer c4a1b28608a6413caaf5980d39b99223 (127.0.99.136:33113): Couldn't send request to peer c4a1b28608a6413caaf5980d39b99223. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c2911cf48f384bf09074f7bea2a4c173. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:18.645107 3957 raft_consensus.cc:2932] T c2911cf48f384bf09074f7bea2a4c173 P 4125b1a70e864e129bd54bdc2720d7fa [term 1 FOLLOWER]: Committing config change with OpId 1.249: config changed from index 143 to 249, NON_VOTER c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) added. New config: { opid_index: 249 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: true } } }
I20240503 01:59:18.651893 3054 consensus_queue.cc:237] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 6568f69a78c44bb985162d2b7d0a9376 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 155, Committed index: 155, Last appended: 1.155, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 156 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { replace: true } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: true } }
I20240503 01:59:18.653709 2924 raft_consensus.cc:1261] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 327aa26a8e434a609ebcb382aa665fce [term 1 FOLLOWER]: Refusing update from remote peer 6568f69a78c44bb985162d2b7d0a9376: Log matching property violated. Preceding OpId in replica: term: 1 index: 155. Preceding OpId from leader: term: 1 index: 156. (index mismatch)
I20240503 01:59:18.653681 3957 raft_consensus.cc:1261] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 4125b1a70e864e129bd54bdc2720d7fa [term 1 FOLLOWER]: Refusing update from remote peer 6568f69a78c44bb985162d2b7d0a9376: Log matching property violated. Preceding OpId in replica: term: 1 index: 155. Preceding OpId from leader: term: 1 index: 156. (index mismatch)
I20240503 01:59:18.653749 2787 raft_consensus.cc:1261] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Refusing update from remote peer 6568f69a78c44bb985162d2b7d0a9376: Log matching property violated. Preceding OpId in replica: term: 1 index: 155. Preceding OpId from leader: term: 1 index: 156. (index mismatch)
W20240503 01:59:18.653929 2988 consensus_peers.cc:485] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 6568f69a78c44bb985162d2b7d0a9376 -> Peer c4a1b28608a6413caaf5980d39b99223 (127.0.99.136:33113): Couldn't send request to peer c4a1b28608a6413caaf5980d39b99223. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8682bff56e604cc3b7e9b07c8ba3fca5. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:18.654266 5242 consensus_queue.cc:1035] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 6568f69a78c44bb985162d2b7d0a9376 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 156, Last known committed idx: 155, Time since last communication: 0.000s
I20240503 01:59:18.654403 2660 raft_consensus.cc:1261] T 8682bff56e604cc3b7e9b07c8ba3fca5 P b40a9a7b30d34a6db007ed8febd8a759 [term 1 FOLLOWER]: Refusing update from remote peer 6568f69a78c44bb985162d2b7d0a9376: Log matching property violated. Preceding OpId in replica: term: 1 index: 155. Preceding OpId from leader: term: 1 index: 156. (index mismatch)
I20240503 01:59:18.654490 4983 consensus_queue.cc:1035] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 6568f69a78c44bb985162d2b7d0a9376 [LEADER]: Connected to new peer: Peer: permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 156, Last known committed idx: 155, Time since last communication: 0.000s
I20240503 01:59:18.654662 5229 consensus_queue.cc:1035] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 6568f69a78c44bb985162d2b7d0a9376 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 156, Last known committed idx: 155, Time since last communication: 0.000s
I20240503 01:59:18.654897 5230 consensus_queue.cc:1035] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 6568f69a78c44bb985162d2b7d0a9376 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 156, Last known committed idx: 155, Time since last communication: 0.000s
I20240503 01:59:18.657477 4983 raft_consensus.cc:2932] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 6568f69a78c44bb985162d2b7d0a9376 [term 1 LEADER]: Committing config change with OpId 1.156: config changed from index 96 to 156, NON_VOTER c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) added. New config: { opid_index: 156 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { replace: true } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: true } } }
I20240503 01:59:18.657833 3957 raft_consensus.cc:2932] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 4125b1a70e864e129bd54bdc2720d7fa [term 1 FOLLOWER]: Committing config change with OpId 1.156: config changed from index 96 to 156, NON_VOTER c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) added. New config: { opid_index: 156 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { replace: true } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: true } } }
I20240503 01:59:18.658063 2660 raft_consensus.cc:2932] T 8682bff56e604cc3b7e9b07c8ba3fca5 P b40a9a7b30d34a6db007ed8febd8a759 [term 1 FOLLOWER]: Committing config change with OpId 1.156: config changed from index 96 to 156, NON_VOTER c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) added. New config: { opid_index: 156 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { replace: true } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: true } } }
I20240503 01:59:18.660851 2253 catalog_manager.cc:5484] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 4125b1a70e864e129bd54bdc2720d7fa reported cstate change: config changed from index 96 to 156, NON_VOTER c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "6568f69a78c44bb985162d2b7d0a9376" committed_config { opid_index: 156 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { replace: true } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: true } } }
I20240503 01:59:18.660962 2787 raft_consensus.cc:2932] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Committing config change with OpId 1.156: config changed from index 96 to 156, NON_VOTER c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) added. New config: { opid_index: 156 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { replace: true } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: true } } }
I20240503 01:59:18.661803 2924 raft_consensus.cc:2932] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 327aa26a8e434a609ebcb382aa665fce [term 1 FOLLOWER]: Committing config change with OpId 1.156: config changed from index 96 to 156, NON_VOTER c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) added. New config: { opid_index: 156 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { replace: true } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: true } } }
I20240503 01:59:18.671485 2660 consensus_queue.cc:237] T 240749b0904f4bb2a4c79b7713473411 P b40a9a7b30d34a6db007ed8febd8a759 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: true } }
I20240503 01:59:18.673338 3957 raft_consensus.cc:1261] T 240749b0904f4bb2a4c79b7713473411 P 4125b1a70e864e129bd54bdc2720d7fa [term 1 FOLLOWER]: Refusing update from remote peer b40a9a7b30d34a6db007ed8febd8a759: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:18.673413 3054 raft_consensus.cc:1261] T 240749b0904f4bb2a4c79b7713473411 P 6568f69a78c44bb985162d2b7d0a9376 [term 1 FOLLOWER]: Refusing update from remote peer b40a9a7b30d34a6db007ed8febd8a759: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:18.673480 2924 raft_consensus.cc:1261] T 240749b0904f4bb2a4c79b7713473411 P 327aa26a8e434a609ebcb382aa665fce [term 1 FOLLOWER]: Refusing update from remote peer b40a9a7b30d34a6db007ed8febd8a759: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240503 01:59:18.673609 2595 consensus_peers.cc:485] T 240749b0904f4bb2a4c79b7713473411 P b40a9a7b30d34a6db007ed8febd8a759 -> Peer c4a1b28608a6413caaf5980d39b99223 (127.0.99.136:33113): Couldn't send request to peer c4a1b28608a6413caaf5980d39b99223. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 240749b0904f4bb2a4c79b7713473411. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:18.673792 5209 consensus_queue.cc:1035] T 240749b0904f4bb2a4c79b7713473411 P b40a9a7b30d34a6db007ed8febd8a759 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:18.673981 5208 consensus_queue.cc:1035] T 240749b0904f4bb2a4c79b7713473411 P b40a9a7b30d34a6db007ed8febd8a759 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:18.674189 5210 consensus_queue.cc:1035] T 240749b0904f4bb2a4c79b7713473411 P b40a9a7b30d34a6db007ed8febd8a759 [LEADER]: Connected to new peer: Peer: permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:18.674413 2787 raft_consensus.cc:1261] T 240749b0904f4bb2a4c79b7713473411 P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Refusing update from remote peer b40a9a7b30d34a6db007ed8febd8a759: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:18.674911 5209 consensus_queue.cc:1035] T 240749b0904f4bb2a4c79b7713473411 P b40a9a7b30d34a6db007ed8febd8a759 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:18.676959 4251 raft_consensus.cc:2932] T 240749b0904f4bb2a4c79b7713473411 P b40a9a7b30d34a6db007ed8febd8a759 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: true } } }
I20240503 01:59:18.677306 2924 raft_consensus.cc:2932] T 240749b0904f4bb2a4c79b7713473411 P 327aa26a8e434a609ebcb382aa665fce [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: true } } }
I20240503 01:59:18.679291 3054 raft_consensus.cc:2932] T 240749b0904f4bb2a4c79b7713473411 P 6568f69a78c44bb985162d2b7d0a9376 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: true } } }
I20240503 01:59:18.680115 2257 catalog_manager.cc:5484] T 240749b0904f4bb2a4c79b7713473411 P 327aa26a8e434a609ebcb382aa665fce reported cstate change: config changed from index 4 to 5, NON_VOTER c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "b40a9a7b30d34a6db007ed8febd8a759" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: true } } }
I20240503 01:59:18.680838 3957 raft_consensus.cc:2932] T 240749b0904f4bb2a4c79b7713473411 P 4125b1a70e864e129bd54bdc2720d7fa [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: true } } }
I20240503 01:59:18.680868 2787 raft_consensus.cc:2932] T 240749b0904f4bb2a4c79b7713473411 P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: true } } }
I20240503 01:59:18.691341 2398 consensus_queue.cc:237] T f40f77238d1e4a34a3f7bea387c6406c P b63928177fa54227a5ff9929265ab9dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 206, Committed index: 206, Last appended: 1.206, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } attrs { replace: true } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: true } }
I20240503 01:59:18.693583 3054 raft_consensus.cc:1261] T f40f77238d1e4a34a3f7bea387c6406c P 6568f69a78c44bb985162d2b7d0a9376 [term 1 FOLLOWER]: Refusing update from remote peer b63928177fa54227a5ff9929265ab9dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 206. Preceding OpId from leader: term: 1 index: 207. (index mismatch)
I20240503 01:59:18.693758 2924 raft_consensus.cc:1261] T f40f77238d1e4a34a3f7bea387c6406c P 327aa26a8e434a609ebcb382aa665fce [term 1 FOLLOWER]: Refusing update from remote peer b63928177fa54227a5ff9929265ab9dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 206. Preceding OpId from leader: term: 1 index: 207. (index mismatch)
I20240503 01:59:18.694087 5264 consensus_queue.cc:1035] T f40f77238d1e4a34a3f7bea387c6406c P b63928177fa54227a5ff9929265ab9dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 207, Last known committed idx: 206, Time since last communication: 0.000s
I20240503 01:59:18.694320 2787 raft_consensus.cc:1261] T f40f77238d1e4a34a3f7bea387c6406c P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Refusing update from remote peer b63928177fa54227a5ff9929265ab9dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 206. Preceding OpId from leader: term: 1 index: 207. (index mismatch)
I20240503 01:59:18.694519 2525 raft_consensus.cc:1261] T f40f77238d1e4a34a3f7bea387c6406c P a1a844ecd71145e4a362f4c42500fbcb [term 1 FOLLOWER]: Refusing update from remote peer b63928177fa54227a5ff9929265ab9dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 206. Preceding OpId from leader: term: 1 index: 207. (index mismatch)
I20240503 01:59:18.694518 5264 consensus_queue.cc:1035] T f40f77238d1e4a34a3f7bea387c6406c P b63928177fa54227a5ff9929265ab9dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 207, Last known committed idx: 206, Time since last communication: 0.000s
I20240503 01:59:18.694967 5263 consensus_queue.cc:1035] T f40f77238d1e4a34a3f7bea387c6406c P b63928177fa54227a5ff9929265ab9dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 207, Last known committed idx: 206, Time since last communication: 0.000s
I20240503 01:59:18.695252 4982 consensus_queue.cc:1035] T f40f77238d1e4a34a3f7bea387c6406c P b63928177fa54227a5ff9929265ab9dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 207, Last known committed idx: 206, Time since last communication: 0.000s
I20240503 01:59:18.697388 5263 raft_consensus.cc:2932] T f40f77238d1e4a34a3f7bea387c6406c P b63928177fa54227a5ff9929265ab9dd [term 1 LEADER]: Committing config change with OpId 1.207: config changed from index -1 to 207, NON_VOTER c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) added. New config: { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } attrs { replace: true } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: true } } }
I20240503 01:59:18.697913 3054 raft_consensus.cc:2932] T f40f77238d1e4a34a3f7bea387c6406c P 6568f69a78c44bb985162d2b7d0a9376 [term 1 FOLLOWER]: Committing config change with OpId 1.207: config changed from index -1 to 207, NON_VOTER c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) added. New config: { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } attrs { replace: true } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: true } } }
W20240503 01:59:18.701427 2333 consensus_peers.cc:485] T f40f77238d1e4a34a3f7bea387c6406c P b63928177fa54227a5ff9929265ab9dd -> Peer c4a1b28608a6413caaf5980d39b99223 (127.0.99.136:33113): Couldn't send request to peer c4a1b28608a6413caaf5980d39b99223. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f40f77238d1e4a34a3f7bea387c6406c. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:18.701249 2253 catalog_manager.cc:5484] T f40f77238d1e4a34a3f7bea387c6406c P 6568f69a78c44bb985162d2b7d0a9376 reported cstate change: config changed from index -1 to 207, NON_VOTER c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "b63928177fa54227a5ff9929265ab9dd" committed_config { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } attrs { replace: true } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: true } } }
I20240503 01:59:18.701771 2787 raft_consensus.cc:2932] T f40f77238d1e4a34a3f7bea387c6406c P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Committing config change with OpId 1.207: config changed from index -1 to 207, NON_VOTER c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) added. New config: { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } attrs { replace: true } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: true } } }
I20240503 01:59:18.702895 2924 raft_consensus.cc:2932] T f40f77238d1e4a34a3f7bea387c6406c P 327aa26a8e434a609ebcb382aa665fce [term 1 FOLLOWER]: Committing config change with OpId 1.207: config changed from index -1 to 207, NON_VOTER c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) added. New config: { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } attrs { replace: true } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: true } } }
I20240503 01:59:18.704777 2525 raft_consensus.cc:2932] T f40f77238d1e4a34a3f7bea387c6406c P a1a844ecd71145e4a362f4c42500fbcb [term 1 FOLLOWER]: Committing config change with OpId 1.207: config changed from index -1 to 207, NON_VOTER c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) added. New config: { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } attrs { replace: true } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: true } } }
I20240503 01:59:19.116336 5113 heartbeater.cc:503] Master 127.0.99.190:38489 was elected leader, sending a full tablet report...
I20240503 01:59:19.159224 5321 ts_tablet_manager.cc:902] T 240749b0904f4bb2a4c79b7713473411 P c4a1b28608a6413caaf5980d39b99223: Initiating tablet copy from peer b40a9a7b30d34a6db007ed8febd8a759 (127.0.99.131:34079)
I20240503 01:59:19.160406 5321 tablet_copy_client.cc:308] T 240749b0904f4bb2a4c79b7713473411 P c4a1b28608a6413caaf5980d39b99223: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:34079
I20240503 01:59:19.160954 5326 ts_tablet_manager.cc:902] T 0b669a47f1d14ce98ccab24479d39304 P c4a1b28608a6413caaf5980d39b99223: Initiating tablet copy from peer b40a9a7b30d34a6db007ed8febd8a759 (127.0.99.131:34079)
I20240503 01:59:19.161634 5326 tablet_copy_client.cc:308] T 0b669a47f1d14ce98ccab24479d39304 P c4a1b28608a6413caaf5980d39b99223: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:34079
I20240503 01:59:19.164767 2682 tablet_copy_service.cc:140] P b40a9a7b30d34a6db007ed8febd8a759: Received BeginTabletCopySession request for tablet 240749b0904f4bb2a4c79b7713473411 from peer c4a1b28608a6413caaf5980d39b99223 ({username='slave'} at 127.0.99.136:58113)
I20240503 01:59:19.164921 2682 tablet_copy_service.cc:161] P b40a9a7b30d34a6db007ed8febd8a759: Beginning new tablet copy session on tablet 240749b0904f4bb2a4c79b7713473411 from peer c4a1b28608a6413caaf5980d39b99223 at {username='slave'} at 127.0.99.136:58113: session id = c4a1b28608a6413caaf5980d39b99223-240749b0904f4bb2a4c79b7713473411
I20240503 01:59:19.164794 2681 tablet_copy_service.cc:140] P b40a9a7b30d34a6db007ed8febd8a759: Received BeginTabletCopySession request for tablet 0b669a47f1d14ce98ccab24479d39304 from peer c4a1b28608a6413caaf5980d39b99223 ({username='slave'} at 127.0.99.136:58113)
I20240503 01:59:19.165107 2681 tablet_copy_service.cc:161] P b40a9a7b30d34a6db007ed8febd8a759: Beginning new tablet copy session on tablet 0b669a47f1d14ce98ccab24479d39304 from peer c4a1b28608a6413caaf5980d39b99223 at {username='slave'} at 127.0.99.136:58113: session id = c4a1b28608a6413caaf5980d39b99223-0b669a47f1d14ce98ccab24479d39304
I20240503 01:59:19.165874 2682 tablet_copy_source_session.cc:205] T 240749b0904f4bb2a4c79b7713473411 P b40a9a7b30d34a6db007ed8febd8a759: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:19.166019 2681 tablet_copy_source_session.cc:205] T 0b669a47f1d14ce98ccab24479d39304 P b40a9a7b30d34a6db007ed8febd8a759: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:19.169256 5321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 240749b0904f4bb2a4c79b7713473411. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:19.169703 5326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b669a47f1d14ce98ccab24479d39304. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:19.176579 5326 tablet_copy_client.cc:783] T 0b669a47f1d14ce98ccab24479d39304 P c4a1b28608a6413caaf5980d39b99223: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:19.176826 5326 tablet_copy_client.cc:647] T 0b669a47f1d14ce98ccab24479d39304 P c4a1b28608a6413caaf5980d39b99223: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:19.176942 5321 tablet_copy_client.cc:783] T 240749b0904f4bb2a4c79b7713473411 P c4a1b28608a6413caaf5980d39b99223: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:19.177121 5321 tablet_copy_client.cc:647] T 240749b0904f4bb2a4c79b7713473411 P c4a1b28608a6413caaf5980d39b99223: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:19.178846 5321 tablet_copy_client.cc:523] T 240749b0904f4bb2a4c79b7713473411 P c4a1b28608a6413caaf5980d39b99223: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:19.179172 5326 tablet_copy_client.cc:523] T 0b669a47f1d14ce98ccab24479d39304 P c4a1b28608a6413caaf5980d39b99223: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:19.180953 5321 tablet_bootstrap.cc:492] T 240749b0904f4bb2a4c79b7713473411 P c4a1b28608a6413caaf5980d39b99223: Bootstrap starting.
I20240503 01:59:19.180953 5326 tablet_bootstrap.cc:492] T 0b669a47f1d14ce98ccab24479d39304 P c4a1b28608a6413caaf5980d39b99223: Bootstrap starting.
I20240503 01:59:19.188174 5321 log.cc:824] T 240749b0904f4bb2a4c79b7713473411 P c4a1b28608a6413caaf5980d39b99223: Log is configured to *not* fsync() on all Append() calls
I20240503 01:59:19.196496 5321 tablet_bootstrap.cc:492] T 240749b0904f4bb2a4c79b7713473411 P c4a1b28608a6413caaf5980d39b99223: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:19.196947 5321 tablet_bootstrap.cc:492] T 240749b0904f4bb2a4c79b7713473411 P c4a1b28608a6413caaf5980d39b99223: Bootstrap complete.
I20240503 01:59:19.197321 5321 ts_tablet_manager.cc:1367] T 240749b0904f4bb2a4c79b7713473411 P c4a1b28608a6413caaf5980d39b99223: Time spent bootstrapping tablet: real 0.017s user 0.011s sys 0.000s
I20240503 01:59:19.201268 5321 raft_consensus.cc:348] T 240749b0904f4bb2a4c79b7713473411 P c4a1b28608a6413caaf5980d39b99223 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: true } }
I20240503 01:59:19.201609 5352 ts_tablet_manager.cc:902] T f40f77238d1e4a34a3f7bea387c6406c P c4a1b28608a6413caaf5980d39b99223: Initiating tablet copy from peer b63928177fa54227a5ff9929265ab9dd (127.0.99.129:36425)
I20240503 01:59:19.201606 5321 raft_consensus.cc:729] T 240749b0904f4bb2a4c79b7713473411 P c4a1b28608a6413caaf5980d39b99223 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c4a1b28608a6413caaf5980d39b99223, State: Initialized, Role: LEARNER
I20240503 01:59:19.202559 5352 tablet_copy_client.cc:308] T f40f77238d1e4a34a3f7bea387c6406c P c4a1b28608a6413caaf5980d39b99223: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:36425
I20240503 01:59:19.202862 5321 consensus_queue.cc:260] T 240749b0904f4bb2a4c79b7713473411 P c4a1b28608a6413caaf5980d39b99223 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: true } }
I20240503 01:59:19.204357 5321 ts_tablet_manager.cc:1398] T 240749b0904f4bb2a4c79b7713473411 P c4a1b28608a6413caaf5980d39b99223: Time spent starting tablet: real 0.007s user 0.000s sys 0.005s
I20240503 01:59:19.205350 2681 tablet_copy_service.cc:342] P b40a9a7b30d34a6db007ed8febd8a759: Request end of tablet copy session c4a1b28608a6413caaf5980d39b99223-240749b0904f4bb2a4c79b7713473411 received from {username='slave'} at 127.0.99.136:58113
I20240503 01:59:19.205451 2681 tablet_copy_service.cc:433] P b40a9a7b30d34a6db007ed8febd8a759: ending tablet copy session c4a1b28608a6413caaf5980d39b99223-240749b0904f4bb2a4c79b7713473411 on tablet 240749b0904f4bb2a4c79b7713473411 with peer c4a1b28608a6413caaf5980d39b99223
I20240503 01:59:19.208750 2420 tablet_copy_service.cc:140] P b63928177fa54227a5ff9929265ab9dd: Received BeginTabletCopySession request for tablet f40f77238d1e4a34a3f7bea387c6406c from peer c4a1b28608a6413caaf5980d39b99223 ({username='slave'} at 127.0.99.136:57695)
I20240503 01:59:19.208904 2420 tablet_copy_service.cc:161] P b63928177fa54227a5ff9929265ab9dd: Beginning new tablet copy session on tablet f40f77238d1e4a34a3f7bea387c6406c from peer c4a1b28608a6413caaf5980d39b99223 at {username='slave'} at 127.0.99.136:57695: session id = c4a1b28608a6413caaf5980d39b99223-f40f77238d1e4a34a3f7bea387c6406c
I20240503 01:59:19.209944 2420 tablet_copy_source_session.cc:205] T f40f77238d1e4a34a3f7bea387c6406c P b63928177fa54227a5ff9929265ab9dd: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:19.210951 5352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f40f77238d1e4a34a3f7bea387c6406c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:19.213176 5352 tablet_copy_client.cc:783] T f40f77238d1e4a34a3f7bea387c6406c P c4a1b28608a6413caaf5980d39b99223: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:19.213330 5352 tablet_copy_client.cc:647] T f40f77238d1e4a34a3f7bea387c6406c P c4a1b28608a6413caaf5980d39b99223: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:19.216069 5352 tablet_copy_client.cc:523] T f40f77238d1e4a34a3f7bea387c6406c P c4a1b28608a6413caaf5980d39b99223: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:19.217787 5352 tablet_bootstrap.cc:492] T f40f77238d1e4a34a3f7bea387c6406c P c4a1b28608a6413caaf5980d39b99223: Bootstrap starting.
I20240503 01:59:19.224200 5321 ts_tablet_manager.cc:902] T c2911cf48f384bf09074f7bea2a4c173 P c4a1b28608a6413caaf5980d39b99223: Initiating tablet copy from peer 6568f69a78c44bb985162d2b7d0a9376 (127.0.99.134:43603)
I20240503 01:59:19.224833 5321 tablet_copy_client.cc:308] T c2911cf48f384bf09074f7bea2a4c173 P c4a1b28608a6413caaf5980d39b99223: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.134:43603
I20240503 01:59:19.231628 3075 tablet_copy_service.cc:140] P 6568f69a78c44bb985162d2b7d0a9376: Received BeginTabletCopySession request for tablet c2911cf48f384bf09074f7bea2a4c173 from peer c4a1b28608a6413caaf5980d39b99223 ({username='slave'} at 127.0.99.136:59125)
I20240503 01:59:19.231761 3075 tablet_copy_service.cc:161] P 6568f69a78c44bb985162d2b7d0a9376: Beginning new tablet copy session on tablet c2911cf48f384bf09074f7bea2a4c173 from peer c4a1b28608a6413caaf5980d39b99223 at {username='slave'} at 127.0.99.136:59125: session id = c4a1b28608a6413caaf5980d39b99223-c2911cf48f384bf09074f7bea2a4c173
I20240503 01:59:19.232806 3075 tablet_copy_source_session.cc:205] T c2911cf48f384bf09074f7bea2a4c173 P 6568f69a78c44bb985162d2b7d0a9376: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:19.233776 5321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2911cf48f384bf09074f7bea2a4c173. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:19.236635 5321 tablet_copy_client.cc:783] T c2911cf48f384bf09074f7bea2a4c173 P c4a1b28608a6413caaf5980d39b99223: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:19.236843 5321 tablet_copy_client.cc:647] T c2911cf48f384bf09074f7bea2a4c173 P c4a1b28608a6413caaf5980d39b99223: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:19.240103 5321 tablet_copy_client.cc:523] T c2911cf48f384bf09074f7bea2a4c173 P c4a1b28608a6413caaf5980d39b99223: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:19.241482 5321 tablet_bootstrap.cc:492] T c2911cf48f384bf09074f7bea2a4c173 P c4a1b28608a6413caaf5980d39b99223: Bootstrap starting.
I20240503 01:59:19.252243 5371 ts_tablet_manager.cc:902] T 8682bff56e604cc3b7e9b07c8ba3fca5 P c4a1b28608a6413caaf5980d39b99223: Initiating tablet copy from peer 6568f69a78c44bb985162d2b7d0a9376 (127.0.99.134:43603)
I20240503 01:59:19.255559 5371 tablet_copy_client.cc:308] T 8682bff56e604cc3b7e9b07c8ba3fca5 P c4a1b28608a6413caaf5980d39b99223: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.134:43603
I20240503 01:59:19.256246 3075 tablet_copy_service.cc:140] P 6568f69a78c44bb985162d2b7d0a9376: Received BeginTabletCopySession request for tablet 8682bff56e604cc3b7e9b07c8ba3fca5 from peer c4a1b28608a6413caaf5980d39b99223 ({username='slave'} at 127.0.99.136:59125)
I20240503 01:59:19.256381 3075 tablet_copy_service.cc:161] P 6568f69a78c44bb985162d2b7d0a9376: Beginning new tablet copy session on tablet 8682bff56e604cc3b7e9b07c8ba3fca5 from peer c4a1b28608a6413caaf5980d39b99223 at {username='slave'} at 127.0.99.136:59125: session id = c4a1b28608a6413caaf5980d39b99223-8682bff56e604cc3b7e9b07c8ba3fca5
I20240503 01:59:19.257594 3075 tablet_copy_source_session.cc:205] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 6568f69a78c44bb985162d2b7d0a9376: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:19.258663 5371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8682bff56e604cc3b7e9b07c8ba3fca5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:19.261693 5371 tablet_copy_client.cc:783] T 8682bff56e604cc3b7e9b07c8ba3fca5 P c4a1b28608a6413caaf5980d39b99223: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:19.262223 5371 tablet_copy_client.cc:647] T 8682bff56e604cc3b7e9b07c8ba3fca5 P c4a1b28608a6413caaf5980d39b99223: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:19.264355 5371 tablet_copy_client.cc:523] T 8682bff56e604cc3b7e9b07c8ba3fca5 P c4a1b28608a6413caaf5980d39b99223: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:19.267244 5371 tablet_bootstrap.cc:492] T 8682bff56e604cc3b7e9b07c8ba3fca5 P c4a1b28608a6413caaf5980d39b99223: Bootstrap starting.
I20240503 01:59:19.390573 5326 tablet_bootstrap.cc:492] T 0b669a47f1d14ce98ccab24479d39304 P c4a1b28608a6413caaf5980d39b99223: Bootstrap replayed 1/1 log segments. Stats: ops{read=221 overwritten=0 applied=221 ignored=0} inserts{seen=903 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:19.391136 5326 tablet_bootstrap.cc:492] T 0b669a47f1d14ce98ccab24479d39304 P c4a1b28608a6413caaf5980d39b99223: Bootstrap complete.
I20240503 01:59:19.391422 5326 ts_tablet_manager.cc:1367] T 0b669a47f1d14ce98ccab24479d39304 P c4a1b28608a6413caaf5980d39b99223: Time spent bootstrapping tablet: real 0.211s user 0.165s sys 0.016s
I20240503 01:59:19.392500 5326 raft_consensus.cc:348] T 0b669a47f1d14ce98ccab24479d39304 P c4a1b28608a6413caaf5980d39b99223 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 221 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } attrs { replace: true } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: true } }
I20240503 01:59:19.393008 5326 raft_consensus.cc:729] T 0b669a47f1d14ce98ccab24479d39304 P c4a1b28608a6413caaf5980d39b99223 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c4a1b28608a6413caaf5980d39b99223, State: Initialized, Role: LEARNER
I20240503 01:59:19.393231 5326 consensus_queue.cc:260] T 0b669a47f1d14ce98ccab24479d39304 P c4a1b28608a6413caaf5980d39b99223 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 221, Last appended: 1.221, Last appended by leader: 221, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 221 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } attrs { replace: true } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: true } }
I20240503 01:59:19.393626 5326 ts_tablet_manager.cc:1398] T 0b669a47f1d14ce98ccab24479d39304 P c4a1b28608a6413caaf5980d39b99223: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:19.394559 2681 tablet_copy_service.cc:342] P b40a9a7b30d34a6db007ed8febd8a759: Request end of tablet copy session c4a1b28608a6413caaf5980d39b99223-0b669a47f1d14ce98ccab24479d39304 received from {username='slave'} at 127.0.99.136:58113
I20240503 01:59:19.394707 2681 tablet_copy_service.cc:433] P b40a9a7b30d34a6db007ed8febd8a759: ending tablet copy session c4a1b28608a6413caaf5980d39b99223-0b669a47f1d14ce98ccab24479d39304 on tablet 0b669a47f1d14ce98ccab24479d39304 with peer c4a1b28608a6413caaf5980d39b99223
I20240503 01:59:19.424194 5371 tablet_bootstrap.cc:492] T 8682bff56e604cc3b7e9b07c8ba3fca5 P c4a1b28608a6413caaf5980d39b99223: Bootstrap replayed 1/1 log segments. Stats: ops{read=156 overwritten=0 applied=156 ignored=0} inserts{seen=784 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:19.424598 5371 tablet_bootstrap.cc:492] T 8682bff56e604cc3b7e9b07c8ba3fca5 P c4a1b28608a6413caaf5980d39b99223: Bootstrap complete.
I20240503 01:59:19.424875 5371 ts_tablet_manager.cc:1367] T 8682bff56e604cc3b7e9b07c8ba3fca5 P c4a1b28608a6413caaf5980d39b99223: Time spent bootstrapping tablet: real 0.158s user 0.127s sys 0.015s
I20240503 01:59:19.425534 5371 raft_consensus.cc:348] T 8682bff56e604cc3b7e9b07c8ba3fca5 P c4a1b28608a6413caaf5980d39b99223 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 156 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { replace: true } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: true } }
I20240503 01:59:19.425696 5371 raft_consensus.cc:729] T 8682bff56e604cc3b7e9b07c8ba3fca5 P c4a1b28608a6413caaf5980d39b99223 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c4a1b28608a6413caaf5980d39b99223, State: Initialized, Role: LEARNER
I20240503 01:59:19.425812 5371 consensus_queue.cc:260] T 8682bff56e604cc3b7e9b07c8ba3fca5 P c4a1b28608a6413caaf5980d39b99223 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 156, Last appended: 1.156, Last appended by leader: 156, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 156 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { replace: true } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: true } }
I20240503 01:59:19.426262 5371 ts_tablet_manager.cc:1398] T 8682bff56e604cc3b7e9b07c8ba3fca5 P c4a1b28608a6413caaf5980d39b99223: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:19.427023 3075 tablet_copy_service.cc:342] P 6568f69a78c44bb985162d2b7d0a9376: Request end of tablet copy session c4a1b28608a6413caaf5980d39b99223-8682bff56e604cc3b7e9b07c8ba3fca5 received from {username='slave'} at 127.0.99.136:59125
I20240503 01:59:19.427126 3075 tablet_copy_service.cc:433] P 6568f69a78c44bb985162d2b7d0a9376: ending tablet copy session c4a1b28608a6413caaf5980d39b99223-8682bff56e604cc3b7e9b07c8ba3fca5 on tablet 8682bff56e604cc3b7e9b07c8ba3fca5 with peer c4a1b28608a6413caaf5980d39b99223
I20240503 01:59:19.433022 5352 tablet_bootstrap.cc:492] T f40f77238d1e4a34a3f7bea387c6406c P c4a1b28608a6413caaf5980d39b99223: Bootstrap replayed 1/1 log segments. Stats: ops{read=207 overwritten=0 applied=207 ignored=0} inserts{seen=879 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:19.433393 5352 tablet_bootstrap.cc:492] T f40f77238d1e4a34a3f7bea387c6406c P c4a1b28608a6413caaf5980d39b99223: Bootstrap complete.
I20240503 01:59:19.433638 5352 ts_tablet_manager.cc:1367] T f40f77238d1e4a34a3f7bea387c6406c P c4a1b28608a6413caaf5980d39b99223: Time spent bootstrapping tablet: real 0.216s user 0.151s sys 0.012s
I20240503 01:59:19.434141 5352 raft_consensus.cc:348] T f40f77238d1e4a34a3f7bea387c6406c P c4a1b28608a6413caaf5980d39b99223 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } attrs { replace: true } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: true } }
I20240503 01:59:19.434266 5352 raft_consensus.cc:729] T f40f77238d1e4a34a3f7bea387c6406c P c4a1b28608a6413caaf5980d39b99223 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c4a1b28608a6413caaf5980d39b99223, State: Initialized, Role: LEARNER
I20240503 01:59:19.434398 5352 consensus_queue.cc:260] T f40f77238d1e4a34a3f7bea387c6406c P c4a1b28608a6413caaf5980d39b99223 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 207, Last appended: 1.207, Last appended by leader: 207, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } attrs { replace: true } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: true } }
I20240503 01:59:19.434659 5352 ts_tablet_manager.cc:1398] T f40f77238d1e4a34a3f7bea387c6406c P c4a1b28608a6413caaf5980d39b99223: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:19.435209 2420 tablet_copy_service.cc:342] P b63928177fa54227a5ff9929265ab9dd: Request end of tablet copy session c4a1b28608a6413caaf5980d39b99223-f40f77238d1e4a34a3f7bea387c6406c received from {username='slave'} at 127.0.99.136:57695
I20240503 01:59:19.435297 2420 tablet_copy_service.cc:433] P b63928177fa54227a5ff9929265ab9dd: ending tablet copy session c4a1b28608a6413caaf5980d39b99223-f40f77238d1e4a34a3f7bea387c6406c on tablet f40f77238d1e4a34a3f7bea387c6406c with peer c4a1b28608a6413caaf5980d39b99223
I20240503 01:59:19.489930 5321 tablet_bootstrap.cc:492] T c2911cf48f384bf09074f7bea2a4c173 P c4a1b28608a6413caaf5980d39b99223: Bootstrap replayed 1/1 log segments. Stats: ops{read=249 overwritten=0 applied=249 ignored=0} inserts{seen=877 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:19.490335 5321 tablet_bootstrap.cc:492] T c2911cf48f384bf09074f7bea2a4c173 P c4a1b28608a6413caaf5980d39b99223: Bootstrap complete.
I20240503 01:59:19.490550 5321 ts_tablet_manager.cc:1367] T c2911cf48f384bf09074f7bea2a4c173 P c4a1b28608a6413caaf5980d39b99223: Time spent bootstrapping tablet: real 0.249s user 0.166s sys 0.027s
I20240503 01:59:19.491173 5321 raft_consensus.cc:348] T c2911cf48f384bf09074f7bea2a4c173 P c4a1b28608a6413caaf5980d39b99223 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 249 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: true } }
I20240503 01:59:19.491324 5321 raft_consensus.cc:729] T c2911cf48f384bf09074f7bea2a4c173 P c4a1b28608a6413caaf5980d39b99223 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c4a1b28608a6413caaf5980d39b99223, State: Initialized, Role: LEARNER
I20240503 01:59:19.491415 5321 consensus_queue.cc:260] T c2911cf48f384bf09074f7bea2a4c173 P c4a1b28608a6413caaf5980d39b99223 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 249, Last appended: 1.249, Last appended by leader: 249, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 249 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: true } }
I20240503 01:59:19.491696 5321 ts_tablet_manager.cc:1398] T c2911cf48f384bf09074f7bea2a4c173 P c4a1b28608a6413caaf5980d39b99223: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:19.492491 3075 tablet_copy_service.cc:342] P 6568f69a78c44bb985162d2b7d0a9376: Request end of tablet copy session c4a1b28608a6413caaf5980d39b99223-c2911cf48f384bf09074f7bea2a4c173 received from {username='slave'} at 127.0.99.136:59125
I20240503 01:59:19.492599 3075 tablet_copy_service.cc:433] P 6568f69a78c44bb985162d2b7d0a9376: ending tablet copy session c4a1b28608a6413caaf5980d39b99223-c2911cf48f384bf09074f7bea2a4c173 on tablet c2911cf48f384bf09074f7bea2a4c173 with peer c4a1b28608a6413caaf5980d39b99223
I20240503 01:59:19.663568 5060 raft_consensus.cc:1203] T f40f77238d1e4a34a3f7bea387c6406c P c4a1b28608a6413caaf5980d39b99223 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.206->[1.207-1.207] Dedup: 1.207->[]
I20240503 01:59:19.720273 5060 raft_consensus.cc:1203] T 8682bff56e604cc3b7e9b07c8ba3fca5 P c4a1b28608a6413caaf5980d39b99223 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.155->[1.156-1.156] Dedup: 1.156->[]
I20240503 01:59:19.741218 5060 raft_consensus.cc:1203] T 240749b0904f4bb2a4c79b7713473411 P c4a1b28608a6413caaf5980d39b99223 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 01:59:19.747962 5060 raft_consensus.cc:1203] T 0b669a47f1d14ce98ccab24479d39304 P c4a1b28608a6413caaf5980d39b99223 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.220->[1.221-1.221] Dedup: 1.221->[]
I20240503 01:59:19.795403 5060 raft_consensus.cc:1203] T c2911cf48f384bf09074f7bea2a4c173 P c4a1b28608a6413caaf5980d39b99223 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.248->[1.249-1.249] Dedup: 1.249->[]
I20240503 01:59:20.160332 5404 raft_consensus.cc:1050] T f40f77238d1e4a34a3f7bea387c6406c P b63928177fa54227a5ff9929265ab9dd: attempting to promote NON_VOTER c4a1b28608a6413caaf5980d39b99223 to VOTER
I20240503 01:59:20.161023 5404 consensus_queue.cc:237] T f40f77238d1e4a34a3f7bea387c6406c P b63928177fa54227a5ff9929265ab9dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 207, Committed index: 207, Last appended: 1.207, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } attrs { replace: true } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } }
I20240503 01:59:20.162128 5432 raft_consensus.cc:1050] T 0b669a47f1d14ce98ccab24479d39304 P b40a9a7b30d34a6db007ed8febd8a759: attempting to promote NON_VOTER c4a1b28608a6413caaf5980d39b99223 to VOTER
I20240503 01:59:20.162675 5432 consensus_queue.cc:237] T 0b669a47f1d14ce98ccab24479d39304 P b40a9a7b30d34a6db007ed8febd8a759 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 221, Committed index: 221, Last appended: 1.221, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 222 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } attrs { replace: true } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } }
I20240503 01:59:20.163056 2787 raft_consensus.cc:1261] T f40f77238d1e4a34a3f7bea387c6406c P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Refusing update from remote peer b63928177fa54227a5ff9929265ab9dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 207. Preceding OpId from leader: term: 1 index: 208. (index mismatch)
I20240503 01:59:20.163074 2525 raft_consensus.cc:1261] T f40f77238d1e4a34a3f7bea387c6406c P a1a844ecd71145e4a362f4c42500fbcb [term 1 FOLLOWER]: Refusing update from remote peer b63928177fa54227a5ff9929265ab9dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 207. Preceding OpId from leader: term: 1 index: 208. (index mismatch)
I20240503 01:59:20.163107 2924 raft_consensus.cc:1261] T f40f77238d1e4a34a3f7bea387c6406c P 327aa26a8e434a609ebcb382aa665fce [term 1 FOLLOWER]: Refusing update from remote peer b63928177fa54227a5ff9929265ab9dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 207. Preceding OpId from leader: term: 1 index: 208. (index mismatch)
I20240503 01:59:20.163718 5378 consensus_queue.cc:1035] T f40f77238d1e4a34a3f7bea387c6406c P b63928177fa54227a5ff9929265ab9dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 208, Last known committed idx: 207, Time since last communication: 0.000s
I20240503 01:59:20.164001 3054 raft_consensus.cc:1261] T f40f77238d1e4a34a3f7bea387c6406c P 6568f69a78c44bb985162d2b7d0a9376 [term 1 FOLLOWER]: Refusing update from remote peer b63928177fa54227a5ff9929265ab9dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 207. Preceding OpId from leader: term: 1 index: 208. (index mismatch)
I20240503 01:59:20.164176 5378 consensus_queue.cc:1035] T f40f77238d1e4a34a3f7bea387c6406c P b63928177fa54227a5ff9929265ab9dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 208, Last known committed idx: 207, Time since last communication: 0.000s
I20240503 01:59:20.164348 5487 consensus_queue.cc:1035] T f40f77238d1e4a34a3f7bea387c6406c P b63928177fa54227a5ff9929265ab9dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 208, Last known committed idx: 207, Time since last communication: 0.000s
I20240503 01:59:20.164924 2525 raft_consensus.cc:1261] T 0b669a47f1d14ce98ccab24479d39304 P a1a844ecd71145e4a362f4c42500fbcb [term 1 FOLLOWER]: Refusing update from remote peer b40a9a7b30d34a6db007ed8febd8a759: Log matching property violated. Preceding OpId in replica: term: 1 index: 221. Preceding OpId from leader: term: 1 index: 222. (index mismatch)
I20240503 01:59:20.164916 5060 raft_consensus.cc:1261] T f40f77238d1e4a34a3f7bea387c6406c P c4a1b28608a6413caaf5980d39b99223 [term 1 LEARNER]: Refusing update from remote peer b63928177fa54227a5ff9929265ab9dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 207. Preceding OpId from leader: term: 1 index: 208. (index mismatch)
I20240503 01:59:20.165304 3957 raft_consensus.cc:1261] T 0b669a47f1d14ce98ccab24479d39304 P 4125b1a70e864e129bd54bdc2720d7fa [term 1 FOLLOWER]: Refusing update from remote peer b40a9a7b30d34a6db007ed8febd8a759: Log matching property violated. Preceding OpId in replica: term: 1 index: 221. Preceding OpId from leader: term: 1 index: 222. (index mismatch)
I20240503 01:59:20.165796 2787 raft_consensus.cc:1261] T 0b669a47f1d14ce98ccab24479d39304 P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Refusing update from remote peer b40a9a7b30d34a6db007ed8febd8a759: Log matching property violated. Preceding OpId in replica: term: 1 index: 221. Preceding OpId from leader: term: 1 index: 222. (index mismatch)
I20240503 01:59:20.166376 5492 consensus_queue.cc:1035] T 0b669a47f1d14ce98ccab24479d39304 P b40a9a7b30d34a6db007ed8febd8a759 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 222, Last known committed idx: 221, Time since last communication: 0.000s
I20240503 01:59:20.166615 5486 consensus_queue.cc:1035] T f40f77238d1e4a34a3f7bea387c6406c P b63928177fa54227a5ff9929265ab9dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 208, Last known committed idx: 207, Time since last communication: 0.000s
I20240503 01:59:20.167068 3054 raft_consensus.cc:1261] T 0b669a47f1d14ce98ccab24479d39304 P 6568f69a78c44bb985162d2b7d0a9376 [term 1 FOLLOWER]: Refusing update from remote peer b40a9a7b30d34a6db007ed8febd8a759: Log matching property violated. Preceding OpId in replica: term: 1 index: 221. Preceding OpId from leader: term: 1 index: 222. (index mismatch)
I20240503 01:59:20.167111 5208 consensus_queue.cc:1035] T 0b669a47f1d14ce98ccab24479d39304 P b40a9a7b30d34a6db007ed8febd8a759 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 222, Last known committed idx: 221, Time since last communication: 0.000s
I20240503 01:59:20.167469 5208 consensus_queue.cc:1035] T 0b669a47f1d14ce98ccab24479d39304 P b40a9a7b30d34a6db007ed8febd8a759 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 222, Last known committed idx: 221, Time since last communication: 0.000s
I20240503 01:59:20.167990 5208 consensus_queue.cc:1035] T 0b669a47f1d14ce98ccab24479d39304 P b40a9a7b30d34a6db007ed8febd8a759 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 222, Last known committed idx: 221, Time since last communication: 0.000s
I20240503 01:59:20.170202 5486 consensus_queue.cc:1035] T f40f77238d1e4a34a3f7bea387c6406c P b63928177fa54227a5ff9929265ab9dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 208, Last known committed idx: 207, Time since last communication: 0.000s
I20240503 01:59:20.171851 5052 raft_consensus.cc:1261] T 0b669a47f1d14ce98ccab24479d39304 P c4a1b28608a6413caaf5980d39b99223 [term 1 LEARNER]: Refusing update from remote peer b40a9a7b30d34a6db007ed8febd8a759: Log matching property violated. Preceding OpId in replica: term: 1 index: 221. Preceding OpId from leader: term: 1 index: 222. (index mismatch)
I20240503 01:59:20.172281 5208 raft_consensus.cc:2932] T 0b669a47f1d14ce98ccab24479d39304 P b40a9a7b30d34a6db007ed8febd8a759 [term 1 LEADER]: Committing config change with OpId 1.222: config changed from index 221 to 222, c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 222 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } attrs { replace: true } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:20.172812 2787 raft_consensus.cc:2932] T 0b669a47f1d14ce98ccab24479d39304 P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Committing config change with OpId 1.222: config changed from index 221 to 222, c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 222 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } attrs { replace: true } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:20.172923 3957 raft_consensus.cc:2932] T 0b669a47f1d14ce98ccab24479d39304 P 4125b1a70e864e129bd54bdc2720d7fa [term 1 FOLLOWER]: Committing config change with OpId 1.222: config changed from index 221 to 222, c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 222 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } attrs { replace: true } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:20.174825 5404 raft_consensus.cc:2932] T f40f77238d1e4a34a3f7bea387c6406c P b63928177fa54227a5ff9929265ab9dd [term 1 LEADER]: Committing config change with OpId 1.208: config changed from index 207 to 208, c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } attrs { replace: true } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:20.174968 5491 consensus_queue.cc:1035] T 0b669a47f1d14ce98ccab24479d39304 P b40a9a7b30d34a6db007ed8febd8a759 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 222, Last known committed idx: 221, Time since last communication: 0.000s
I20240503 01:59:20.175400 2525 raft_consensus.cc:2932] T 0b669a47f1d14ce98ccab24479d39304 P a1a844ecd71145e4a362f4c42500fbcb [term 1 FOLLOWER]: Committing config change with OpId 1.222: config changed from index 221 to 222, c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 222 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } attrs { replace: true } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:20.177551 2252 catalog_manager.cc:5484] T 0b669a47f1d14ce98ccab24479d39304 P 4125b1a70e864e129bd54bdc2720d7fa reported cstate change: config changed from index 221 to 222, c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b40a9a7b30d34a6db007ed8febd8a759" committed_config { opid_index: 222 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } attrs { replace: true } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:20.177670 3052 raft_consensus.cc:2932] T 0b669a47f1d14ce98ccab24479d39304 P 6568f69a78c44bb985162d2b7d0a9376 [term 1 FOLLOWER]: Committing config change with OpId 1.222: config changed from index 221 to 222, c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 222 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } attrs { replace: true } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:20.178664 2251 catalog_manager.cc:5484] T f40f77238d1e4a34a3f7bea387c6406c P b63928177fa54227a5ff9929265ab9dd reported cstate change: config changed from index 207 to 208, c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b63928177fa54227a5ff9929265ab9dd" committed_config { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:20.178735 3054 raft_consensus.cc:2932] T f40f77238d1e4a34a3f7bea387c6406c P 6568f69a78c44bb985162d2b7d0a9376 [term 1 FOLLOWER]: Committing config change with OpId 1.208: config changed from index 207 to 208, c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } attrs { replace: true } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:20.179217 5052 raft_consensus.cc:2932] T 0b669a47f1d14ce98ccab24479d39304 P c4a1b28608a6413caaf5980d39b99223 [term 1 FOLLOWER]: Committing config change with OpId 1.222: config changed from index 221 to 222, c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 222 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } attrs { replace: true } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:20.181360 2924 raft_consensus.cc:2932] T f40f77238d1e4a34a3f7bea387c6406c P 327aa26a8e434a609ebcb382aa665fce [term 1 FOLLOWER]: Committing config change with OpId 1.208: config changed from index 207 to 208, c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } attrs { replace: true } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:20.183655 2525 raft_consensus.cc:2932] T f40f77238d1e4a34a3f7bea387c6406c P a1a844ecd71145e4a362f4c42500fbcb [term 1 FOLLOWER]: Committing config change with OpId 1.208: config changed from index 207 to 208, c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } attrs { replace: true } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:20.183892 5060 raft_consensus.cc:2932] T f40f77238d1e4a34a3f7bea387c6406c P c4a1b28608a6413caaf5980d39b99223 [term 1 FOLLOWER]: Committing config change with OpId 1.208: config changed from index 207 to 208, c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } attrs { replace: true } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:20.184911 2787 raft_consensus.cc:2932] T f40f77238d1e4a34a3f7bea387c6406c P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Committing config change with OpId 1.208: config changed from index 207 to 208, c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } attrs { replace: true } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:20.192180 2398 consensus_queue.cc:237] T f40f77238d1e4a34a3f7bea387c6406c P b63928177fa54227a5ff9929265ab9dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 208, Committed index: 208, Last appended: 1.208, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 209 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } }
I20240503 01:59:20.193790 5060 raft_consensus.cc:1261] T f40f77238d1e4a34a3f7bea387c6406c P c4a1b28608a6413caaf5980d39b99223 [term 1 FOLLOWER]: Refusing update from remote peer b63928177fa54227a5ff9929265ab9dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 208. Preceding OpId from leader: term: 1 index: 209. (index mismatch)
I20240503 01:59:20.194082 2924 raft_consensus.cc:1261] T f40f77238d1e4a34a3f7bea387c6406c P 327aa26a8e434a609ebcb382aa665fce [term 1 FOLLOWER]: Refusing update from remote peer b63928177fa54227a5ff9929265ab9dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 208. Preceding OpId from leader: term: 1 index: 209. (index mismatch)
I20240503 01:59:20.194525 5486 consensus_queue.cc:1035] T f40f77238d1e4a34a3f7bea387c6406c P b63928177fa54227a5ff9929265ab9dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 209, Last known committed idx: 208, Time since last communication: 0.000s
I20240503 01:59:20.194779 3054 raft_consensus.cc:1261] T f40f77238d1e4a34a3f7bea387c6406c P 6568f69a78c44bb985162d2b7d0a9376 [term 1 FOLLOWER]: Refusing update from remote peer b63928177fa54227a5ff9929265ab9dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 208. Preceding OpId from leader: term: 1 index: 209. (index mismatch)
I20240503 01:59:20.194893 5486 consensus_queue.cc:1035] T f40f77238d1e4a34a3f7bea387c6406c P b63928177fa54227a5ff9929265ab9dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 209, Last known committed idx: 208, Time since last communication: 0.000s
I20240503 01:59:20.195641 5486 consensus_queue.cc:1035] T f40f77238d1e4a34a3f7bea387c6406c P b63928177fa54227a5ff9929265ab9dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 209, Last known committed idx: 208, Time since last communication: 0.000s
I20240503 01:59:20.195909 2525 raft_consensus.cc:1261] T f40f77238d1e4a34a3f7bea387c6406c P a1a844ecd71145e4a362f4c42500fbcb [term 1 FOLLOWER]: Refusing update from remote peer b63928177fa54227a5ff9929265ab9dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 208. Preceding OpId from leader: term: 1 index: 209. (index mismatch)
I20240503 01:59:20.196475 5265 consensus_queue.cc:1035] T f40f77238d1e4a34a3f7bea387c6406c P b63928177fa54227a5ff9929265ab9dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 209, Last known committed idx: 208, Time since last communication: 0.000s
I20240503 01:59:20.197338 5265 raft_consensus.cc:2932] T f40f77238d1e4a34a3f7bea387c6406c P b63928177fa54227a5ff9929265ab9dd [term 1 LEADER]: Committing config change with OpId 1.209: config changed from index 208 to 209, VOTER 52c196c15e3348b985c689652bc49918 (127.0.99.132) evicted. New config: { opid_index: 209 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:20.197721 2924 raft_consensus.cc:2932] T f40f77238d1e4a34a3f7bea387c6406c P 327aa26a8e434a609ebcb382aa665fce [term 1 FOLLOWER]: Committing config change with OpId 1.209: config changed from index 208 to 209, VOTER 52c196c15e3348b985c689652bc49918 (127.0.99.132) evicted. New config: { opid_index: 209 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:20.198079 3054 raft_consensus.cc:2932] T f40f77238d1e4a34a3f7bea387c6406c P 6568f69a78c44bb985162d2b7d0a9376 [term 1 FOLLOWER]: Committing config change with OpId 1.209: config changed from index 208 to 209, VOTER 52c196c15e3348b985c689652bc49918 (127.0.99.132) evicted. New config: { opid_index: 209 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:20.198691 2525 raft_consensus.cc:2932] T f40f77238d1e4a34a3f7bea387c6406c P a1a844ecd71145e4a362f4c42500fbcb [term 1 FOLLOWER]: Committing config change with OpId 1.209: config changed from index 208 to 209, VOTER 52c196c15e3348b985c689652bc49918 (127.0.99.132) evicted. New config: { opid_index: 209 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:20.200062 5060 raft_consensus.cc:2932] T f40f77238d1e4a34a3f7bea387c6406c P c4a1b28608a6413caaf5980d39b99223 [term 1 FOLLOWER]: Committing config change with OpId 1.209: config changed from index 208 to 209, VOTER 52c196c15e3348b985c689652bc49918 (127.0.99.132) evicted. New config: { opid_index: 209 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:20.200331 2245 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f40f77238d1e4a34a3f7bea387c6406c with cas_config_opid_index 208: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:20.200450 2257 catalog_manager.cc:5484] T f40f77238d1e4a34a3f7bea387c6406c P b63928177fa54227a5ff9929265ab9dd reported cstate change: config changed from index 208 to 209, VOTER 52c196c15e3348b985c689652bc49918 (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "b63928177fa54227a5ff9929265ab9dd" committed_config { opid_index: 209 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:20.204231 2772 tablet_service.cc:1514] Processing DeleteTablet for tablet f40f77238d1e4a34a3f7bea387c6406c with delete_type TABLET_DATA_TOMBSTONED (TS 52c196c15e3348b985c689652bc49918 not found in new config with opid_index 209) from {username='slave'} at 127.0.0.1:45406
I20240503 01:59:20.204800 5510 tablet_replica.cc:330] T f40f77238d1e4a34a3f7bea387c6406c P 52c196c15e3348b985c689652bc49918: stopping tablet replica
I20240503 01:59:20.205137 5510 raft_consensus.cc:2229] T f40f77238d1e4a34a3f7bea387c6406c P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:20.205350 5510 raft_consensus.cc:2258] T f40f77238d1e4a34a3f7bea387c6406c P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:20.206985 5510 ts_tablet_manager.cc:1875] T f40f77238d1e4a34a3f7bea387c6406c P 52c196c15e3348b985c689652bc49918: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:20.209249 5510 ts_tablet_manager.cc:1888] T f40f77238d1e4a34a3f7bea387c6406c P 52c196c15e3348b985c689652bc49918: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.208
I20240503 01:59:20.209334 5510 log.cc:1196] T f40f77238d1e4a34a3f7bea387c6406c P 52c196c15e3348b985c689652bc49918: Deleting WAL directory at /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/f40f77238d1e4a34a3f7bea387c6406c
I20240503 01:59:20.209934 2245 catalog_manager.cc:4830] TS 52c196c15e3348b985c689652bc49918 (127.0.99.132:42887): tablet f40f77238d1e4a34a3f7bea387c6406c (table rebalance_test_table_4 [id=e8f9ec3f55d044f8b23964df20e79f65]) successfully deleted
I20240503 01:59:20.299705 5432 raft_consensus.cc:1050] T 240749b0904f4bb2a4c79b7713473411 P b40a9a7b30d34a6db007ed8febd8a759: attempting to promote NON_VOTER c4a1b28608a6413caaf5980d39b99223 to VOTER
I20240503 01:59:20.300300 5432 consensus_queue.cc:237] T 240749b0904f4bb2a4c79b7713473411 P b40a9a7b30d34a6db007ed8febd8a759 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } }
I20240503 01:59:20.301645 5433 raft_consensus.cc:1050] T c2911cf48f384bf09074f7bea2a4c173 P 6568f69a78c44bb985162d2b7d0a9376: attempting to promote NON_VOTER c4a1b28608a6413caaf5980d39b99223 to VOTER
I20240503 01:59:20.302268 5433 consensus_queue.cc:237] T c2911cf48f384bf09074f7bea2a4c173 P 6568f69a78c44bb985162d2b7d0a9376 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 249, Committed index: 249, Last appended: 1.249, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 250 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } }
I20240503 01:59:20.302335 3957 raft_consensus.cc:1261] T 240749b0904f4bb2a4c79b7713473411 P 4125b1a70e864e129bd54bdc2720d7fa [term 1 FOLLOWER]: Refusing update from remote peer b40a9a7b30d34a6db007ed8febd8a759: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:20.302564 2924 raft_consensus.cc:1261] T 240749b0904f4bb2a4c79b7713473411 P 327aa26a8e434a609ebcb382aa665fce [term 1 FOLLOWER]: Refusing update from remote peer b40a9a7b30d34a6db007ed8febd8a759: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:20.303092 3054 raft_consensus.cc:1261] T 240749b0904f4bb2a4c79b7713473411 P 6568f69a78c44bb985162d2b7d0a9376 [term 1 FOLLOWER]: Refusing update from remote peer b40a9a7b30d34a6db007ed8febd8a759: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:20.303309 5431 consensus_queue.cc:1035] T 240749b0904f4bb2a4c79b7713473411 P b40a9a7b30d34a6db007ed8febd8a759 [LEADER]: Connected to new peer: Peer: permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:20.303382 2787 raft_consensus.cc:1261] T 240749b0904f4bb2a4c79b7713473411 P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Refusing update from remote peer b40a9a7b30d34a6db007ed8febd8a759: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:20.303774 5208 consensus_queue.cc:1035] T 240749b0904f4bb2a4c79b7713473411 P b40a9a7b30d34a6db007ed8febd8a759 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:20.303977 5492 consensus_queue.cc:1035] T 240749b0904f4bb2a4c79b7713473411 P b40a9a7b30d34a6db007ed8febd8a759 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:20.304052 5060 raft_consensus.cc:1261] T 240749b0904f4bb2a4c79b7713473411 P c4a1b28608a6413caaf5980d39b99223 [term 1 LEARNER]: Refusing update from remote peer b40a9a7b30d34a6db007ed8febd8a759: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:20.304160 5490 consensus_queue.cc:1035] T 240749b0904f4bb2a4c79b7713473411 P b40a9a7b30d34a6db007ed8febd8a759 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:20.304543 5432 consensus_queue.cc:1035] T 240749b0904f4bb2a4c79b7713473411 P b40a9a7b30d34a6db007ed8febd8a759 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:20.305212 3957 raft_consensus.cc:1261] T c2911cf48f384bf09074f7bea2a4c173 P 4125b1a70e864e129bd54bdc2720d7fa [term 1 FOLLOWER]: Refusing update from remote peer 6568f69a78c44bb985162d2b7d0a9376: Log matching property violated. Preceding OpId in replica: term: 1 index: 249. Preceding OpId from leader: term: 1 index: 250. (index mismatch)
I20240503 01:59:20.305707 5242 consensus_queue.cc:1035] T c2911cf48f384bf09074f7bea2a4c173 P 6568f69a78c44bb985162d2b7d0a9376 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 250, Last known committed idx: 249, Time since last communication: 0.000s
I20240503 01:59:20.307726 2920 raft_consensus.cc:1261] T c2911cf48f384bf09074f7bea2a4c173 P 327aa26a8e434a609ebcb382aa665fce [term 1 FOLLOWER]: Refusing update from remote peer 6568f69a78c44bb985162d2b7d0a9376: Log matching property violated. Preceding OpId in replica: term: 1 index: 249. Preceding OpId from leader: term: 1 index: 250. (index mismatch)
I20240503 01:59:20.308022 5432 raft_consensus.cc:2932] T 240749b0904f4bb2a4c79b7713473411 P b40a9a7b30d34a6db007ed8febd8a759 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:20.308192 5433 consensus_queue.cc:1035] T c2911cf48f384bf09074f7bea2a4c173 P 6568f69a78c44bb985162d2b7d0a9376 [LEADER]: Connected to new peer: Peer: permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 250, Last known committed idx: 249, Time since last communication: 0.000s
I20240503 01:59:20.308877 2660 raft_consensus.cc:1261] T c2911cf48f384bf09074f7bea2a4c173 P b40a9a7b30d34a6db007ed8febd8a759 [term 1 FOLLOWER]: Refusing update from remote peer 6568f69a78c44bb985162d2b7d0a9376: Log matching property violated. Preceding OpId in replica: term: 1 index: 249. Preceding OpId from leader: term: 1 index: 250. (index mismatch)
I20240503 01:59:20.308686 3954 raft_consensus.cc:2932] T 240749b0904f4bb2a4c79b7713473411 P 4125b1a70e864e129bd54bdc2720d7fa [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:20.308969 2398 raft_consensus.cc:1261] T c2911cf48f384bf09074f7bea2a4c173 P b63928177fa54227a5ff9929265ab9dd [term 1 FOLLOWER]: Refusing update from remote peer 6568f69a78c44bb985162d2b7d0a9376: Log matching property violated. Preceding OpId in replica: term: 1 index: 249. Preceding OpId from leader: term: 1 index: 250. (index mismatch)
I20240503 01:59:20.309289 5242 consensus_queue.cc:1035] T c2911cf48f384bf09074f7bea2a4c173 P 6568f69a78c44bb985162d2b7d0a9376 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 250, Last known committed idx: 249, Time since last communication: 0.000s
I20240503 01:59:20.309922 5242 consensus_queue.cc:1035] T c2911cf48f384bf09074f7bea2a4c173 P 6568f69a78c44bb985162d2b7d0a9376 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 250, Last known committed idx: 249, Time since last communication: 0.000s
I20240503 01:59:20.312033 2787 raft_consensus.cc:2932] T 240749b0904f4bb2a4c79b7713473411 P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:20.312623 2253 catalog_manager.cc:5484] T 240749b0904f4bb2a4c79b7713473411 P b40a9a7b30d34a6db007ed8febd8a759 reported cstate change: config changed from index 5 to 6, c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b40a9a7b30d34a6db007ed8febd8a759" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:20.312984 5060 raft_consensus.cc:1261] T c2911cf48f384bf09074f7bea2a4c173 P c4a1b28608a6413caaf5980d39b99223 [term 1 LEARNER]: Refusing update from remote peer 6568f69a78c44bb985162d2b7d0a9376: Log matching property violated. Preceding OpId in replica: term: 1 index: 249. Preceding OpId from leader: term: 1 index: 250. (index mismatch)
I20240503 01:59:20.312836 2920 raft_consensus.cc:2932] T 240749b0904f4bb2a4c79b7713473411 P 327aa26a8e434a609ebcb382aa665fce [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:20.313383 5242 consensus_queue.cc:1035] T c2911cf48f384bf09074f7bea2a4c173 P 6568f69a78c44bb985162d2b7d0a9376 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 250, Last known committed idx: 249, Time since last communication: 0.000s
I20240503 01:59:20.315538 5242 raft_consensus.cc:2932] T c2911cf48f384bf09074f7bea2a4c173 P 6568f69a78c44bb985162d2b7d0a9376 [term 1 LEADER]: Committing config change with OpId 1.250: config changed from index 249 to 250, c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 250 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:20.318051 2398 raft_consensus.cc:2932] T c2911cf48f384bf09074f7bea2a4c173 P b63928177fa54227a5ff9929265ab9dd [term 1 FOLLOWER]: Committing config change with OpId 1.250: config changed from index 249 to 250, c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 250 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:20.318389 3954 raft_consensus.cc:2932] T c2911cf48f384bf09074f7bea2a4c173 P 4125b1a70e864e129bd54bdc2720d7fa [term 1 FOLLOWER]: Committing config change with OpId 1.250: config changed from index 249 to 250, c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 250 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:20.318545 5067 raft_consensus.cc:2932] T c2911cf48f384bf09074f7bea2a4c173 P c4a1b28608a6413caaf5980d39b99223 [term 1 FOLLOWER]: Committing config change with OpId 1.250: config changed from index 249 to 250, c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 250 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:20.320904 2250 catalog_manager.cc:5484] T c2911cf48f384bf09074f7bea2a4c173 P b63928177fa54227a5ff9929265ab9dd reported cstate change: config changed from index 249 to 250, c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6568f69a78c44bb985162d2b7d0a9376" committed_config { opid_index: 250 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:20.321491 5052 raft_consensus.cc:2932] T 240749b0904f4bb2a4c79b7713473411 P c4a1b28608a6413caaf5980d39b99223 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:20.322870 5433 raft_consensus.cc:1050] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 6568f69a78c44bb985162d2b7d0a9376: attempting to promote NON_VOTER c4a1b28608a6413caaf5980d39b99223 to VOTER
I20240503 01:59:20.323403 5433 consensus_queue.cc:237] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 6568f69a78c44bb985162d2b7d0a9376 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 156, Committed index: 156, Last appended: 1.156, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 157 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { replace: true } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } }
I20240503 01:59:20.323762 3054 raft_consensus.cc:2932] T 240749b0904f4bb2a4c79b7713473411 P 6568f69a78c44bb985162d2b7d0a9376 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:20.325525 2661 consensus_queue.cc:237] T 240749b0904f4bb2a4c79b7713473411 P b40a9a7b30d34a6db007ed8febd8a759 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } }
I20240503 01:59:20.327595 5052 raft_consensus.cc:1261] T 240749b0904f4bb2a4c79b7713473411 P c4a1b28608a6413caaf5980d39b99223 [term 1 FOLLOWER]: Refusing update from remote peer b40a9a7b30d34a6db007ed8febd8a759: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:20.327939 2924 raft_consensus.cc:1261] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 327aa26a8e434a609ebcb382aa665fce [term 1 FOLLOWER]: Refusing update from remote peer 6568f69a78c44bb985162d2b7d0a9376: Log matching property violated. Preceding OpId in replica: term: 1 index: 156. Preceding OpId from leader: term: 1 index: 157. (index mismatch)
I20240503 01:59:20.328088 5490 consensus_queue.cc:1035] T 240749b0904f4bb2a4c79b7713473411 P b40a9a7b30d34a6db007ed8febd8a759 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:20.328176 5052 raft_consensus.cc:1261] T 8682bff56e604cc3b7e9b07c8ba3fca5 P c4a1b28608a6413caaf5980d39b99223 [term 1 LEARNER]: Refusing update from remote peer 6568f69a78c44bb985162d2b7d0a9376: Log matching property violated. Preceding OpId in replica: term: 1 index: 156. Preceding OpId from leader: term: 1 index: 157. (index mismatch)
I20240503 01:59:20.328791 3054 raft_consensus.cc:1261] T 240749b0904f4bb2a4c79b7713473411 P 6568f69a78c44bb985162d2b7d0a9376 [term 1 FOLLOWER]: Refusing update from remote peer b40a9a7b30d34a6db007ed8febd8a759: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:20.328904 2787 raft_consensus.cc:1261] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Refusing update from remote peer 6568f69a78c44bb985162d2b7d0a9376: Log matching property violated. Preceding OpId in replica: term: 1 index: 156. Preceding OpId from leader: term: 1 index: 157. (index mismatch)
I20240503 01:59:20.329138 5242 consensus_queue.cc:1035] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 6568f69a78c44bb985162d2b7d0a9376 [LEADER]: Connected to new peer: Peer: permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 157, Last known committed idx: 156, Time since last communication: 0.000s
I20240503 01:59:20.329349 5208 consensus_queue.cc:1035] T 240749b0904f4bb2a4c79b7713473411 P b40a9a7b30d34a6db007ed8febd8a759 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:20.329540 5242 consensus_queue.cc:1035] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 6568f69a78c44bb985162d2b7d0a9376 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 157, Last known committed idx: 156, Time since last communication: 0.000s
I20240503 01:59:20.329737 5525 consensus_queue.cc:1035] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 6568f69a78c44bb985162d2b7d0a9376 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 157, Last known committed idx: 156, Time since last communication: 0.000s
I20240503 01:59:20.331041 2661 raft_consensus.cc:1261] T 8682bff56e604cc3b7e9b07c8ba3fca5 P b40a9a7b30d34a6db007ed8febd8a759 [term 1 FOLLOWER]: Refusing update from remote peer 6568f69a78c44bb985162d2b7d0a9376: Log matching property violated. Preceding OpId in replica: term: 1 index: 156. Preceding OpId from leader: term: 1 index: 157. (index mismatch)
I20240503 01:59:20.331583 5530 consensus_queue.cc:1035] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 6568f69a78c44bb985162d2b7d0a9376 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 157, Last known committed idx: 156, Time since last communication: 0.000s
I20240503 01:59:20.333109 5525 raft_consensus.cc:2932] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 6568f69a78c44bb985162d2b7d0a9376 [term 1 LEADER]: Committing config change with OpId 1.157: config changed from index 156 to 157, c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 157 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { replace: true } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:20.333304 2920 raft_consensus.cc:1261] T 240749b0904f4bb2a4c79b7713473411 P 327aa26a8e434a609ebcb382aa665fce [term 1 FOLLOWER]: Refusing update from remote peer b40a9a7b30d34a6db007ed8febd8a759: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:20.333302 5490 raft_consensus.cc:2932] T 240749b0904f4bb2a4c79b7713473411 P b40a9a7b30d34a6db007ed8febd8a759 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:20.333832 5491 consensus_queue.cc:1035] T 240749b0904f4bb2a4c79b7713473411 P b40a9a7b30d34a6db007ed8febd8a759 [LEADER]: Connected to new peer: Peer: permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:20.335143 2787 raft_consensus.cc:1261] T 240749b0904f4bb2a4c79b7713473411 P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Refusing update from remote peer b40a9a7b30d34a6db007ed8febd8a759: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:20.335155 2924 raft_consensus.cc:2932] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 327aa26a8e434a609ebcb382aa665fce [term 1 FOLLOWER]: Committing config change with OpId 1.157: config changed from index 156 to 157, c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 157 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { replace: true } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:20.335409 2242 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 240749b0904f4bb2a4c79b7713473411 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:20.335700 5052 raft_consensus.cc:2932] T 240749b0904f4bb2a4c79b7713473411 P c4a1b28608a6413caaf5980d39b99223 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:20.335930 5208 consensus_queue.cc:1035] T 240749b0904f4bb2a4c79b7713473411 P b40a9a7b30d34a6db007ed8febd8a759 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:20.336028 3054 raft_consensus.cc:2932] T 240749b0904f4bb2a4c79b7713473411 P 6568f69a78c44bb985162d2b7d0a9376 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:20.336839 2789 raft_consensus.cc:2932] T 240749b0904f4bb2a4c79b7713473411 P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:20.337813 5060 raft_consensus.cc:2932] T 8682bff56e604cc3b7e9b07c8ba3fca5 P c4a1b28608a6413caaf5980d39b99223 [term 1 FOLLOWER]: Committing config change with OpId 1.157: config changed from index 156 to 157, c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 157 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { replace: true } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:20.338945 2257 catalog_manager.cc:5484] T 240749b0904f4bb2a4c79b7713473411 P c4a1b28608a6413caaf5980d39b99223 reported cstate change: config changed from index 6 to 7, VOTER 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) evicted. New cstate: current_term: 1 leader_uuid: "b40a9a7b30d34a6db007ed8febd8a759" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:20.339958 2920 raft_consensus.cc:2932] T 240749b0904f4bb2a4c79b7713473411 P 327aa26a8e434a609ebcb382aa665fce [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:20.340322 2253 catalog_manager.cc:5484] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 327aa26a8e434a609ebcb382aa665fce reported cstate change: config changed from index 156 to 157, c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6568f69a78c44bb985162d2b7d0a9376" committed_config { opid_index: 157 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { replace: true } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:20.341770 2661 raft_consensus.cc:2932] T 8682bff56e604cc3b7e9b07c8ba3fca5 P b40a9a7b30d34a6db007ed8febd8a759 [term 1 FOLLOWER]: Committing config change with OpId 1.157: config changed from index 156 to 157, c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 157 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { replace: true } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:20.344229 2787 raft_consensus.cc:2932] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Committing config change with OpId 1.157: config changed from index 156 to 157, c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 157 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { replace: true } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:20.345153 3954 raft_consensus.cc:1261] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 4125b1a70e864e129bd54bdc2720d7fa [term 1 FOLLOWER]: Refusing update from remote peer 6568f69a78c44bb985162d2b7d0a9376: Log matching property violated. Preceding OpId in replica: term: 1 index: 156. Preceding OpId from leader: term: 1 index: 157. (index mismatch)
I20240503 01:59:20.346120 5528 consensus_queue.cc:1035] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 6568f69a78c44bb985162d2b7d0a9376 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 157, Last known committed idx: 156, Time since last communication: 0.000s
I20240503 01:59:20.346797 2920 raft_consensus.cc:2932] T c2911cf48f384bf09074f7bea2a4c173 P 327aa26a8e434a609ebcb382aa665fce [term 1 FOLLOWER]: Committing config change with OpId 1.250: config changed from index 249 to 250, c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 250 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:20.347975 3054 consensus_queue.cc:237] T c2911cf48f384bf09074f7bea2a4c173 P 6568f69a78c44bb985162d2b7d0a9376 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 250, Committed index: 250, Last appended: 1.250, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 251 OBSOLETE_local: false peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } }
I20240503 01:59:20.348104 3954 raft_consensus.cc:2932] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 4125b1a70e864e129bd54bdc2720d7fa [term 1 FOLLOWER]: Committing config change with OpId 1.157: config changed from index 156 to 157, c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 157 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { replace: true } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:20.349423 5060 raft_consensus.cc:1261] T c2911cf48f384bf09074f7bea2a4c173 P c4a1b28608a6413caaf5980d39b99223 [term 1 FOLLOWER]: Refusing update from remote peer 6568f69a78c44bb985162d2b7d0a9376: Log matching property violated. Preceding OpId in replica: term: 1 index: 250. Preceding OpId from leader: term: 1 index: 251. (index mismatch)
I20240503 01:59:20.349787 3957 raft_consensus.cc:1261] T c2911cf48f384bf09074f7bea2a4c173 P 4125b1a70e864e129bd54bdc2720d7fa [term 1 FOLLOWER]: Refusing update from remote peer 6568f69a78c44bb985162d2b7d0a9376: Log matching property violated. Preceding OpId in replica: term: 1 index: 250. Preceding OpId from leader: term: 1 index: 251. (index mismatch)
I20240503 01:59:20.349880 2660 raft_consensus.cc:1261] T c2911cf48f384bf09074f7bea2a4c173 P b40a9a7b30d34a6db007ed8febd8a759 [term 1 FOLLOWER]: Refusing update from remote peer 6568f69a78c44bb985162d2b7d0a9376: Log matching property violated. Preceding OpId in replica: term: 1 index: 250. Preceding OpId from leader: term: 1 index: 251. (index mismatch)
I20240503 01:59:20.350423 2661 raft_consensus.cc:2932] T c2911cf48f384bf09074f7bea2a4c173 P b40a9a7b30d34a6db007ed8febd8a759 [term 1 FOLLOWER]: Committing config change with OpId 1.250: config changed from index 249 to 250, c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 250 OBSOLETE_local: false peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:20.350785 2924 raft_consensus.cc:1261] T c2911cf48f384bf09074f7bea2a4c173 P 327aa26a8e434a609ebcb382aa665fce [term 1 FOLLOWER]: Refusing update from remote peer 6568f69a78c44bb985162d2b7d0a9376: Log matching property violated. Preceding OpId in replica: term: 1 index: 250. Preceding OpId from leader: term: 1 index: 251. (index mismatch)
I20240503 01:59:20.350783 5519 consensus_queue.cc:1035] T c2911cf48f384bf09074f7bea2a4c173 P 6568f69a78c44bb985162d2b7d0a9376 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 251, Last known committed idx: 249, Time since last communication: 0.000s
I20240503 01:59:20.351296 5520 consensus_queue.cc:1035] T c2911cf48f384bf09074f7bea2a4c173 P 6568f69a78c44bb985162d2b7d0a9376 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 251, Last known committed idx: 250, Time since last communication: 0.000s
I20240503 01:59:20.351622 5520 consensus_queue.cc:1035] T c2911cf48f384bf09074f7bea2a4c173 P 6568f69a78c44bb985162d2b7d0a9376 [LEADER]: Connected to new peer: Peer: permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 251, Last known committed idx: 250, Time since last communication: 0.000s
I20240503 01:59:20.352880 5520 consensus_queue.cc:1035] T c2911cf48f384bf09074f7bea2a4c173 P 6568f69a78c44bb985162d2b7d0a9376 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 251, Last known committed idx: 250, Time since last communication: 0.000s
I20240503 01:59:20.354771 5242 raft_consensus.cc:2932] T c2911cf48f384bf09074f7bea2a4c173 P 6568f69a78c44bb985162d2b7d0a9376 [term 1 LEADER]: Committing config change with OpId 1.251: config changed from index 250 to 251, VOTER b63928177fa54227a5ff9929265ab9dd (127.0.99.129) evicted. New config: { opid_index: 251 OBSOLETE_local: false peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:20.355305 5060 raft_consensus.cc:2932] T c2911cf48f384bf09074f7bea2a4c173 P c4a1b28608a6413caaf5980d39b99223 [term 1 FOLLOWER]: Committing config change with OpId 1.251: config changed from index 250 to 251, VOTER b63928177fa54227a5ff9929265ab9dd (127.0.99.129) evicted. New config: { opid_index: 251 OBSOLETE_local: false peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:20.356810 2661 raft_consensus.cc:2932] T c2911cf48f384bf09074f7bea2a4c173 P b40a9a7b30d34a6db007ed8febd8a759 [term 1 FOLLOWER]: Committing config change with OpId 1.251: config changed from index 250 to 251, VOTER b63928177fa54227a5ff9929265ab9dd (127.0.99.129) evicted. New config: { opid_index: 251 OBSOLETE_local: false peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:20.357470 2924 raft_consensus.cc:2932] T c2911cf48f384bf09074f7bea2a4c173 P 327aa26a8e434a609ebcb382aa665fce [term 1 FOLLOWER]: Committing config change with OpId 1.251: config changed from index 250 to 251, VOTER b63928177fa54227a5ff9929265ab9dd (127.0.99.129) evicted. New config: { opid_index: 251 OBSOLETE_local: false peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:20.357750 2243 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c2911cf48f384bf09074f7bea2a4c173 with cas_config_opid_index 250: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:20.358709 3957 raft_consensus.cc:2932] T c2911cf48f384bf09074f7bea2a4c173 P 4125b1a70e864e129bd54bdc2720d7fa [term 1 FOLLOWER]: Committing config change with OpId 1.251: config changed from index 250 to 251, VOTER b63928177fa54227a5ff9929265ab9dd (127.0.99.129) evicted. New config: { opid_index: 251 OBSOLETE_local: false peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:20.359246 3938 tablet_service.cc:1514] Processing DeleteTablet for tablet 240749b0904f4bb2a4c79b7713473411 with delete_type TABLET_DATA_TOMBSTONED (TS 4125b1a70e864e129bd54bdc2720d7fa not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:55716
I20240503 01:59:20.360165 2256 catalog_manager.cc:5484] T c2911cf48f384bf09074f7bea2a4c173 P c4a1b28608a6413caaf5980d39b99223 reported cstate change: config changed from index 250 to 251, VOTER b63928177fa54227a5ff9929265ab9dd (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "6568f69a78c44bb985162d2b7d0a9376" committed_config { opid_index: 251 OBSOLETE_local: false peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:20.362490 5538 tablet_replica.cc:330] stopping tablet replica
I20240503 01:59:20.363066 2378 tablet_service.cc:1514] Processing DeleteTablet for tablet c2911cf48f384bf09074f7bea2a4c173 with delete_type TABLET_DATA_TOMBSTONED (TS b63928177fa54227a5ff9929265ab9dd not found in new config with opid_index 251) from {username='slave'} at 127.0.0.1:44104
I20240503 01:59:20.364598 5543 tablet_replica.cc:330] T c2911cf48f384bf09074f7bea2a4c173 P b63928177fa54227a5ff9929265ab9dd: stopping tablet replica
I20240503 01:59:20.364795 5543 raft_consensus.cc:2229] T c2911cf48f384bf09074f7bea2a4c173 P b63928177fa54227a5ff9929265ab9dd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:20.365028 5538 raft_consensus.cc:2229] T 240749b0904f4bb2a4c79b7713473411 P 4125b1a70e864e129bd54bdc2720d7fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:20.365073 5543 raft_consensus.cc:2258] T c2911cf48f384bf09074f7bea2a4c173 P b63928177fa54227a5ff9929265ab9dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:20.365461 5538 raft_consensus.cc:2258] T 240749b0904f4bb2a4c79b7713473411 P 4125b1a70e864e129bd54bdc2720d7fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:20.367002 5543 ts_tablet_manager.cc:1875] T c2911cf48f384bf09074f7bea2a4c173 P b63928177fa54227a5ff9929265ab9dd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:20.367496 5538 ts_tablet_manager.cc:1875] T 240749b0904f4bb2a4c79b7713473411 P 4125b1a70e864e129bd54bdc2720d7fa: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:20.369451 5543 ts_tablet_manager.cc:1888] T c2911cf48f384bf09074f7bea2a4c173 P b63928177fa54227a5ff9929265ab9dd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.250
I20240503 01:59:20.369614 5543 log.cc:1196] T c2911cf48f384bf09074f7bea2a4c173 P b63928177fa54227a5ff9929265ab9dd: Deleting WAL directory at /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/c2911cf48f384bf09074f7bea2a4c173
I20240503 01:59:20.370122 5538 ts_tablet_manager.cc:1888] T 240749b0904f4bb2a4c79b7713473411 P 4125b1a70e864e129bd54bdc2720d7fa: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 01:59:20.370354 5538 log.cc:1196] T 240749b0904f4bb2a4c79b7713473411 P 4125b1a70e864e129bd54bdc2720d7fa: Deleting WAL directory at /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/240749b0904f4bb2a4c79b7713473411
I20240503 01:59:20.370338 2245 catalog_manager.cc:4830] TS b63928177fa54227a5ff9929265ab9dd (127.0.99.129:36425): tablet c2911cf48f384bf09074f7bea2a4c173 (table rebalance_test_table_2 [id=18b5c04f80684606bc736d3ca6780b16]) successfully deleted
I20240503 01:59:20.370988 2245 catalog_manager.cc:4830] TS 4125b1a70e864e129bd54bdc2720d7fa (127.0.99.135:39501): tablet 240749b0904f4bb2a4c79b7713473411 (table TestTable [id=e423c6476d1b43779e0a839bf55a520d]) successfully deleted
I20240503 01:59:20.400709 3054 consensus_queue.cc:237] T 8f4c3f70f2714cc299bc0d595f838c85 P 6568f69a78c44bb985162d2b7d0a9376 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 217, Committed index: 217, Last appended: 1.217, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 218 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: true } }
I20240503 01:59:20.402560 2920 raft_consensus.cc:1261] T 8f4c3f70f2714cc299bc0d595f838c85 P 327aa26a8e434a609ebcb382aa665fce [term 1 FOLLOWER]: Refusing update from remote peer 6568f69a78c44bb985162d2b7d0a9376: Log matching property violated. Preceding OpId in replica: term: 1 index: 217. Preceding OpId from leader: term: 1 index: 218. (index mismatch)
I20240503 01:59:20.402628 2787 raft_consensus.cc:1261] T 8f4c3f70f2714cc299bc0d595f838c85 P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Refusing update from remote peer 6568f69a78c44bb985162d2b7d0a9376: Log matching property violated. Preceding OpId in replica: term: 1 index: 217. Preceding OpId from leader: term: 1 index: 218. (index mismatch)
I20240503 01:59:20.402755 2398 raft_consensus.cc:1261] T 8f4c3f70f2714cc299bc0d595f838c85 P b63928177fa54227a5ff9929265ab9dd [term 1 FOLLOWER]: Refusing update from remote peer 6568f69a78c44bb985162d2b7d0a9376: Log matching property violated. Preceding OpId in replica: term: 1 index: 217. Preceding OpId from leader: term: 1 index: 218. (index mismatch)
W20240503 01:59:20.402878 2988 consensus_peers.cc:485] T 8f4c3f70f2714cc299bc0d595f838c85 P 6568f69a78c44bb985162d2b7d0a9376 -> Peer c4a1b28608a6413caaf5980d39b99223 (127.0.99.136:33113): Couldn't send request to peer c4a1b28608a6413caaf5980d39b99223. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8f4c3f70f2714cc299bc0d595f838c85. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:20.403080 5242 consensus_queue.cc:1035] T 8f4c3f70f2714cc299bc0d595f838c85 P 6568f69a78c44bb985162d2b7d0a9376 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 218, Last known committed idx: 217, Time since last communication: 0.000s
I20240503 01:59:20.403465 5242 consensus_queue.cc:1035] T 8f4c3f70f2714cc299bc0d595f838c85 P 6568f69a78c44bb985162d2b7d0a9376 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 218, Last known committed idx: 217, Time since last communication: 0.000s
I20240503 01:59:20.403573 2525 raft_consensus.cc:1261] T 8f4c3f70f2714cc299bc0d595f838c85 P a1a844ecd71145e4a362f4c42500fbcb [term 1 FOLLOWER]: Refusing update from remote peer 6568f69a78c44bb985162d2b7d0a9376: Log matching property violated. Preceding OpId in replica: term: 1 index: 217. Preceding OpId from leader: term: 1 index: 218. (index mismatch)
I20240503 01:59:20.403676 5527 consensus_queue.cc:1035] T 8f4c3f70f2714cc299bc0d595f838c85 P 6568f69a78c44bb985162d2b7d0a9376 [LEADER]: Connected to new peer: Peer: permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 218, Last known committed idx: 217, Time since last communication: 0.000s
I20240503 01:59:20.404153 5527 consensus_queue.cc:1035] T 8f4c3f70f2714cc299bc0d595f838c85 P 6568f69a78c44bb985162d2b7d0a9376 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 218, Last known committed idx: 217, Time since last communication: 0.000s
I20240503 01:59:20.405778 5530 raft_consensus.cc:2932] T 8f4c3f70f2714cc299bc0d595f838c85 P 6568f69a78c44bb985162d2b7d0a9376 [term 1 LEADER]: Committing config change with OpId 1.218: config changed from index -1 to 218, NON_VOTER c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) added. New config: { opid_index: 218 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: true } } }
I20240503 01:59:20.406284 2920 raft_consensus.cc:2932] T 8f4c3f70f2714cc299bc0d595f838c85 P 327aa26a8e434a609ebcb382aa665fce [term 1 FOLLOWER]: Committing config change with OpId 1.218: config changed from index -1 to 218, NON_VOTER c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) added. New config: { opid_index: 218 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: true } } }
I20240503 01:59:20.406877 2525 raft_consensus.cc:2932] T 8f4c3f70f2714cc299bc0d595f838c85 P a1a844ecd71145e4a362f4c42500fbcb [term 1 FOLLOWER]: Committing config change with OpId 1.218: config changed from index -1 to 218, NON_VOTER c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) added. New config: { opid_index: 218 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: true } } }
I20240503 01:59:20.406913 2787 raft_consensus.cc:2932] T 8f4c3f70f2714cc299bc0d595f838c85 P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Committing config change with OpId 1.218: config changed from index -1 to 218, NON_VOTER c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) added. New config: { opid_index: 218 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: true } } }
I20240503 01:59:20.409579 2253 catalog_manager.cc:5484] T 8f4c3f70f2714cc299bc0d595f838c85 P 6568f69a78c44bb985162d2b7d0a9376 reported cstate change: config changed from index -1 to 218, NON_VOTER c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "6568f69a78c44bb985162d2b7d0a9376" committed_config { opid_index: 218 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:20.412065 2398 raft_consensus.cc:2932] T 8f4c3f70f2714cc299bc0d595f838c85 P b63928177fa54227a5ff9929265ab9dd [term 1 FOLLOWER]: Committing config change with OpId 1.218: config changed from index -1 to 218, NON_VOTER c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) added. New config: { opid_index: 218 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: true } } }
I20240503 01:59:20.421797 2920 consensus_queue.cc:237] T 10ecacdfb9a44d068b0df24e991612ee P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 148, Committed index: 148, Last appended: 1.148, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 149 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: true } }
I20240503 01:59:20.424151 3954 raft_consensus.cc:1261] T 10ecacdfb9a44d068b0df24e991612ee P 4125b1a70e864e129bd54bdc2720d7fa [term 1 FOLLOWER]: Refusing update from remote peer 327aa26a8e434a609ebcb382aa665fce: Log matching property violated. Preceding OpId in replica: term: 1 index: 148. Preceding OpId from leader: term: 1 index: 149. (index mismatch)
I20240503 01:59:20.424321 2525 raft_consensus.cc:1261] T 10ecacdfb9a44d068b0df24e991612ee P a1a844ecd71145e4a362f4c42500fbcb [term 1 FOLLOWER]: Refusing update from remote peer 327aa26a8e434a609ebcb382aa665fce: Log matching property violated. Preceding OpId in replica: term: 1 index: 148. Preceding OpId from leader: term: 1 index: 149. (index mismatch)
I20240503 01:59:20.424450 2398 raft_consensus.cc:1261] T 10ecacdfb9a44d068b0df24e991612ee P b63928177fa54227a5ff9929265ab9dd [term 1 FOLLOWER]: Refusing update from remote peer 327aa26a8e434a609ebcb382aa665fce: Log matching property violated. Preceding OpId in replica: term: 1 index: 148. Preceding OpId from leader: term: 1 index: 149. (index mismatch)
I20240503 01:59:20.424515 2661 raft_consensus.cc:1261] T 10ecacdfb9a44d068b0df24e991612ee P b40a9a7b30d34a6db007ed8febd8a759 [term 1 FOLLOWER]: Refusing update from remote peer 327aa26a8e434a609ebcb382aa665fce: Log matching property violated. Preceding OpId in replica: term: 1 index: 148. Preceding OpId from leader: term: 1 index: 149. (index mismatch)
I20240503 01:59:20.424973 5583 consensus_queue.cc:1035] T 10ecacdfb9a44d068b0df24e991612ee P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 149, Last known committed idx: 148, Time since last communication: 0.000s
I20240503 01:59:20.425503 5583 consensus_queue.cc:1035] T 10ecacdfb9a44d068b0df24e991612ee P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 149, Last known committed idx: 148, Time since last communication: 0.000s
I20240503 01:59:20.425733 5459 consensus_queue.cc:1035] T 10ecacdfb9a44d068b0df24e991612ee P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 149, Last known committed idx: 148, Time since last communication: 0.000s
I20240503 01:59:20.425920 5582 consensus_queue.cc:1035] T 10ecacdfb9a44d068b0df24e991612ee P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 149, Last known committed idx: 148, Time since last communication: 0.000s
I20240503 01:59:20.428289 5583 raft_consensus.cc:2932] T 10ecacdfb9a44d068b0df24e991612ee P 327aa26a8e434a609ebcb382aa665fce [term 1 LEADER]: Committing config change with OpId 1.149: config changed from index 66 to 149, NON_VOTER c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) added. New config: { opid_index: 149 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: true } } }
I20240503 01:59:20.429302 2398 raft_consensus.cc:2932] T 10ecacdfb9a44d068b0df24e991612ee P b63928177fa54227a5ff9929265ab9dd [term 1 FOLLOWER]: Committing config change with OpId 1.149: config changed from index 66 to 149, NON_VOTER c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) added. New config: { opid_index: 149 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: true } } }
W20240503 01:59:20.429903 2857 consensus_peers.cc:485] T 10ecacdfb9a44d068b0df24e991612ee P 327aa26a8e434a609ebcb382aa665fce -> Peer c4a1b28608a6413caaf5980d39b99223 (127.0.99.136:33113): Couldn't send request to peer c4a1b28608a6413caaf5980d39b99223. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 10ecacdfb9a44d068b0df24e991612ee. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:20.431145 2661 raft_consensus.cc:2932] T 10ecacdfb9a44d068b0df24e991612ee P b40a9a7b30d34a6db007ed8febd8a759 [term 1 FOLLOWER]: Committing config change with OpId 1.149: config changed from index 66 to 149, NON_VOTER c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) added. New config: { opid_index: 149 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: true } } }
I20240503 01:59:20.432184 2249 catalog_manager.cc:5484] T 10ecacdfb9a44d068b0df24e991612ee P b63928177fa54227a5ff9929265ab9dd reported cstate change: config changed from index 66 to 149, NON_VOTER c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "327aa26a8e434a609ebcb382aa665fce" committed_config { opid_index: 149 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: true } } }
I20240503 01:59:20.433338 3954 raft_consensus.cc:2932] T 10ecacdfb9a44d068b0df24e991612ee P 4125b1a70e864e129bd54bdc2720d7fa [term 1 FOLLOWER]: Committing config change with OpId 1.149: config changed from index 66 to 149, NON_VOTER c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) added. New config: { opid_index: 149 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: true } } }
I20240503 01:59:20.434162 2525 raft_consensus.cc:2932] T 10ecacdfb9a44d068b0df24e991612ee P a1a844ecd71145e4a362f4c42500fbcb [term 1 FOLLOWER]: Committing config change with OpId 1.149: config changed from index 66 to 149, NON_VOTER c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) added. New config: { opid_index: 149 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: true } } }
I20240503 01:59:20.444538 2525 consensus_queue.cc:237] T 43008abe74604666b330d78ec0f51c7c P a1a844ecd71145e4a362f4c42500fbcb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 278, Committed index: 278, Last appended: 1.278, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 279 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { replace: true } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: true } }
I20240503 01:59:20.447263 3954 raft_consensus.cc:1261] T 43008abe74604666b330d78ec0f51c7c P 4125b1a70e864e129bd54bdc2720d7fa [term 1 FOLLOWER]: Refusing update from remote peer a1a844ecd71145e4a362f4c42500fbcb: Log matching property violated. Preceding OpId in replica: term: 1 index: 278. Preceding OpId from leader: term: 1 index: 279. (index mismatch)
I20240503 01:59:20.447537 3054 raft_consensus.cc:1261] T 43008abe74604666b330d78ec0f51c7c P 6568f69a78c44bb985162d2b7d0a9376 [term 1 FOLLOWER]: Refusing update from remote peer a1a844ecd71145e4a362f4c42500fbcb: Log matching property violated. Preceding OpId in replica: term: 1 index: 278. Preceding OpId from leader: term: 1 index: 279. (index mismatch)
I20240503 01:59:20.447705 3624 consensus_queue.cc:1035] T 43008abe74604666b330d78ec0f51c7c P a1a844ecd71145e4a362f4c42500fbcb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 279, Last known committed idx: 278, Time since last communication: 0.000s
I20240503 01:59:20.448272 3624 consensus_queue.cc:1035] T 43008abe74604666b330d78ec0f51c7c P a1a844ecd71145e4a362f4c42500fbcb [LEADER]: Connected to new peer: Peer: permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 279, Last known committed idx: 278, Time since last communication: 0.000s
I20240503 01:59:20.449424 2661 raft_consensus.cc:1261] T 43008abe74604666b330d78ec0f51c7c P b40a9a7b30d34a6db007ed8febd8a759 [term 1 FOLLOWER]: Refusing update from remote peer a1a844ecd71145e4a362f4c42500fbcb: Log matching property violated. Preceding OpId in replica: term: 1 index: 278. Preceding OpId from leader: term: 1 index: 279. (index mismatch)
I20240503 01:59:20.449630 2398 raft_consensus.cc:1261] T 43008abe74604666b330d78ec0f51c7c P b63928177fa54227a5ff9929265ab9dd [term 1 FOLLOWER]: Refusing update from remote peer a1a844ecd71145e4a362f4c42500fbcb: Log matching property violated. Preceding OpId in replica: term: 1 index: 278. Preceding OpId from leader: term: 1 index: 279. (index mismatch)
I20240503 01:59:20.450240 3624 consensus_queue.cc:1035] T 43008abe74604666b330d78ec0f51c7c P a1a844ecd71145e4a362f4c42500fbcb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 279, Last known committed idx: 278, Time since last communication: 0.000s
I20240503 01:59:20.450457 5599 consensus_queue.cc:1035] T 43008abe74604666b330d78ec0f51c7c P a1a844ecd71145e4a362f4c42500fbcb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 279, Last known committed idx: 278, Time since last communication: 0.000s
I20240503 01:59:20.454465 5599 raft_consensus.cc:2932] T 43008abe74604666b330d78ec0f51c7c P a1a844ecd71145e4a362f4c42500fbcb [term 1 LEADER]: Committing config change with OpId 1.279: config changed from index 159 to 279, NON_VOTER c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) added. New config: { opid_index: 279 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { replace: true } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: true } } }
W20240503 01:59:20.455659 2464 consensus_peers.cc:485] T 43008abe74604666b330d78ec0f51c7c P a1a844ecd71145e4a362f4c42500fbcb -> Peer c4a1b28608a6413caaf5980d39b99223 (127.0.99.136:33113): Couldn't send request to peer c4a1b28608a6413caaf5980d39b99223. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 43008abe74604666b330d78ec0f51c7c. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:20.455802 2398 raft_consensus.cc:2932] T 43008abe74604666b330d78ec0f51c7c P b63928177fa54227a5ff9929265ab9dd [term 1 FOLLOWER]: Committing config change with OpId 1.279: config changed from index 159 to 279, NON_VOTER c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) added. New config: { opid_index: 279 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { replace: true } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: true } } }
I20240503 01:59:20.456378 3954 raft_consensus.cc:2932] T 43008abe74604666b330d78ec0f51c7c P 4125b1a70e864e129bd54bdc2720d7fa [term 1 FOLLOWER]: Committing config change with OpId 1.279: config changed from index 159 to 279, NON_VOTER c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) added. New config: { opid_index: 279 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { replace: true } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: true } } }
I20240503 01:59:20.457955 3054 raft_consensus.cc:2932] T 43008abe74604666b330d78ec0f51c7c P 6568f69a78c44bb985162d2b7d0a9376 [term 1 FOLLOWER]: Committing config change with OpId 1.279: config changed from index 159 to 279, NON_VOTER c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) added. New config: { opid_index: 279 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { replace: true } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: true } } }
I20240503 01:59:20.458282 2661 raft_consensus.cc:2932] T 43008abe74604666b330d78ec0f51c7c P b40a9a7b30d34a6db007ed8febd8a759 [term 1 FOLLOWER]: Committing config change with OpId 1.279: config changed from index 159 to 279, NON_VOTER c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) added. New config: { opid_index: 279 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { replace: true } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: true } } }
I20240503 01:59:20.459509 2257 catalog_manager.cc:5484] T 43008abe74604666b330d78ec0f51c7c P 4125b1a70e864e129bd54bdc2720d7fa reported cstate change: config changed from index 159 to 279, NON_VOTER c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "a1a844ecd71145e4a362f4c42500fbcb" committed_config { opid_index: 279 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { replace: true } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: true } } }
I20240503 01:59:20.671301 2661 consensus_queue.cc:237] T 0b669a47f1d14ce98ccab24479d39304 P b40a9a7b30d34a6db007ed8febd8a759 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 222, Committed index: 222, Last appended: 1.222, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 223 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } }
I20240503 01:59:20.672763 5060 raft_consensus.cc:1261] T 0b669a47f1d14ce98ccab24479d39304 P c4a1b28608a6413caaf5980d39b99223 [term 1 FOLLOWER]: Refusing update from remote peer b40a9a7b30d34a6db007ed8febd8a759: Log matching property violated. Preceding OpId in replica: term: 1 index: 222. Preceding OpId from leader: term: 1 index: 223. (index mismatch)
I20240503 01:59:20.672864 3954 raft_consensus.cc:1261] T 0b669a47f1d14ce98ccab24479d39304 P 4125b1a70e864e129bd54bdc2720d7fa [term 1 FOLLOWER]: Refusing update from remote peer b40a9a7b30d34a6db007ed8febd8a759: Log matching property violated. Preceding OpId in replica: term: 1 index: 222. Preceding OpId from leader: term: 1 index: 223. (index mismatch)
I20240503 01:59:20.672988 3054 raft_consensus.cc:1261] T 0b669a47f1d14ce98ccab24479d39304 P 6568f69a78c44bb985162d2b7d0a9376 [term 1 FOLLOWER]: Refusing update from remote peer b40a9a7b30d34a6db007ed8febd8a759: Log matching property violated. Preceding OpId in replica: term: 1 index: 222. Preceding OpId from leader: term: 1 index: 223. (index mismatch)
I20240503 01:59:20.673244 5492 consensus_queue.cc:1035] T 0b669a47f1d14ce98ccab24479d39304 P b40a9a7b30d34a6db007ed8febd8a759 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 223, Last known committed idx: 222, Time since last communication: 0.000s
I20240503 01:59:20.673621 5492 consensus_queue.cc:1035] T 0b669a47f1d14ce98ccab24479d39304 P b40a9a7b30d34a6db007ed8febd8a759 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 223, Last known committed idx: 222, Time since last communication: 0.000s
I20240503 01:59:20.674355 2787 raft_consensus.cc:1261] T 0b669a47f1d14ce98ccab24479d39304 P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Refusing update from remote peer b40a9a7b30d34a6db007ed8febd8a759: Log matching property violated. Preceding OpId in replica: term: 1 index: 222. Preceding OpId from leader: term: 1 index: 223. (index mismatch)
I20240503 01:59:20.675050 5432 consensus_queue.cc:1035] T 0b669a47f1d14ce98ccab24479d39304 P b40a9a7b30d34a6db007ed8febd8a759 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 223, Last known committed idx: 222, Time since last communication: 0.000s
I20240503 01:59:20.675312 5492 consensus_queue.cc:1035] T 0b669a47f1d14ce98ccab24479d39304 P b40a9a7b30d34a6db007ed8febd8a759 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 223, Last known committed idx: 222, Time since last communication: 0.000s
I20240503 01:59:20.675630 5208 raft_consensus.cc:2932] T 0b669a47f1d14ce98ccab24479d39304 P b40a9a7b30d34a6db007ed8febd8a759 [term 1 LEADER]: Committing config change with OpId 1.223: config changed from index 222 to 223, VOTER a1a844ecd71145e4a362f4c42500fbcb (127.0.99.130) evicted. New config: { opid_index: 223 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:20.677654 3054 raft_consensus.cc:2932] T 0b669a47f1d14ce98ccab24479d39304 P 6568f69a78c44bb985162d2b7d0a9376 [term 1 FOLLOWER]: Committing config change with OpId 1.223: config changed from index 222 to 223, VOTER a1a844ecd71145e4a362f4c42500fbcb (127.0.99.130) evicted. New config: { opid_index: 223 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:20.678279 2787 raft_consensus.cc:2932] T 0b669a47f1d14ce98ccab24479d39304 P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Committing config change with OpId 1.223: config changed from index 222 to 223, VOTER a1a844ecd71145e4a362f4c42500fbcb (127.0.99.130) evicted. New config: { opid_index: 223 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:20.678501 2242 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0b669a47f1d14ce98ccab24479d39304 with cas_config_opid_index 222: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:20.678360 3954 raft_consensus.cc:2932] T 0b669a47f1d14ce98ccab24479d39304 P 4125b1a70e864e129bd54bdc2720d7fa [term 1 FOLLOWER]: Committing config change with OpId 1.223: config changed from index 222 to 223, VOTER a1a844ecd71145e4a362f4c42500fbcb (127.0.99.130) evicted. New config: { opid_index: 223 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:20.681102 2249 catalog_manager.cc:5484] T 0b669a47f1d14ce98ccab24479d39304 P 52c196c15e3348b985c689652bc49918 reported cstate change: config changed from index 222 to 223, VOTER a1a844ecd71145e4a362f4c42500fbcb (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "b40a9a7b30d34a6db007ed8febd8a759" committed_config { opid_index: 223 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:20.681268 5060 raft_consensus.cc:2932] T 0b669a47f1d14ce98ccab24479d39304 P c4a1b28608a6413caaf5980d39b99223 [term 1 FOLLOWER]: Committing config change with OpId 1.223: config changed from index 222 to 223, VOTER a1a844ecd71145e4a362f4c42500fbcb (127.0.99.130) evicted. New config: { opid_index: 223 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:20.684017 2510 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b669a47f1d14ce98ccab24479d39304 with delete_type TABLET_DATA_TOMBSTONED (TS a1a844ecd71145e4a362f4c42500fbcb not found in new config with opid_index 223) from {username='slave'} at 127.0.0.1:59126
I20240503 01:59:20.684578 5631 tablet_replica.cc:330] T 0b669a47f1d14ce98ccab24479d39304 P a1a844ecd71145e4a362f4c42500fbcb: stopping tablet replica
I20240503 01:59:20.684815 5631 raft_consensus.cc:2229] T 0b669a47f1d14ce98ccab24479d39304 P a1a844ecd71145e4a362f4c42500fbcb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:20.685075 5631 raft_consensus.cc:2258] T 0b669a47f1d14ce98ccab24479d39304 P a1a844ecd71145e4a362f4c42500fbcb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:20.686813 5631 ts_tablet_manager.cc:1875] T 0b669a47f1d14ce98ccab24479d39304 P a1a844ecd71145e4a362f4c42500fbcb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:20.688755 5631 ts_tablet_manager.cc:1888] T 0b669a47f1d14ce98ccab24479d39304 P a1a844ecd71145e4a362f4c42500fbcb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.222
I20240503 01:59:20.688823 5631 log.cc:1196] T 0b669a47f1d14ce98ccab24479d39304 P a1a844ecd71145e4a362f4c42500fbcb: Deleting WAL directory at /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/0b669a47f1d14ce98ccab24479d39304
I20240503 01:59:20.689373 2245 catalog_manager.cc:4830] TS a1a844ecd71145e4a362f4c42500fbcb (127.0.99.130:46033): tablet 0b669a47f1d14ce98ccab24479d39304 (table rebalance_test_table_4 [id=e8f9ec3f55d044f8b23964df20e79f65]) successfully deleted
I20240503 01:59:20.732074 2398 consensus_queue.cc:237] T c10343409a2a423eb0b21759bfc9ba61 P b63928177fa54227a5ff9929265ab9dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 179, Committed index: 179, Last appended: 1.179, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 180 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } attrs { replace: true } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: true } }
I20240503 01:59:20.734181 2661 raft_consensus.cc:1261] T c10343409a2a423eb0b21759bfc9ba61 P b40a9a7b30d34a6db007ed8febd8a759 [term 1 FOLLOWER]: Refusing update from remote peer b63928177fa54227a5ff9929265ab9dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 179. Preceding OpId from leader: term: 1 index: 180. (index mismatch)
I20240503 01:59:20.734382 3954 raft_consensus.cc:1261] T c10343409a2a423eb0b21759bfc9ba61 P 4125b1a70e864e129bd54bdc2720d7fa [term 1 FOLLOWER]: Refusing update from remote peer b63928177fa54227a5ff9929265ab9dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 179. Preceding OpId from leader: term: 1 index: 180. (index mismatch)
I20240503 01:59:20.734447 2523 raft_consensus.cc:1261] T c10343409a2a423eb0b21759bfc9ba61 P a1a844ecd71145e4a362f4c42500fbcb [term 1 FOLLOWER]: Refusing update from remote peer b63928177fa54227a5ff9929265ab9dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 179. Preceding OpId from leader: term: 1 index: 180. (index mismatch)
I20240503 01:59:20.734445 2787 raft_consensus.cc:1261] T c10343409a2a423eb0b21759bfc9ba61 P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Refusing update from remote peer b63928177fa54227a5ff9929265ab9dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 179. Preceding OpId from leader: term: 1 index: 180. (index mismatch)
W20240503 01:59:20.734717 2333 consensus_peers.cc:485] T c10343409a2a423eb0b21759bfc9ba61 P b63928177fa54227a5ff9929265ab9dd -> Peer c4a1b28608a6413caaf5980d39b99223 (127.0.99.136:33113): Couldn't send request to peer c4a1b28608a6413caaf5980d39b99223. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c10343409a2a423eb0b21759bfc9ba61. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:20.734934 5665 consensus_queue.cc:1035] T c10343409a2a423eb0b21759bfc9ba61 P b63928177fa54227a5ff9929265ab9dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 180, Last known committed idx: 179, Time since last communication: 0.000s
I20240503 01:59:20.735321 5664 consensus_queue.cc:1035] T c10343409a2a423eb0b21759bfc9ba61 P b63928177fa54227a5ff9929265ab9dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 180, Last known committed idx: 179, Time since last communication: 0.000s
I20240503 01:59:20.735491 5378 consensus_queue.cc:1035] T c10343409a2a423eb0b21759bfc9ba61 P b63928177fa54227a5ff9929265ab9dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 180, Last known committed idx: 179, Time since last communication: 0.000s
I20240503 01:59:20.735710 5663 consensus_queue.cc:1035] T c10343409a2a423eb0b21759bfc9ba61 P b63928177fa54227a5ff9929265ab9dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 180, Last known committed idx: 179, Time since last communication: 0.000s
I20240503 01:59:20.738144 5665 raft_consensus.cc:2932] T c10343409a2a423eb0b21759bfc9ba61 P b63928177fa54227a5ff9929265ab9dd [term 1 LEADER]: Committing config change with OpId 1.180: config changed from index 57 to 180, NON_VOTER c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) added. New config: { opid_index: 180 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } attrs { replace: true } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: true } } }
I20240503 01:59:20.738899 2787 raft_consensus.cc:2932] T c10343409a2a423eb0b21759bfc9ba61 P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Committing config change with OpId 1.180: config changed from index 57 to 180, NON_VOTER c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) added. New config: { opid_index: 180 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } attrs { replace: true } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: true } } }
I20240503 01:59:20.739153 2523 raft_consensus.cc:2932] T c10343409a2a423eb0b21759bfc9ba61 P a1a844ecd71145e4a362f4c42500fbcb [term 1 FOLLOWER]: Committing config change with OpId 1.180: config changed from index 57 to 180, NON_VOTER c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) added. New config: { opid_index: 180 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } attrs { replace: true } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: true } } }
I20240503 01:59:20.740029 3954 raft_consensus.cc:2932] T c10343409a2a423eb0b21759bfc9ba61 P 4125b1a70e864e129bd54bdc2720d7fa [term 1 FOLLOWER]: Committing config change with OpId 1.180: config changed from index 57 to 180, NON_VOTER c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) added. New config: { opid_index: 180 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } attrs { replace: true } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: true } } }
I20240503 01:59:20.741310 2661 raft_consensus.cc:2932] T c10343409a2a423eb0b21759bfc9ba61 P b40a9a7b30d34a6db007ed8febd8a759 [term 1 FOLLOWER]: Committing config change with OpId 1.180: config changed from index 57 to 180, NON_VOTER c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) added. New config: { opid_index: 180 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } attrs { replace: true } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: true } } }
I20240503 01:59:20.742470 2249 catalog_manager.cc:5484] T c10343409a2a423eb0b21759bfc9ba61 P b63928177fa54227a5ff9929265ab9dd reported cstate change: config changed from index 57 to 180, NON_VOTER c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "b63928177fa54227a5ff9929265ab9dd" committed_config { opid_index: 180 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:20.784308 5695 ts_tablet_manager.cc:902] T 8f4c3f70f2714cc299bc0d595f838c85 P c4a1b28608a6413caaf5980d39b99223: Initiating tablet copy from peer 6568f69a78c44bb985162d2b7d0a9376 (127.0.99.134:43603)
I20240503 01:59:20.785069 5695 tablet_copy_client.cc:308] T 8f4c3f70f2714cc299bc0d595f838c85 P c4a1b28608a6413caaf5980d39b99223: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.134:43603
I20240503 01:59:20.785614 3075 tablet_copy_service.cc:140] P 6568f69a78c44bb985162d2b7d0a9376: Received BeginTabletCopySession request for tablet 8f4c3f70f2714cc299bc0d595f838c85 from peer c4a1b28608a6413caaf5980d39b99223 ({username='slave'} at 127.0.99.136:59125)
I20240503 01:59:20.785729 3075 tablet_copy_service.cc:161] P 6568f69a78c44bb985162d2b7d0a9376: Beginning new tablet copy session on tablet 8f4c3f70f2714cc299bc0d595f838c85 from peer c4a1b28608a6413caaf5980d39b99223 at {username='slave'} at 127.0.99.136:59125: session id = c4a1b28608a6413caaf5980d39b99223-8f4c3f70f2714cc299bc0d595f838c85
I20240503 01:59:20.786921 3075 tablet_copy_source_session.cc:205] T 8f4c3f70f2714cc299bc0d595f838c85 P 6568f69a78c44bb985162d2b7d0a9376: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:20.787889 5695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f4c3f70f2714cc299bc0d595f838c85. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:20.789937 5695 tablet_copy_client.cc:783] T 8f4c3f70f2714cc299bc0d595f838c85 P c4a1b28608a6413caaf5980d39b99223: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:20.790117 5695 tablet_copy_client.cc:647] T 8f4c3f70f2714cc299bc0d595f838c85 P c4a1b28608a6413caaf5980d39b99223: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:20.791870 5695 tablet_copy_client.cc:523] T 8f4c3f70f2714cc299bc0d595f838c85 P c4a1b28608a6413caaf5980d39b99223: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:20.792966 5695 tablet_bootstrap.cc:492] T 8f4c3f70f2714cc299bc0d595f838c85 P c4a1b28608a6413caaf5980d39b99223: Bootstrap starting.
I20240503 01:59:20.822139 5701 ts_tablet_manager.cc:902] T 10ecacdfb9a44d068b0df24e991612ee P c4a1b28608a6413caaf5980d39b99223: Initiating tablet copy from peer 327aa26a8e434a609ebcb382aa665fce (127.0.99.133:38853)
I20240503 01:59:20.822739 5701 tablet_copy_client.cc:308] T 10ecacdfb9a44d068b0df24e991612ee P c4a1b28608a6413caaf5980d39b99223: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.133:38853
I20240503 01:59:20.826545 2944 tablet_copy_service.cc:140] P 327aa26a8e434a609ebcb382aa665fce: Received BeginTabletCopySession request for tablet 10ecacdfb9a44d068b0df24e991612ee from peer c4a1b28608a6413caaf5980d39b99223 ({username='slave'} at 127.0.99.136:60173)
I20240503 01:59:20.826705 2944 tablet_copy_service.cc:161] P 327aa26a8e434a609ebcb382aa665fce: Beginning new tablet copy session on tablet 10ecacdfb9a44d068b0df24e991612ee from peer c4a1b28608a6413caaf5980d39b99223 at {username='slave'} at 127.0.99.136:60173: session id = c4a1b28608a6413caaf5980d39b99223-10ecacdfb9a44d068b0df24e991612ee
I20240503 01:59:20.828011 2944 tablet_copy_source_session.cc:205] T 10ecacdfb9a44d068b0df24e991612ee P 327aa26a8e434a609ebcb382aa665fce: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:20.829200 5701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10ecacdfb9a44d068b0df24e991612ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:20.831898 5701 tablet_copy_client.cc:783] T 10ecacdfb9a44d068b0df24e991612ee P c4a1b28608a6413caaf5980d39b99223: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:20.832068 5701 tablet_copy_client.cc:647] T 10ecacdfb9a44d068b0df24e991612ee P c4a1b28608a6413caaf5980d39b99223: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:20.833865 5701 tablet_copy_client.cc:523] T 10ecacdfb9a44d068b0df24e991612ee P c4a1b28608a6413caaf5980d39b99223: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:20.835160 5701 tablet_bootstrap.cc:492] T 10ecacdfb9a44d068b0df24e991612ee P c4a1b28608a6413caaf5980d39b99223: Bootstrap starting.
I20240503 01:59:20.876157 3054 consensus_queue.cc:237] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 6568f69a78c44bb985162d2b7d0a9376 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 157, Committed index: 157, Last appended: 1.157, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 158 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } }
I20240503 01:59:20.877898 2787 raft_consensus.cc:1261] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Refusing update from remote peer 6568f69a78c44bb985162d2b7d0a9376: Log matching property violated. Preceding OpId in replica: term: 1 index: 157. Preceding OpId from leader: term: 1 index: 158. (index mismatch)
I20240503 01:59:20.877878 3954 raft_consensus.cc:1261] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 4125b1a70e864e129bd54bdc2720d7fa [term 1 FOLLOWER]: Refusing update from remote peer 6568f69a78c44bb985162d2b7d0a9376: Log matching property violated. Preceding OpId in replica: term: 1 index: 157. Preceding OpId from leader: term: 1 index: 158. (index mismatch)
I20240503 01:59:20.878651 5060 raft_consensus.cc:1261] T 8682bff56e604cc3b7e9b07c8ba3fca5 P c4a1b28608a6413caaf5980d39b99223 [term 1 FOLLOWER]: Refusing update from remote peer 6568f69a78c44bb985162d2b7d0a9376: Log matching property violated. Preceding OpId in replica: term: 1 index: 157. Preceding OpId from leader: term: 1 index: 158. (index mismatch)
I20240503 01:59:20.878654 2661 raft_consensus.cc:1261] T 8682bff56e604cc3b7e9b07c8ba3fca5 P b40a9a7b30d34a6db007ed8febd8a759 [term 1 FOLLOWER]: Refusing update from remote peer 6568f69a78c44bb985162d2b7d0a9376: Log matching property violated. Preceding OpId in replica: term: 1 index: 157. Preceding OpId from leader: term: 1 index: 158. (index mismatch)
I20240503 01:59:20.879066 5242 consensus_queue.cc:1035] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 6568f69a78c44bb985162d2b7d0a9376 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 158, Last known committed idx: 157, Time since last communication: 0.000s
I20240503 01:59:20.879427 5433 consensus_queue.cc:1035] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 6568f69a78c44bb985162d2b7d0a9376 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 158, Last known committed idx: 157, Time since last communication: 0.000s
I20240503 01:59:20.879544 5527 consensus_queue.cc:1035] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 6568f69a78c44bb985162d2b7d0a9376 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 158, Last known committed idx: 157, Time since last communication: 0.000s
I20240503 01:59:20.879654 5242 consensus_queue.cc:1035] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 6568f69a78c44bb985162d2b7d0a9376 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 158, Last known committed idx: 157, Time since last communication: 0.000s
I20240503 01:59:20.882850 5433 raft_consensus.cc:2932] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 6568f69a78c44bb985162d2b7d0a9376 [term 1 LEADER]: Committing config change with OpId 1.158: config changed from index 157 to 158, VOTER 327aa26a8e434a609ebcb382aa665fce (127.0.99.133) evicted. New config: { opid_index: 158 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:20.884841 2243 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8682bff56e604cc3b7e9b07c8ba3fca5 with cas_config_opid_index 157: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:20.885309 2787 raft_consensus.cc:2932] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Committing config change with OpId 1.158: config changed from index 157 to 158, VOTER 327aa26a8e434a609ebcb382aa665fce (127.0.99.133) evicted. New config: { opid_index: 158 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:20.885521 2250 catalog_manager.cc:5484] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 6568f69a78c44bb985162d2b7d0a9376 reported cstate change: config changed from index 157 to 158, VOTER 327aa26a8e434a609ebcb382aa665fce (127.0.99.133) evicted. New cstate: current_term: 1 leader_uuid: "6568f69a78c44bb985162d2b7d0a9376" committed_config { opid_index: 158 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:20.888909 5060 raft_consensus.cc:2932] T 8682bff56e604cc3b7e9b07c8ba3fca5 P c4a1b28608a6413caaf5980d39b99223 [term 1 FOLLOWER]: Committing config change with OpId 1.158: config changed from index 157 to 158, VOTER 327aa26a8e434a609ebcb382aa665fce (127.0.99.133) evicted. New config: { opid_index: 158 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:20.890583 2661 raft_consensus.cc:2932] T 8682bff56e604cc3b7e9b07c8ba3fca5 P b40a9a7b30d34a6db007ed8febd8a759 [term 1 FOLLOWER]: Committing config change with OpId 1.158: config changed from index 157 to 158, VOTER 327aa26a8e434a609ebcb382aa665fce (127.0.99.133) evicted. New config: { opid_index: 158 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:20.892233 3954 raft_consensus.cc:2932] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 4125b1a70e864e129bd54bdc2720d7fa [term 1 FOLLOWER]: Committing config change with OpId 1.158: config changed from index 157 to 158, VOTER 327aa26a8e434a609ebcb382aa665fce (127.0.99.133) evicted. New config: { opid_index: 158 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:20.895141 2904 tablet_service.cc:1514] Processing DeleteTablet for tablet 8682bff56e604cc3b7e9b07c8ba3fca5 with delete_type TABLET_DATA_TOMBSTONED (TS 327aa26a8e434a609ebcb382aa665fce not found in new config with opid_index 158) from {username='slave'} at 127.0.0.1:60782
I20240503 01:59:20.897388 5708 tablet_replica.cc:330] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 327aa26a8e434a609ebcb382aa665fce: stopping tablet replica
I20240503 01:59:20.897655 5708 raft_consensus.cc:2229] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 327aa26a8e434a609ebcb382aa665fce [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:20.897925 5708 raft_consensus.cc:2258] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 327aa26a8e434a609ebcb382aa665fce [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:20.899518 5708 ts_tablet_manager.cc:1875] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 327aa26a8e434a609ebcb382aa665fce: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:20.901933 5708 ts_tablet_manager.cc:1888] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 327aa26a8e434a609ebcb382aa665fce: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.157
I20240503 01:59:20.902033 5708 log.cc:1196] T 8682bff56e604cc3b7e9b07c8ba3fca5 P 327aa26a8e434a609ebcb382aa665fce: Deleting WAL directory at /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/8682bff56e604cc3b7e9b07c8ba3fca5
I20240503 01:59:20.904940 2243 catalog_manager.cc:4830] TS 327aa26a8e434a609ebcb382aa665fce (127.0.99.133:38853): tablet 8682bff56e604cc3b7e9b07c8ba3fca5 (table rebalance_test_table_3 [id=f7b0d84dbe0541aaa2606d6e366719c8]) successfully deleted
I20240503 01:59:20.946655 5695 tablet_bootstrap.cc:492] T 8f4c3f70f2714cc299bc0d595f838c85 P c4a1b28608a6413caaf5980d39b99223: Bootstrap replayed 1/1 log segments. Stats: ops{read=218 overwritten=0 applied=218 ignored=0} inserts{seen=890 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:20.947042 5695 tablet_bootstrap.cc:492] T 8f4c3f70f2714cc299bc0d595f838c85 P c4a1b28608a6413caaf5980d39b99223: Bootstrap complete.
I20240503 01:59:20.947250 5695 ts_tablet_manager.cc:1367] T 8f4c3f70f2714cc299bc0d595f838c85 P c4a1b28608a6413caaf5980d39b99223: Time spent bootstrapping tablet: real 0.154s user 0.135s sys 0.014s
I20240503 01:59:20.947746 5695 raft_consensus.cc:348] T 8f4c3f70f2714cc299bc0d595f838c85 P c4a1b28608a6413caaf5980d39b99223 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 218 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: true } }
I20240503 01:59:20.947896 5695 raft_consensus.cc:729] T 8f4c3f70f2714cc299bc0d595f838c85 P c4a1b28608a6413caaf5980d39b99223 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c4a1b28608a6413caaf5980d39b99223, State: Initialized, Role: LEARNER
I20240503 01:59:20.948019 5695 consensus_queue.cc:260] T 8f4c3f70f2714cc299bc0d595f838c85 P c4a1b28608a6413caaf5980d39b99223 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 218, Last appended: 1.218, Last appended by leader: 218, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 218 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: true } }
I20240503 01:59:20.948338 5695 ts_tablet_manager.cc:1398] T 8f4c3f70f2714cc299bc0d595f838c85 P c4a1b28608a6413caaf5980d39b99223: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:20.948948 3075 tablet_copy_service.cc:342] P 6568f69a78c44bb985162d2b7d0a9376: Request end of tablet copy session c4a1b28608a6413caaf5980d39b99223-8f4c3f70f2714cc299bc0d595f838c85 received from {username='slave'} at 127.0.99.136:59125
I20240503 01:59:20.949043 3075 tablet_copy_service.cc:433] P 6568f69a78c44bb985162d2b7d0a9376: ending tablet copy session c4a1b28608a6413caaf5980d39b99223-8f4c3f70f2714cc299bc0d595f838c85 on tablet 8f4c3f70f2714cc299bc0d595f838c85 with peer c4a1b28608a6413caaf5980d39b99223
I20240503 01:59:20.961416 5701 tablet_bootstrap.cc:492] T 10ecacdfb9a44d068b0df24e991612ee P c4a1b28608a6413caaf5980d39b99223: Bootstrap replayed 1/1 log segments. Stats: ops{read=149 overwritten=0 applied=149 ignored=0} inserts{seen=818 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:20.961851 5701 tablet_bootstrap.cc:492] T 10ecacdfb9a44d068b0df24e991612ee P c4a1b28608a6413caaf5980d39b99223: Bootstrap complete.
I20240503 01:59:20.962097 5701 ts_tablet_manager.cc:1367] T 10ecacdfb9a44d068b0df24e991612ee P c4a1b28608a6413caaf5980d39b99223: Time spent bootstrapping tablet: real 0.127s user 0.117s sys 0.004s
I20240503 01:59:20.962594 5701 raft_consensus.cc:348] T 10ecacdfb9a44d068b0df24e991612ee P c4a1b28608a6413caaf5980d39b99223 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 149 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: true } }
I20240503 01:59:20.962735 5701 raft_consensus.cc:729] T 10ecacdfb9a44d068b0df24e991612ee P c4a1b28608a6413caaf5980d39b99223 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c4a1b28608a6413caaf5980d39b99223, State: Initialized, Role: LEARNER
I20240503 01:59:20.962829 5701 consensus_queue.cc:260] T 10ecacdfb9a44d068b0df24e991612ee P c4a1b28608a6413caaf5980d39b99223 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 149, Last appended: 1.149, Last appended by leader: 149, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 149 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: true } }
I20240503 01:59:20.963151 5701 ts_tablet_manager.cc:1398] T 10ecacdfb9a44d068b0df24e991612ee P c4a1b28608a6413caaf5980d39b99223: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:20.963829 2944 tablet_copy_service.cc:342] P 327aa26a8e434a609ebcb382aa665fce: Request end of tablet copy session c4a1b28608a6413caaf5980d39b99223-10ecacdfb9a44d068b0df24e991612ee received from {username='slave'} at 127.0.99.136:60173
I20240503 01:59:20.963955 2944 tablet_copy_service.cc:433] P 327aa26a8e434a609ebcb382aa665fce: ending tablet copy session c4a1b28608a6413caaf5980d39b99223-10ecacdfb9a44d068b0df24e991612ee on tablet 10ecacdfb9a44d068b0df24e991612ee with peer c4a1b28608a6413caaf5980d39b99223
I20240503 01:59:20.971395 5701 ts_tablet_manager.cc:902] T 43008abe74604666b330d78ec0f51c7c P c4a1b28608a6413caaf5980d39b99223: Initiating tablet copy from peer a1a844ecd71145e4a362f4c42500fbcb (127.0.99.130:46033)
I20240503 01:59:20.971980 5701 tablet_copy_client.cc:308] T 43008abe74604666b330d78ec0f51c7c P c4a1b28608a6413caaf5980d39b99223: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:46033
I20240503 01:59:20.976305 2551 tablet_copy_service.cc:140] P a1a844ecd71145e4a362f4c42500fbcb: Received BeginTabletCopySession request for tablet 43008abe74604666b330d78ec0f51c7c from peer c4a1b28608a6413caaf5980d39b99223 ({username='slave'} at 127.0.99.136:48841)
I20240503 01:59:20.976433 2551 tablet_copy_service.cc:161] P a1a844ecd71145e4a362f4c42500fbcb: Beginning new tablet copy session on tablet 43008abe74604666b330d78ec0f51c7c from peer c4a1b28608a6413caaf5980d39b99223 at {username='slave'} at 127.0.99.136:48841: session id = c4a1b28608a6413caaf5980d39b99223-43008abe74604666b330d78ec0f51c7c
I20240503 01:59:20.977484 2551 tablet_copy_source_session.cc:205] T 43008abe74604666b330d78ec0f51c7c P a1a844ecd71145e4a362f4c42500fbcb: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:20.978456 5701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43008abe74604666b330d78ec0f51c7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:20.981149 5701 tablet_copy_client.cc:783] T 43008abe74604666b330d78ec0f51c7c P c4a1b28608a6413caaf5980d39b99223: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:20.981323 5701 tablet_copy_client.cc:647] T 43008abe74604666b330d78ec0f51c7c P c4a1b28608a6413caaf5980d39b99223: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:20.983546 5701 tablet_copy_client.cc:523] T 43008abe74604666b330d78ec0f51c7c P c4a1b28608a6413caaf5980d39b99223: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:20.984947 5701 tablet_bootstrap.cc:492] T 43008abe74604666b330d78ec0f51c7c P c4a1b28608a6413caaf5980d39b99223: Bootstrap starting.
I20240503 01:59:21.023788 3054 consensus_queue.cc:237] T c9787dcf6d674611a2f06c143a291bb5 P 6568f69a78c44bb985162d2b7d0a9376 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 262, Committed index: 262, Last appended: 1.262, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 263 OBSOLETE_local: false peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { replace: true } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: true } }
I20240503 01:59:21.025640 2523 raft_consensus.cc:1261] T c9787dcf6d674611a2f06c143a291bb5 P a1a844ecd71145e4a362f4c42500fbcb [term 1 FOLLOWER]: Refusing update from remote peer 6568f69a78c44bb985162d2b7d0a9376: Log matching property violated. Preceding OpId in replica: term: 1 index: 262. Preceding OpId from leader: term: 1 index: 263. (index mismatch)
I20240503 01:59:21.025851 2398 raft_consensus.cc:1261] T c9787dcf6d674611a2f06c143a291bb5 P b63928177fa54227a5ff9929265ab9dd [term 1 FOLLOWER]: Refusing update from remote peer 6568f69a78c44bb985162d2b7d0a9376: Log matching property violated. Preceding OpId in replica: term: 1 index: 262. Preceding OpId from leader: term: 1 index: 263. (index mismatch)
I20240503 01:59:21.026228 2789 raft_consensus.cc:1261] T c9787dcf6d674611a2f06c143a291bb5 P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Refusing update from remote peer 6568f69a78c44bb985162d2b7d0a9376: Log matching property violated. Preceding OpId in replica: term: 1 index: 262. Preceding OpId from leader: term: 1 index: 263. (index mismatch)
I20240503 01:59:21.026683 5530 consensus_queue.cc:1035] T c9787dcf6d674611a2f06c143a291bb5 P 6568f69a78c44bb985162d2b7d0a9376 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 263, Last known committed idx: 262, Time since last communication: 0.000s
I20240503 01:59:21.026854 5519 consensus_queue.cc:1035] T c9787dcf6d674611a2f06c143a291bb5 P 6568f69a78c44bb985162d2b7d0a9376 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 263, Last known committed idx: 262, Time since last communication: 0.000s
I20240503 01:59:21.027119 5530 consensus_queue.cc:1035] T c9787dcf6d674611a2f06c143a291bb5 P 6568f69a78c44bb985162d2b7d0a9376 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 263, Last known committed idx: 262, Time since last communication: 0.000s
I20240503 01:59:21.027734 2920 raft_consensus.cc:1261] T c9787dcf6d674611a2f06c143a291bb5 P 327aa26a8e434a609ebcb382aa665fce [term 1 FOLLOWER]: Refusing update from remote peer 6568f69a78c44bb985162d2b7d0a9376: Log matching property violated. Preceding OpId in replica: term: 1 index: 262. Preceding OpId from leader: term: 1 index: 263. (index mismatch)
W20240503 01:59:21.028380 2988 consensus_peers.cc:485] T c9787dcf6d674611a2f06c143a291bb5 P 6568f69a78c44bb985162d2b7d0a9376 -> Peer c4a1b28608a6413caaf5980d39b99223 (127.0.99.136:33113): Couldn't send request to peer c4a1b28608a6413caaf5980d39b99223. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c9787dcf6d674611a2f06c143a291bb5. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:21.028510 5528 consensus_queue.cc:1035] T c9787dcf6d674611a2f06c143a291bb5 P 6568f69a78c44bb985162d2b7d0a9376 [LEADER]: Connected to new peer: Peer: permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 263, Last known committed idx: 262, Time since last communication: 0.000s
I20240503 01:59:21.029794 5530 raft_consensus.cc:2932] T c9787dcf6d674611a2f06c143a291bb5 P 6568f69a78c44bb985162d2b7d0a9376 [term 1 LEADER]: Committing config change with OpId 1.263: config changed from index -1 to 263, NON_VOTER c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) added. New config: { opid_index: 263 OBSOLETE_local: false peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { replace: true } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: true } } }
I20240503 01:59:21.030248 2789 raft_consensus.cc:2932] T c9787dcf6d674611a2f06c143a291bb5 P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Committing config change with OpId 1.263: config changed from index -1 to 263, NON_VOTER c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) added. New config: { opid_index: 263 OBSOLETE_local: false peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { replace: true } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: true } } }
I20240503 01:59:21.030817 2920 raft_consensus.cc:2932] T c9787dcf6d674611a2f06c143a291bb5 P 327aa26a8e434a609ebcb382aa665fce [term 1 FOLLOWER]: Committing config change with OpId 1.263: config changed from index -1 to 263, NON_VOTER c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) added. New config: { opid_index: 263 OBSOLETE_local: false peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { replace: true } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: true } } }
I20240503 01:59:21.033725 2253 catalog_manager.cc:5484] T c9787dcf6d674611a2f06c143a291bb5 P 52c196c15e3348b985c689652bc49918 reported cstate change: config changed from index -1 to 263, NON_VOTER c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "6568f69a78c44bb985162d2b7d0a9376" committed_config { opid_index: 263 OBSOLETE_local: false peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { replace: true } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: true } } }
I20240503 01:59:21.033910 2398 raft_consensus.cc:2932] T c9787dcf6d674611a2f06c143a291bb5 P b63928177fa54227a5ff9929265ab9dd [term 1 FOLLOWER]: Committing config change with OpId 1.263: config changed from index -1 to 263, NON_VOTER c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) added. New config: { opid_index: 263 OBSOLETE_local: false peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { replace: true } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: true } } }
I20240503 01:59:21.038367 2523 raft_consensus.cc:2932] T c9787dcf6d674611a2f06c143a291bb5 P a1a844ecd71145e4a362f4c42500fbcb [term 1 FOLLOWER]: Committing config change with OpId 1.263: config changed from index -1 to 263, NON_VOTER c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) added. New config: { opid_index: 263 OBSOLETE_local: false peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { replace: true } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: true } } }
I20240503 01:59:21.183380 5701 tablet_bootstrap.cc:492] T 43008abe74604666b330d78ec0f51c7c P c4a1b28608a6413caaf5980d39b99223: Bootstrap replayed 1/1 log segments. Stats: ops{read=279 overwritten=0 applied=279 ignored=0} inserts{seen=907 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:21.183672 5701 tablet_bootstrap.cc:492] T 43008abe74604666b330d78ec0f51c7c P c4a1b28608a6413caaf5980d39b99223: Bootstrap complete.
I20240503 01:59:21.183854 5701 ts_tablet_manager.cc:1367] T 43008abe74604666b330d78ec0f51c7c P c4a1b28608a6413caaf5980d39b99223: Time spent bootstrapping tablet: real 0.199s user 0.183s sys 0.012s
I20240503 01:59:21.184262 5701 raft_consensus.cc:348] T 43008abe74604666b330d78ec0f51c7c P c4a1b28608a6413caaf5980d39b99223 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 279 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { replace: true } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: true } }
I20240503 01:59:21.184379 5701 raft_consensus.cc:729] T 43008abe74604666b330d78ec0f51c7c P c4a1b28608a6413caaf5980d39b99223 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c4a1b28608a6413caaf5980d39b99223, State: Initialized, Role: LEARNER
I20240503 01:59:21.184489 5701 consensus_queue.cc:260] T 43008abe74604666b330d78ec0f51c7c P c4a1b28608a6413caaf5980d39b99223 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 279, Last appended: 1.279, Last appended by leader: 279, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 279 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { replace: true } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: true } }
I20240503 01:59:21.184813 5701 ts_tablet_manager.cc:1398] T 43008abe74604666b330d78ec0f51c7c P c4a1b28608a6413caaf5980d39b99223: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240503 01:59:21.185532 2551 tablet_copy_service.cc:342] P a1a844ecd71145e4a362f4c42500fbcb: Request end of tablet copy session c4a1b28608a6413caaf5980d39b99223-43008abe74604666b330d78ec0f51c7c received from {username='slave'} at 127.0.99.136:48841
I20240503 01:59:21.185678 2551 tablet_copy_service.cc:433] P a1a844ecd71145e4a362f4c42500fbcb: ending tablet copy session c4a1b28608a6413caaf5980d39b99223-43008abe74604666b330d78ec0f51c7c on tablet 43008abe74604666b330d78ec0f51c7c with peer c4a1b28608a6413caaf5980d39b99223
I20240503 01:59:21.188858 5060 raft_consensus.cc:1203] T 8f4c3f70f2714cc299bc0d595f838c85 P c4a1b28608a6413caaf5980d39b99223 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.217->[1.218-1.218] Dedup: 1.218->[]
I20240503 01:59:21.316437 5060 raft_consensus.cc:1203] T 10ecacdfb9a44d068b0df24e991612ee P c4a1b28608a6413caaf5980d39b99223 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.148->[1.149-1.149] Dedup: 1.149->[]
I20240503 01:59:21.344543 5701 ts_tablet_manager.cc:902] T c10343409a2a423eb0b21759bfc9ba61 P c4a1b28608a6413caaf5980d39b99223: Initiating tablet copy from peer b63928177fa54227a5ff9929265ab9dd (127.0.99.129:36425)
I20240503 01:59:21.345146 5701 tablet_copy_client.cc:308] T c10343409a2a423eb0b21759bfc9ba61 P c4a1b28608a6413caaf5980d39b99223: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:36425
I20240503 01:59:21.345674 2420 tablet_copy_service.cc:140] P b63928177fa54227a5ff9929265ab9dd: Received BeginTabletCopySession request for tablet c10343409a2a423eb0b21759bfc9ba61 from peer c4a1b28608a6413caaf5980d39b99223 ({username='slave'} at 127.0.99.136:57695)
I20240503 01:59:21.345795 2420 tablet_copy_service.cc:161] P b63928177fa54227a5ff9929265ab9dd: Beginning new tablet copy session on tablet c10343409a2a423eb0b21759bfc9ba61 from peer c4a1b28608a6413caaf5980d39b99223 at {username='slave'} at 127.0.99.136:57695: session id = c4a1b28608a6413caaf5980d39b99223-c10343409a2a423eb0b21759bfc9ba61
I20240503 01:59:21.346678 2420 tablet_copy_source_session.cc:205] T c10343409a2a423eb0b21759bfc9ba61 P b63928177fa54227a5ff9929265ab9dd: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:21.347565 5701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c10343409a2a423eb0b21759bfc9ba61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:21.349618 5701 tablet_copy_client.cc:783] T c10343409a2a423eb0b21759bfc9ba61 P c4a1b28608a6413caaf5980d39b99223: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:21.349792 5701 tablet_copy_client.cc:647] T c10343409a2a423eb0b21759bfc9ba61 P c4a1b28608a6413caaf5980d39b99223: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:21.352061 5701 tablet_copy_client.cc:523] T c10343409a2a423eb0b21759bfc9ba61 P c4a1b28608a6413caaf5980d39b99223: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:21.353148 5701 tablet_bootstrap.cc:492] T c10343409a2a423eb0b21759bfc9ba61 P c4a1b28608a6413caaf5980d39b99223: Bootstrap starting.
I20240503 01:59:21.468590 5701 tablet_bootstrap.cc:492] T c10343409a2a423eb0b21759bfc9ba61 P c4a1b28608a6413caaf5980d39b99223: Bootstrap replayed 1/1 log segments. Stats: ops{read=180 overwritten=0 applied=180 ignored=0} inserts{seen=849 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:21.469002 5701 tablet_bootstrap.cc:492] T c10343409a2a423eb0b21759bfc9ba61 P c4a1b28608a6413caaf5980d39b99223: Bootstrap complete.
I20240503 01:59:21.469208 5701 ts_tablet_manager.cc:1367] T c10343409a2a423eb0b21759bfc9ba61 P c4a1b28608a6413caaf5980d39b99223: Time spent bootstrapping tablet: real 0.116s user 0.105s sys 0.011s
I20240503 01:59:21.469686 5701 raft_consensus.cc:348] T c10343409a2a423eb0b21759bfc9ba61 P c4a1b28608a6413caaf5980d39b99223 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 180 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } attrs { replace: true } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: true } }
I20240503 01:59:21.469813 5701 raft_consensus.cc:729] T c10343409a2a423eb0b21759bfc9ba61 P c4a1b28608a6413caaf5980d39b99223 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c4a1b28608a6413caaf5980d39b99223, State: Initialized, Role: LEARNER
I20240503 01:59:21.469906 5701 consensus_queue.cc:260] T c10343409a2a423eb0b21759bfc9ba61 P c4a1b28608a6413caaf5980d39b99223 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 180, Last appended: 1.180, Last appended by leader: 180, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 180 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } attrs { replace: true } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: true } }
I20240503 01:59:21.470197 5701 ts_tablet_manager.cc:1398] T c10343409a2a423eb0b21759bfc9ba61 P c4a1b28608a6413caaf5980d39b99223: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:21.470806 2420 tablet_copy_service.cc:342] P b63928177fa54227a5ff9929265ab9dd: Request end of tablet copy session c4a1b28608a6413caaf5980d39b99223-c10343409a2a423eb0b21759bfc9ba61 received from {username='slave'} at 127.0.99.136:57695
I20240503 01:59:21.470923 2420 tablet_copy_service.cc:433] P b63928177fa54227a5ff9929265ab9dd: ending tablet copy session c4a1b28608a6413caaf5980d39b99223-c10343409a2a423eb0b21759bfc9ba61 on tablet c10343409a2a423eb0b21759bfc9ba61 with peer c4a1b28608a6413caaf5980d39b99223
I20240503 01:59:21.544525 5701 ts_tablet_manager.cc:902] T c9787dcf6d674611a2f06c143a291bb5 P c4a1b28608a6413caaf5980d39b99223: Initiating tablet copy from peer 6568f69a78c44bb985162d2b7d0a9376 (127.0.99.134:43603)
I20240503 01:59:21.545116 5701 tablet_copy_client.cc:308] T c9787dcf6d674611a2f06c143a291bb5 P c4a1b28608a6413caaf5980d39b99223: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.134:43603
I20240503 01:59:21.545634 3075 tablet_copy_service.cc:140] P 6568f69a78c44bb985162d2b7d0a9376: Received BeginTabletCopySession request for tablet c9787dcf6d674611a2f06c143a291bb5 from peer c4a1b28608a6413caaf5980d39b99223 ({username='slave'} at 127.0.99.136:59125)
I20240503 01:59:21.545743 3075 tablet_copy_service.cc:161] P 6568f69a78c44bb985162d2b7d0a9376: Beginning new tablet copy session on tablet c9787dcf6d674611a2f06c143a291bb5 from peer c4a1b28608a6413caaf5980d39b99223 at {username='slave'} at 127.0.99.136:59125: session id = c4a1b28608a6413caaf5980d39b99223-c9787dcf6d674611a2f06c143a291bb5
I20240503 01:59:21.546728 3075 tablet_copy_source_session.cc:205] T c9787dcf6d674611a2f06c143a291bb5 P 6568f69a78c44bb985162d2b7d0a9376: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:21.547746 5701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9787dcf6d674611a2f06c143a291bb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:21.549768 5701 tablet_copy_client.cc:783] T c9787dcf6d674611a2f06c143a291bb5 P c4a1b28608a6413caaf5980d39b99223: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:21.549947 5701 tablet_copy_client.cc:647] T c9787dcf6d674611a2f06c143a291bb5 P c4a1b28608a6413caaf5980d39b99223: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:21.551510 5701 tablet_copy_client.cc:523] T c9787dcf6d674611a2f06c143a291bb5 P c4a1b28608a6413caaf5980d39b99223: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:21.552616 5701 tablet_bootstrap.cc:492] T c9787dcf6d674611a2f06c143a291bb5 P c4a1b28608a6413caaf5980d39b99223: Bootstrap starting.
I20240503 01:59:21.564695 5060 raft_consensus.cc:1203] T 43008abe74604666b330d78ec0f51c7c P c4a1b28608a6413caaf5980d39b99223 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.278->[1.279-1.279] Dedup: 1.279->[]
I20240503 01:59:21.593534 5529 raft_consensus.cc:1050] T 8f4c3f70f2714cc299bc0d595f838c85 P 6568f69a78c44bb985162d2b7d0a9376: attempting to promote NON_VOTER c4a1b28608a6413caaf5980d39b99223 to VOTER
I20240503 01:59:21.594135 5529 consensus_queue.cc:237] T 8f4c3f70f2714cc299bc0d595f838c85 P 6568f69a78c44bb985162d2b7d0a9376 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 218, Committed index: 218, Last appended: 1.218, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 219 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } }
I20240503 01:59:21.596153 2398 raft_consensus.cc:1261] T 8f4c3f70f2714cc299bc0d595f838c85 P b63928177fa54227a5ff9929265ab9dd [term 1 FOLLOWER]: Refusing update from remote peer 6568f69a78c44bb985162d2b7d0a9376: Log matching property violated. Preceding OpId in replica: term: 1 index: 218. Preceding OpId from leader: term: 1 index: 219. (index mismatch)
I20240503 01:59:21.596166 2924 raft_consensus.cc:1261] T 8f4c3f70f2714cc299bc0d595f838c85 P 327aa26a8e434a609ebcb382aa665fce [term 1 FOLLOWER]: Refusing update from remote peer 6568f69a78c44bb985162d2b7d0a9376: Log matching property violated. Preceding OpId in replica: term: 1 index: 218. Preceding OpId from leader: term: 1 index: 219. (index mismatch)
I20240503 01:59:21.596760 5838 consensus_queue.cc:1035] T 8f4c3f70f2714cc299bc0d595f838c85 P 6568f69a78c44bb985162d2b7d0a9376 [LEADER]: Connected to new peer: Peer: permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 219, Last known committed idx: 218, Time since last communication: 0.000s
I20240503 01:59:21.596946 2789 raft_consensus.cc:1261] T 8f4c3f70f2714cc299bc0d595f838c85 P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Refusing update from remote peer 6568f69a78c44bb985162d2b7d0a9376: Log matching property violated. Preceding OpId in replica: term: 1 index: 218. Preceding OpId from leader: term: 1 index: 219. (index mismatch)
I20240503 01:59:21.597579 5519 consensus_queue.cc:1035] T 8f4c3f70f2714cc299bc0d595f838c85 P 6568f69a78c44bb985162d2b7d0a9376 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 219, Last known committed idx: 218, Time since last communication: 0.000s
I20240503 01:59:21.597870 5519 consensus_queue.cc:1035] T 8f4c3f70f2714cc299bc0d595f838c85 P 6568f69a78c44bb985162d2b7d0a9376 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 219, Last known committed idx: 218, Time since last communication: 0.000s
I20240503 01:59:21.598335 5060 raft_consensus.cc:1261] T 8f4c3f70f2714cc299bc0d595f838c85 P c4a1b28608a6413caaf5980d39b99223 [term 1 LEARNER]: Refusing update from remote peer 6568f69a78c44bb985162d2b7d0a9376: Log matching property violated. Preceding OpId in replica: term: 1 index: 218. Preceding OpId from leader: term: 1 index: 219. (index mismatch)
I20240503 01:59:21.599489 5519 consensus_queue.cc:1035] T 8f4c3f70f2714cc299bc0d595f838c85 P 6568f69a78c44bb985162d2b7d0a9376 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 219, Last known committed idx: 218, Time since last communication: 0.000s
I20240503 01:59:21.600932 2523 raft_consensus.cc:1261] T 8f4c3f70f2714cc299bc0d595f838c85 P a1a844ecd71145e4a362f4c42500fbcb [term 1 FOLLOWER]: Refusing update from remote peer 6568f69a78c44bb985162d2b7d0a9376: Log matching property violated. Preceding OpId in replica: term: 1 index: 218. Preceding OpId from leader: term: 1 index: 219. (index mismatch)
I20240503 01:59:21.601480 5837 consensus_queue.cc:1035] T 8f4c3f70f2714cc299bc0d595f838c85 P 6568f69a78c44bb985162d2b7d0a9376 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 219, Last known committed idx: 218, Time since last communication: 0.000s
I20240503 01:59:21.602648 5529 raft_consensus.cc:2932] T 8f4c3f70f2714cc299bc0d595f838c85 P 6568f69a78c44bb985162d2b7d0a9376 [term 1 LEADER]: Committing config change with OpId 1.219: config changed from index 218 to 219, c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 219 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:21.603780 2398 raft_consensus.cc:2932] T 8f4c3f70f2714cc299bc0d595f838c85 P b63928177fa54227a5ff9929265ab9dd [term 1 FOLLOWER]: Committing config change with OpId 1.219: config changed from index 218 to 219, c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 219 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:21.605302 5060 raft_consensus.cc:2932] T 8f4c3f70f2714cc299bc0d595f838c85 P c4a1b28608a6413caaf5980d39b99223 [term 1 FOLLOWER]: Committing config change with OpId 1.219: config changed from index 218 to 219, c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 219 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:21.607151 2789 raft_consensus.cc:2932] T 8f4c3f70f2714cc299bc0d595f838c85 P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Committing config change with OpId 1.219: config changed from index 218 to 219, c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 219 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:21.607107 2256 catalog_manager.cc:5484] T 8f4c3f70f2714cc299bc0d595f838c85 P 6568f69a78c44bb985162d2b7d0a9376 reported cstate change: config changed from index 218 to 219, c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6568f69a78c44bb985162d2b7d0a9376" committed_config { opid_index: 219 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:21.611869 2924 raft_consensus.cc:2932] T 8f4c3f70f2714cc299bc0d595f838c85 P 327aa26a8e434a609ebcb382aa665fce [term 1 FOLLOWER]: Committing config change with OpId 1.219: config changed from index 218 to 219, c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 219 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:21.612372 2523 raft_consensus.cc:2932] T 8f4c3f70f2714cc299bc0d595f838c85 P a1a844ecd71145e4a362f4c42500fbcb [term 1 FOLLOWER]: Committing config change with OpId 1.219: config changed from index 218 to 219, c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 219 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:21.719254 5701 tablet_bootstrap.cc:492] T c9787dcf6d674611a2f06c143a291bb5 P c4a1b28608a6413caaf5980d39b99223: Bootstrap replayed 1/1 log segments. Stats: ops{read=263 overwritten=0 applied=263 ignored=0} inserts{seen=935 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:21.719615 5701 tablet_bootstrap.cc:492] T c9787dcf6d674611a2f06c143a291bb5 P c4a1b28608a6413caaf5980d39b99223: Bootstrap complete.
I20240503 01:59:21.719854 5701 ts_tablet_manager.cc:1367] T c9787dcf6d674611a2f06c143a291bb5 P c4a1b28608a6413caaf5980d39b99223: Time spent bootstrapping tablet: real 0.167s user 0.142s sys 0.021s
I20240503 01:59:21.720368 5701 raft_consensus.cc:348] T c9787dcf6d674611a2f06c143a291bb5 P c4a1b28608a6413caaf5980d39b99223 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 263 OBSOLETE_local: false peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { replace: true } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: true } }
I20240503 01:59:21.720513 5701 raft_consensus.cc:729] T c9787dcf6d674611a2f06c143a291bb5 P c4a1b28608a6413caaf5980d39b99223 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c4a1b28608a6413caaf5980d39b99223, State: Initialized, Role: LEARNER
I20240503 01:59:21.720610 5701 consensus_queue.cc:260] T c9787dcf6d674611a2f06c143a291bb5 P c4a1b28608a6413caaf5980d39b99223 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 263, Last appended: 1.263, Last appended by leader: 263, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 263 OBSOLETE_local: false peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { replace: true } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: true } }
I20240503 01:59:21.720942 5701 ts_tablet_manager.cc:1398] T c9787dcf6d674611a2f06c143a291bb5 P c4a1b28608a6413caaf5980d39b99223: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240503 01:59:21.721863 3075 tablet_copy_service.cc:342] P 6568f69a78c44bb985162d2b7d0a9376: Request end of tablet copy session c4a1b28608a6413caaf5980d39b99223-c9787dcf6d674611a2f06c143a291bb5 received from {username='slave'} at 127.0.99.136:59125
I20240503 01:59:21.722002 3075 tablet_copy_service.cc:433] P 6568f69a78c44bb985162d2b7d0a9376: ending tablet copy session c4a1b28608a6413caaf5980d39b99223-c9787dcf6d674611a2f06c143a291bb5 on tablet c9787dcf6d674611a2f06c143a291bb5 with peer c4a1b28608a6413caaf5980d39b99223
I20240503 01:59:21.781749 3052 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8f4c3f70f2714cc299bc0d595f838c85"
dest_uuid: "6568f69a78c44bb985162d2b7d0a9376"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:55782
I20240503 01:59:21.781921 3052 raft_consensus.cc:595] T 8f4c3f70f2714cc299bc0d595f838c85 P 6568f69a78c44bb985162d2b7d0a9376 [term 1 LEADER]: Received request to transfer leadership
I20240503 01:59:21.814647 5582 raft_consensus.cc:1050] T 10ecacdfb9a44d068b0df24e991612ee P 327aa26a8e434a609ebcb382aa665fce: attempting to promote NON_VOTER c4a1b28608a6413caaf5980d39b99223 to VOTER
I20240503 01:59:21.815110 5582 consensus_queue.cc:237] T 10ecacdfb9a44d068b0df24e991612ee P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 149, Committed index: 149, Last appended: 1.149, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 150 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } }
I20240503 01:59:21.816967 3954 raft_consensus.cc:1261] T 10ecacdfb9a44d068b0df24e991612ee P 4125b1a70e864e129bd54bdc2720d7fa [term 1 FOLLOWER]: Refusing update from remote peer 327aa26a8e434a609ebcb382aa665fce: Log matching property violated. Preceding OpId in replica: term: 1 index: 149. Preceding OpId from leader: term: 1 index: 150. (index mismatch)
I20240503 01:59:21.817036 5060 raft_consensus.cc:1261] T 10ecacdfb9a44d068b0df24e991612ee P c4a1b28608a6413caaf5980d39b99223 [term 1 LEARNER]: Refusing update from remote peer 327aa26a8e434a609ebcb382aa665fce: Log matching property violated. Preceding OpId in replica: term: 1 index: 149. Preceding OpId from leader: term: 1 index: 150. (index mismatch)
I20240503 01:59:21.817248 2661 raft_consensus.cc:1261] T 10ecacdfb9a44d068b0df24e991612ee P b40a9a7b30d34a6db007ed8febd8a759 [term 1 FOLLOWER]: Refusing update from remote peer 327aa26a8e434a609ebcb382aa665fce: Log matching property violated. Preceding OpId in replica: term: 1 index: 149. Preceding OpId from leader: term: 1 index: 150. (index mismatch)
I20240503 01:59:21.817251 2398 raft_consensus.cc:1261] T 10ecacdfb9a44d068b0df24e991612ee P b63928177fa54227a5ff9929265ab9dd [term 1 FOLLOWER]: Refusing update from remote peer 327aa26a8e434a609ebcb382aa665fce: Log matching property violated. Preceding OpId in replica: term: 1 index: 149. Preceding OpId from leader: term: 1 index: 150. (index mismatch)
I20240503 01:59:21.817538 2523 raft_consensus.cc:1261] T 10ecacdfb9a44d068b0df24e991612ee P a1a844ecd71145e4a362f4c42500fbcb [term 1 FOLLOWER]: Refusing update from remote peer 327aa26a8e434a609ebcb382aa665fce: Log matching property violated. Preceding OpId in replica: term: 1 index: 149. Preceding OpId from leader: term: 1 index: 150. (index mismatch)
I20240503 01:59:21.817445 5880 consensus_queue.cc:1035] T 10ecacdfb9a44d068b0df24e991612ee P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 150, Last known committed idx: 149, Time since last communication: 0.000s
I20240503 01:59:21.817914 5799 consensus_queue.cc:1035] T 10ecacdfb9a44d068b0df24e991612ee P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 150, Last known committed idx: 149, Time since last communication: 0.000s
I20240503 01:59:21.818114 5882 consensus_queue.cc:1035] T 10ecacdfb9a44d068b0df24e991612ee P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 150, Last known committed idx: 149, Time since last communication: 0.000s
I20240503 01:59:21.818277 5881 consensus_queue.cc:1035] T 10ecacdfb9a44d068b0df24e991612ee P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 150, Last known committed idx: 149, Time since last communication: 0.000s
I20240503 01:59:21.818748 5883 consensus_queue.cc:1035] T 10ecacdfb9a44d068b0df24e991612ee P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 150, Last known committed idx: 149, Time since last communication: 0.000s
I20240503 01:59:21.821635 5883 raft_consensus.cc:2932] T 10ecacdfb9a44d068b0df24e991612ee P 327aa26a8e434a609ebcb382aa665fce [term 1 LEADER]: Committing config change with OpId 1.150: config changed from index 149 to 150, c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 150 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:21.822259 2523 raft_consensus.cc:2932] T 10ecacdfb9a44d068b0df24e991612ee P a1a844ecd71145e4a362f4c42500fbcb [term 1 FOLLOWER]: Committing config change with OpId 1.150: config changed from index 149 to 150, c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 150 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:21.822350 5060 raft_consensus.cc:2932] T 10ecacdfb9a44d068b0df24e991612ee P c4a1b28608a6413caaf5980d39b99223 [term 1 FOLLOWER]: Committing config change with OpId 1.150: config changed from index 149 to 150, c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 150 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:21.824736 2398 raft_consensus.cc:2932] T 10ecacdfb9a44d068b0df24e991612ee P b63928177fa54227a5ff9929265ab9dd [term 1 FOLLOWER]: Committing config change with OpId 1.150: config changed from index 149 to 150, c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 150 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:21.824770 3954 raft_consensus.cc:2932] T 10ecacdfb9a44d068b0df24e991612ee P 4125b1a70e864e129bd54bdc2720d7fa [term 1 FOLLOWER]: Committing config change with OpId 1.150: config changed from index 149 to 150, c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 150 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:21.825839 2661 raft_consensus.cc:2932] T 10ecacdfb9a44d068b0df24e991612ee P b40a9a7b30d34a6db007ed8febd8a759 [term 1 FOLLOWER]: Committing config change with OpId 1.150: config changed from index 149 to 150, c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 150 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:21.825919 2250 catalog_manager.cc:5484] T 10ecacdfb9a44d068b0df24e991612ee P c4a1b28608a6413caaf5980d39b99223 reported cstate change: config changed from index 149 to 150, c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "327aa26a8e434a609ebcb382aa665fce" committed_config { opid_index: 150 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:21.831552 2924 consensus_queue.cc:237] T 10ecacdfb9a44d068b0df24e991612ee P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 150, Committed index: 150, Last appended: 1.150, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 151 OBSOLETE_local: false peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } }
I20240503 01:59:21.833096 2398 raft_consensus.cc:1261] T 10ecacdfb9a44d068b0df24e991612ee P b63928177fa54227a5ff9929265ab9dd [term 1 FOLLOWER]: Refusing update from remote peer 327aa26a8e434a609ebcb382aa665fce: Log matching property violated. Preceding OpId in replica: term: 1 index: 150. Preceding OpId from leader: term: 1 index: 151. (index mismatch)
I20240503 01:59:21.833359 2523 raft_consensus.cc:1261] T 10ecacdfb9a44d068b0df24e991612ee P a1a844ecd71145e4a362f4c42500fbcb [term 1 FOLLOWER]: Refusing update from remote peer 327aa26a8e434a609ebcb382aa665fce: Log matching property violated. Preceding OpId in replica: term: 1 index: 150. Preceding OpId from leader: term: 1 index: 151. (index mismatch)
I20240503 01:59:21.833490 5060 raft_consensus.cc:1261] T 10ecacdfb9a44d068b0df24e991612ee P c4a1b28608a6413caaf5980d39b99223 [term 1 FOLLOWER]: Refusing update from remote peer 327aa26a8e434a609ebcb382aa665fce: Log matching property violated. Preceding OpId in replica: term: 1 index: 150. Preceding OpId from leader: term: 1 index: 151. (index mismatch)
I20240503 01:59:21.833514 3954 raft_consensus.cc:1261] T 10ecacdfb9a44d068b0df24e991612ee P 4125b1a70e864e129bd54bdc2720d7fa [term 1 FOLLOWER]: Refusing update from remote peer 327aa26a8e434a609ebcb382aa665fce: Log matching property violated. Preceding OpId in replica: term: 1 index: 150. Preceding OpId from leader: term: 1 index: 151. (index mismatch)
I20240503 01:59:21.833818 5880 consensus_queue.cc:1035] T 10ecacdfb9a44d068b0df24e991612ee P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 151, Last known committed idx: 150, Time since last communication: 0.000s
I20240503 01:59:21.833979 5882 consensus_queue.cc:1035] T 10ecacdfb9a44d068b0df24e991612ee P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 151, Last known committed idx: 150, Time since last communication: 0.000s
I20240503 01:59:21.834190 5881 consensus_queue.cc:1035] T 10ecacdfb9a44d068b0df24e991612ee P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 151, Last known committed idx: 150, Time since last communication: 0.000s
I20240503 01:59:21.834378 5883 consensus_queue.cc:1035] T 10ecacdfb9a44d068b0df24e991612ee P 327aa26a8e434a609ebcb382aa665fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 151, Last known committed idx: 150, Time since last communication: 0.000s
I20240503 01:59:21.836369 5582 raft_consensus.cc:2932] T 10ecacdfb9a44d068b0df24e991612ee P 327aa26a8e434a609ebcb382aa665fce [term 1 LEADER]: Committing config change with OpId 1.151: config changed from index 150 to 151, VOTER b40a9a7b30d34a6db007ed8febd8a759 (127.0.99.131) evicted. New config: { opid_index: 151 OBSOLETE_local: false peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:21.836969 5060 raft_consensus.cc:2932] T 10ecacdfb9a44d068b0df24e991612ee P c4a1b28608a6413caaf5980d39b99223 [term 1 FOLLOWER]: Committing config change with OpId 1.151: config changed from index 150 to 151, VOTER b40a9a7b30d34a6db007ed8febd8a759 (127.0.99.131) evicted. New config: { opid_index: 151 OBSOLETE_local: false peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:21.837007 2523 raft_consensus.cc:2932] T 10ecacdfb9a44d068b0df24e991612ee P a1a844ecd71145e4a362f4c42500fbcb [term 1 FOLLOWER]: Committing config change with OpId 1.151: config changed from index 150 to 151, VOTER b40a9a7b30d34a6db007ed8febd8a759 (127.0.99.131) evicted. New config: { opid_index: 151 OBSOLETE_local: false peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:21.838192 2243 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 10ecacdfb9a44d068b0df24e991612ee with cas_config_opid_index 150: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:21.838430 3954 raft_consensus.cc:2932] T 10ecacdfb9a44d068b0df24e991612ee P 4125b1a70e864e129bd54bdc2720d7fa [term 1 FOLLOWER]: Committing config change with OpId 1.151: config changed from index 150 to 151, VOTER b40a9a7b30d34a6db007ed8febd8a759 (127.0.99.131) evicted. New config: { opid_index: 151 OBSOLETE_local: false peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:21.839116 2398 raft_consensus.cc:2932] T 10ecacdfb9a44d068b0df24e991612ee P b63928177fa54227a5ff9929265ab9dd [term 1 FOLLOWER]: Committing config change with OpId 1.151: config changed from index 150 to 151, VOTER b40a9a7b30d34a6db007ed8febd8a759 (127.0.99.131) evicted. New config: { opid_index: 151 OBSOLETE_local: false peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:21.839200 2253 catalog_manager.cc:5484] T 10ecacdfb9a44d068b0df24e991612ee P 327aa26a8e434a609ebcb382aa665fce reported cstate change: config changed from index 150 to 151, VOTER b40a9a7b30d34a6db007ed8febd8a759 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "327aa26a8e434a609ebcb382aa665fce" committed_config { opid_index: 151 OBSOLETE_local: false peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:21.843115 2641 tablet_service.cc:1514] Processing DeleteTablet for tablet 10ecacdfb9a44d068b0df24e991612ee with delete_type TABLET_DATA_TOMBSTONED (TS b40a9a7b30d34a6db007ed8febd8a759 not found in new config with opid_index 151) from {username='slave'} at 127.0.0.1:40954
I20240503 01:59:21.844625 5890 tablet_replica.cc:330] T 10ecacdfb9a44d068b0df24e991612ee P b40a9a7b30d34a6db007ed8febd8a759: stopping tablet replica
I20240503 01:59:21.844900 5890 raft_consensus.cc:2229] T 10ecacdfb9a44d068b0df24e991612ee P b40a9a7b30d34a6db007ed8febd8a759 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:21.845124 5890 raft_consensus.cc:2258] T 10ecacdfb9a44d068b0df24e991612ee P b40a9a7b30d34a6db007ed8febd8a759 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:21.846524 5890 ts_tablet_manager.cc:1875] T 10ecacdfb9a44d068b0df24e991612ee P b40a9a7b30d34a6db007ed8febd8a759: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:21.848336 5890 ts_tablet_manager.cc:1888] T 10ecacdfb9a44d068b0df24e991612ee P b40a9a7b30d34a6db007ed8febd8a759: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.150
I20240503 01:59:21.848405 5890 log.cc:1196] T 10ecacdfb9a44d068b0df24e991612ee P b40a9a7b30d34a6db007ed8febd8a759: Deleting WAL directory at /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/10ecacdfb9a44d068b0df24e991612ee
I20240503 01:59:21.848937 2242 catalog_manager.cc:4830] TS b40a9a7b30d34a6db007ed8febd8a759 (127.0.99.131:34079): tablet 10ecacdfb9a44d068b0df24e991612ee (table rebalance_test_table_3 [id=f7b0d84dbe0541aaa2606d6e366719c8]) successfully deleted
I20240503 01:59:21.920064 5060 raft_consensus.cc:1203] T c9787dcf6d674611a2f06c143a291bb5 P c4a1b28608a6413caaf5980d39b99223 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.262->[1.263-1.263] Dedup: 1.263->[]
I20240503 01:59:21.955189 5060 raft_consensus.cc:1203] T c10343409a2a423eb0b21759bfc9ba61 P c4a1b28608a6413caaf5980d39b99223 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.179->[1.180-1.180] Dedup: 1.180->[]
I20240503 01:59:22.018415 3052 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8f4c3f70f2714cc299bc0d595f838c85"
dest_uuid: "6568f69a78c44bb985162d2b7d0a9376"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:55798
I20240503 01:59:22.018587 3052 raft_consensus.cc:595] T 8f4c3f70f2714cc299bc0d595f838c85 P 6568f69a78c44bb985162d2b7d0a9376 [term 1 LEADER]: Received request to transfer leadership
I20240503 01:59:22.020395 5837 raft_consensus.cc:979] T 8f4c3f70f2714cc299bc0d595f838c85 P 6568f69a78c44bb985162d2b7d0a9376: : Instructing follower c4a1b28608a6413caaf5980d39b99223 to start an election
I20240503 01:59:22.020540 5837 raft_consensus.cc:1067] T 8f4c3f70f2714cc299bc0d595f838c85 P 6568f69a78c44bb985162d2b7d0a9376 [term 1 LEADER]: Signalling peer c4a1b28608a6413caaf5980d39b99223 to start an election
I20240503 01:59:22.021608 5060 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8f4c3f70f2714cc299bc0d595f838c85"
dest_uuid: "c4a1b28608a6413caaf5980d39b99223"
from {username='slave'} at 127.0.99.134:42379
I20240503 01:59:22.021828 5060 raft_consensus.cc:482] T 8f4c3f70f2714cc299bc0d595f838c85 P c4a1b28608a6413caaf5980d39b99223 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 01:59:22.021960 5060 raft_consensus.cc:3037] T 8f4c3f70f2714cc299bc0d595f838c85 P c4a1b28608a6413caaf5980d39b99223 [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:22.023207 5060 raft_consensus.cc:504] T 8f4c3f70f2714cc299bc0d595f838c85 P c4a1b28608a6413caaf5980d39b99223 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 219 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } }
I20240503 01:59:22.026082 2397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f4c3f70f2714cc299bc0d595f838c85" candidate_uuid: "c4a1b28608a6413caaf5980d39b99223" candidate_term: 2 candidate_status { last_received { term: 1 index: 219 } } ignore_live_leader: true dest_uuid: "b63928177fa54227a5ff9929265ab9dd"
I20240503 01:59:22.026072 3052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f4c3f70f2714cc299bc0d595f838c85" candidate_uuid: "c4a1b28608a6413caaf5980d39b99223" candidate_term: 2 candidate_status { last_received { term: 1 index: 219 } } ignore_live_leader: true dest_uuid: "6568f69a78c44bb985162d2b7d0a9376"
I20240503 01:59:22.026248 2397 raft_consensus.cc:3037] T 8f4c3f70f2714cc299bc0d595f838c85 P b63928177fa54227a5ff9929265ab9dd [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:22.026249 3052 raft_consensus.cc:3032] T 8f4c3f70f2714cc299bc0d595f838c85 P 6568f69a78c44bb985162d2b7d0a9376 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 01:59:22.026305 3052 raft_consensus.cc:729] T 8f4c3f70f2714cc299bc0d595f838c85 P 6568f69a78c44bb985162d2b7d0a9376 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6568f69a78c44bb985162d2b7d0a9376, State: Running, Role: LEADER
I20240503 01:59:22.026530 3052 consensus_queue.cc:260] T 8f4c3f70f2714cc299bc0d595f838c85 P 6568f69a78c44bb985162d2b7d0a9376 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 219, Committed index: 219, Last appended: 1.219, Last appended by leader: 219, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 219 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } }
I20240503 01:59:22.026806 3052 raft_consensus.cc:3037] T 8f4c3f70f2714cc299bc0d595f838c85 P 6568f69a78c44bb985162d2b7d0a9376 [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:22.027647 2397 raft_consensus.cc:2454] T 8f4c3f70f2714cc299bc0d595f838c85 P b63928177fa54227a5ff9929265ab9dd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4a1b28608a6413caaf5980d39b99223 in term 2.
I20240503 01:59:22.028061 3052 raft_consensus.cc:2454] T 8f4c3f70f2714cc299bc0d595f838c85 P 6568f69a78c44bb985162d2b7d0a9376 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4a1b28608a6413caaf5980d39b99223 in term 2.
I20240503 01:59:22.028760 5060 leader_election.cc:288] T 8f4c3f70f2714cc299bc0d595f838c85 P c4a1b28608a6413caaf5980d39b99223 [CANDIDATE]: Term 2 election: Requested vote from peers 52c196c15e3348b985c689652bc49918 (127.0.99.132:42887), b63928177fa54227a5ff9929265ab9dd (127.0.99.129:36425), 6568f69a78c44bb985162d2b7d0a9376 (127.0.99.134:43603), 327aa26a8e434a609ebcb382aa665fce (127.0.99.133:38853), a1a844ecd71145e4a362f4c42500fbcb (127.0.99.130:46033)
I20240503 01:59:22.028903 2924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f4c3f70f2714cc299bc0d595f838c85" candidate_uuid: "c4a1b28608a6413caaf5980d39b99223" candidate_term: 2 candidate_status { last_received { term: 1 index: 219 } } ignore_live_leader: true dest_uuid: "327aa26a8e434a609ebcb382aa665fce"
I20240503 01:59:22.029058 2924 raft_consensus.cc:3037] T 8f4c3f70f2714cc299bc0d595f838c85 P 327aa26a8e434a609ebcb382aa665fce [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:22.030232 2924 raft_consensus.cc:2454] T 8f4c3f70f2714cc299bc0d595f838c85 P 327aa26a8e434a609ebcb382aa665fce [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4a1b28608a6413caaf5980d39b99223 in term 2.
I20240503 01:59:22.030761 5001 leader_election.cc:302] T 8f4c3f70f2714cc299bc0d595f838c85 P c4a1b28608a6413caaf5980d39b99223 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 6 voters: 4 yes votes; 0 no votes. yes voters: 327aa26a8e434a609ebcb382aa665fce, 6568f69a78c44bb985162d2b7d0a9376, b63928177fa54227a5ff9929265ab9dd, c4a1b28608a6413caaf5980d39b99223; no voters:
I20240503 01:59:22.031031 5357 raft_consensus.cc:2781] T 8f4c3f70f2714cc299bc0d595f838c85 P c4a1b28608a6413caaf5980d39b99223 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 01:59:22.031239 5357 raft_consensus.cc:686] T 8f4c3f70f2714cc299bc0d595f838c85 P c4a1b28608a6413caaf5980d39b99223 [term 2 LEADER]: Becoming Leader. State: Replica: c4a1b28608a6413caaf5980d39b99223, State: Running, Role: LEADER
I20240503 01:59:22.031507 2523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f4c3f70f2714cc299bc0d595f838c85" candidate_uuid: "c4a1b28608a6413caaf5980d39b99223" candidate_term: 2 candidate_status { last_received { term: 1 index: 219 } } ignore_live_leader: true dest_uuid: "a1a844ecd71145e4a362f4c42500fbcb"
I20240503 01:59:22.031623 2523 raft_consensus.cc:3037] T 8f4c3f70f2714cc299bc0d595f838c85 P a1a844ecd71145e4a362f4c42500fbcb [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:22.031626 5357 consensus_queue.cc:237] T 8f4c3f70f2714cc299bc0d595f838c85 P c4a1b28608a6413caaf5980d39b99223 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 219, Committed index: 219, Last appended: 1.219, Last appended by leader: 219, Current term: 2, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 219 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } attrs { replace: true } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } }
I20240503 01:59:22.032514 2523 raft_consensus.cc:2454] T 8f4c3f70f2714cc299bc0d595f838c85 P a1a844ecd71145e4a362f4c42500fbcb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4a1b28608a6413caaf5980d39b99223 in term 2.
I20240503 01:59:22.033077 2789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f4c3f70f2714cc299bc0d595f838c85" candidate_uuid: "c4a1b28608a6413caaf5980d39b99223" candidate_term: 2 candidate_status { last_received { term: 1 index: 219 } } ignore_live_leader: true dest_uuid: "52c196c15e3348b985c689652bc49918"
I20240503 01:59:22.033237 2789 raft_consensus.cc:3037] T 8f4c3f70f2714cc299bc0d595f838c85 P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:22.034420 2789 raft_consensus.cc:2454] T 8f4c3f70f2714cc299bc0d595f838c85 P 52c196c15e3348b985c689652bc49918 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4a1b28608a6413caaf5980d39b99223 in term 2.
I20240503 01:59:22.036422 2257 catalog_manager.cc:5484] T 8f4c3f70f2714cc299bc0d595f838c85 P c4a1b28608a6413caaf5980d39b99223 reported cstate change: term changed from 1 to 2, leader changed from 6568f69a78c44bb985162d2b7d0a9376 (127.0.99.134) to c4a1b28608a6413caaf5980d39b99223 (127.0.99.136). New cstate: current_term: 2 leader_uuid: "c4a1b28608a6413caaf5980d39b99223" committed_config { opid_index: 219 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:22.157546 5946 raft_consensus.cc:1050] T 43008abe74604666b330d78ec0f51c7c P a1a844ecd71145e4a362f4c42500fbcb: attempting to promote NON_VOTER c4a1b28608a6413caaf5980d39b99223 to VOTER
I20240503 01:59:22.158116 5946 consensus_queue.cc:237] T 43008abe74604666b330d78ec0f51c7c P a1a844ecd71145e4a362f4c42500fbcb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 279, Committed index: 279, Last appended: 1.279, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 280 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { replace: true } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } }
I20240503 01:59:22.159943 3954 raft_consensus.cc:1261] T 43008abe74604666b330d78ec0f51c7c P 4125b1a70e864e129bd54bdc2720d7fa [term 1 FOLLOWER]: Refusing update from remote peer a1a844ecd71145e4a362f4c42500fbcb: Log matching property violated. Preceding OpId in replica: term: 1 index: 279. Preceding OpId from leader: term: 1 index: 280. (index mismatch)
I20240503 01:59:22.160099 2661 raft_consensus.cc:1261] T 43008abe74604666b330d78ec0f51c7c P b40a9a7b30d34a6db007ed8febd8a759 [term 1 FOLLOWER]: Refusing update from remote peer a1a844ecd71145e4a362f4c42500fbcb: Log matching property violated. Preceding OpId in replica: term: 1 index: 279. Preceding OpId from leader: term: 1 index: 280. (index mismatch)
I20240503 01:59:22.160360 5060 raft_consensus.cc:1261] T 43008abe74604666b330d78ec0f51c7c P c4a1b28608a6413caaf5980d39b99223 [term 1 LEARNER]: Refusing update from remote peer a1a844ecd71145e4a362f4c42500fbcb: Log matching property violated. Preceding OpId in replica: term: 1 index: 279. Preceding OpId from leader: term: 1 index: 280. (index mismatch)
I20240503 01:59:22.160472 5949 consensus_queue.cc:1035] T 43008abe74604666b330d78ec0f51c7c P a1a844ecd71145e4a362f4c42500fbcb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 280, Last known committed idx: 279, Time since last communication: 0.000s
I20240503 01:59:22.160809 3052 raft_consensus.cc:1261] T 43008abe74604666b330d78ec0f51c7c P 6568f69a78c44bb985162d2b7d0a9376 [term 1 FOLLOWER]: Refusing update from remote peer a1a844ecd71145e4a362f4c42500fbcb: Log matching property violated. Preceding OpId in replica: term: 1 index: 279. Preceding OpId from leader: term: 1 index: 280. (index mismatch)
I20240503 01:59:22.160883 5834 consensus_queue.cc:1035] T 43008abe74604666b330d78ec0f51c7c P a1a844ecd71145e4a362f4c42500fbcb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 280, Last known committed idx: 279, Time since last communication: 0.000s
I20240503 01:59:22.161024 2397 raft_consensus.cc:1261] T 43008abe74604666b330d78ec0f51c7c P b63928177fa54227a5ff9929265ab9dd [term 1 FOLLOWER]: Refusing update from remote peer a1a844ecd71145e4a362f4c42500fbcb: Log matching property violated. Preceding OpId in replica: term: 1 index: 279. Preceding OpId from leader: term: 1 index: 280. (index mismatch)
I20240503 01:59:22.161062 5946 consensus_queue.cc:1035] T 43008abe74604666b330d78ec0f51c7c P a1a844ecd71145e4a362f4c42500fbcb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 280, Last known committed idx: 279, Time since last communication: 0.000s
I20240503 01:59:22.161226 5948 consensus_queue.cc:1035] T 43008abe74604666b330d78ec0f51c7c P a1a844ecd71145e4a362f4c42500fbcb [LEADER]: Connected to new peer: Peer: permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 280, Last known committed idx: 279, Time since last communication: 0.000s
I20240503 01:59:22.161532 5949 consensus_queue.cc:1035] T 43008abe74604666b330d78ec0f51c7c P a1a844ecd71145e4a362f4c42500fbcb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 280, Last known committed idx: 279, Time since last communication: 0.000s
I20240503 01:59:22.164101 5946 raft_consensus.cc:2932] T 43008abe74604666b330d78ec0f51c7c P a1a844ecd71145e4a362f4c42500fbcb [term 1 LEADER]: Committing config change with OpId 1.280: config changed from index 279 to 280, c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 280 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { replace: true } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:22.164669 2397 raft_consensus.cc:2932] T 43008abe74604666b330d78ec0f51c7c P b63928177fa54227a5ff9929265ab9dd [term 1 FOLLOWER]: Committing config change with OpId 1.280: config changed from index 279 to 280, c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 280 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { replace: true } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:22.165040 2661 raft_consensus.cc:2932] T 43008abe74604666b330d78ec0f51c7c P b40a9a7b30d34a6db007ed8febd8a759 [term 1 FOLLOWER]: Committing config change with OpId 1.280: config changed from index 279 to 280, c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 280 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { replace: true } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:22.165248 5060 raft_consensus.cc:2932] T 43008abe74604666b330d78ec0f51c7c P c4a1b28608a6413caaf5980d39b99223 [term 1 FOLLOWER]: Committing config change with OpId 1.280: config changed from index 279 to 280, c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 280 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { replace: true } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:22.167361 3052 raft_consensus.cc:2932] T 43008abe74604666b330d78ec0f51c7c P 6568f69a78c44bb985162d2b7d0a9376 [term 1 FOLLOWER]: Committing config change with OpId 1.280: config changed from index 279 to 280, c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 280 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { replace: true } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:22.167616 3954 raft_consensus.cc:2932] T 43008abe74604666b330d78ec0f51c7c P 4125b1a70e864e129bd54bdc2720d7fa [term 1 FOLLOWER]: Committing config change with OpId 1.280: config changed from index 279 to 280, c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 280 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { replace: true } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:22.168190 2255 catalog_manager.cc:5484] T 43008abe74604666b330d78ec0f51c7c P a1a844ecd71145e4a362f4c42500fbcb reported cstate change: config changed from index 279 to 280, c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a1a844ecd71145e4a362f4c42500fbcb" committed_config { opid_index: 280 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:22.172231 2523 consensus_queue.cc:237] T 43008abe74604666b330d78ec0f51c7c P a1a844ecd71145e4a362f4c42500fbcb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 280, Committed index: 280, Last appended: 1.280, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 281 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } }
I20240503 01:59:22.174177 3052 raft_consensus.cc:1261] T 43008abe74604666b330d78ec0f51c7c P 6568f69a78c44bb985162d2b7d0a9376 [term 1 FOLLOWER]: Refusing update from remote peer a1a844ecd71145e4a362f4c42500fbcb: Log matching property violated. Preceding OpId in replica: term: 1 index: 280. Preceding OpId from leader: term: 1 index: 281. (index mismatch)
I20240503 01:59:22.174284 2661 raft_consensus.cc:1261] T 43008abe74604666b330d78ec0f51c7c P b40a9a7b30d34a6db007ed8febd8a759 [term 1 FOLLOWER]: Refusing update from remote peer a1a844ecd71145e4a362f4c42500fbcb: Log matching property violated. Preceding OpId in replica: term: 1 index: 280. Preceding OpId from leader: term: 1 index: 281. (index mismatch)
I20240503 01:59:22.174325 3954 raft_consensus.cc:1261] T 43008abe74604666b330d78ec0f51c7c P 4125b1a70e864e129bd54bdc2720d7fa [term 1 FOLLOWER]: Refusing update from remote peer a1a844ecd71145e4a362f4c42500fbcb: Log matching property violated. Preceding OpId in replica: term: 1 index: 280. Preceding OpId from leader: term: 1 index: 281. (index mismatch)
I20240503 01:59:22.174767 5834 consensus_queue.cc:1035] T 43008abe74604666b330d78ec0f51c7c P a1a844ecd71145e4a362f4c42500fbcb [LEADER]: Connected to new peer: Peer: permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 281, Last known committed idx: 280, Time since last communication: 0.000s
I20240503 01:59:22.174973 5060 raft_consensus.cc:1261] T 43008abe74604666b330d78ec0f51c7c P c4a1b28608a6413caaf5980d39b99223 [term 1 FOLLOWER]: Refusing update from remote peer a1a844ecd71145e4a362f4c42500fbcb: Log matching property violated. Preceding OpId in replica: term: 1 index: 280. Preceding OpId from leader: term: 1 index: 281. (index mismatch)
I20240503 01:59:22.174988 5946 consensus_queue.cc:1035] T 43008abe74604666b330d78ec0f51c7c P a1a844ecd71145e4a362f4c42500fbcb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 281, Last known committed idx: 280, Time since last communication: 0.000s
I20240503 01:59:22.175305 5834 consensus_queue.cc:1035] T 43008abe74604666b330d78ec0f51c7c P a1a844ecd71145e4a362f4c42500fbcb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 281, Last known committed idx: 280, Time since last communication: 0.000s
I20240503 01:59:22.175525 5949 consensus_queue.cc:1035] T 43008abe74604666b330d78ec0f51c7c P a1a844ecd71145e4a362f4c42500fbcb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 281, Last known committed idx: 280, Time since last communication: 0.000s
I20240503 01:59:22.177502 5834 raft_consensus.cc:2932] T 43008abe74604666b330d78ec0f51c7c P a1a844ecd71145e4a362f4c42500fbcb [term 1 LEADER]: Committing config change with OpId 1.281: config changed from index 280 to 281, VOTER b63928177fa54227a5ff9929265ab9dd (127.0.99.129) evicted. New config: { opid_index: 281 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:22.178158 3954 raft_consensus.cc:2932] T 43008abe74604666b330d78ec0f51c7c P 4125b1a70e864e129bd54bdc2720d7fa [term 1 FOLLOWER]: Committing config change with OpId 1.281: config changed from index 280 to 281, VOTER b63928177fa54227a5ff9929265ab9dd (127.0.99.129) evicted. New config: { opid_index: 281 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:22.179507 2245 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 43008abe74604666b330d78ec0f51c7c with cas_config_opid_index 280: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:22.179926 3052 raft_consensus.cc:2932] T 43008abe74604666b330d78ec0f51c7c P 6568f69a78c44bb985162d2b7d0a9376 [term 1 FOLLOWER]: Committing config change with OpId 1.281: config changed from index 280 to 281, VOTER b63928177fa54227a5ff9929265ab9dd (127.0.99.129) evicted. New config: { opid_index: 281 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:22.180037 2661 raft_consensus.cc:2932] T 43008abe74604666b330d78ec0f51c7c P b40a9a7b30d34a6db007ed8febd8a759 [term 1 FOLLOWER]: Committing config change with OpId 1.281: config changed from index 280 to 281, VOTER b63928177fa54227a5ff9929265ab9dd (127.0.99.129) evicted. New config: { opid_index: 281 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:22.180138 2257 catalog_manager.cc:5484] T 43008abe74604666b330d78ec0f51c7c P a1a844ecd71145e4a362f4c42500fbcb reported cstate change: config changed from index 280 to 281, VOTER b63928177fa54227a5ff9929265ab9dd (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "a1a844ecd71145e4a362f4c42500fbcb" committed_config { opid_index: 281 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:22.181327 5060 raft_consensus.cc:2932] T 43008abe74604666b330d78ec0f51c7c P c4a1b28608a6413caaf5980d39b99223 [term 1 FOLLOWER]: Committing config change with OpId 1.281: config changed from index 280 to 281, VOTER b63928177fa54227a5ff9929265ab9dd (127.0.99.129) evicted. New config: { opid_index: 281 OBSOLETE_local: false peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:22.184146 2378 tablet_service.cc:1514] Processing DeleteTablet for tablet 43008abe74604666b330d78ec0f51c7c with delete_type TABLET_DATA_TOMBSTONED (TS b63928177fa54227a5ff9929265ab9dd not found in new config with opid_index 281) from {username='slave'} at 127.0.0.1:44104
I20240503 01:59:22.184653 5957 tablet_replica.cc:330] T 43008abe74604666b330d78ec0f51c7c P b63928177fa54227a5ff9929265ab9dd: stopping tablet replica
I20240503 01:59:22.184973 5957 raft_consensus.cc:2229] T 43008abe74604666b330d78ec0f51c7c P b63928177fa54227a5ff9929265ab9dd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:22.185544 5957 raft_consensus.cc:2258] T 43008abe74604666b330d78ec0f51c7c P b63928177fa54227a5ff9929265ab9dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:22.187731 5957 ts_tablet_manager.cc:1875] T 43008abe74604666b330d78ec0f51c7c P b63928177fa54227a5ff9929265ab9dd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:22.190196 5957 ts_tablet_manager.cc:1888] T 43008abe74604666b330d78ec0f51c7c P b63928177fa54227a5ff9929265ab9dd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.280
I20240503 01:59:22.190277 5957 log.cc:1196] T 43008abe74604666b330d78ec0f51c7c P b63928177fa54227a5ff9929265ab9dd: Deleting WAL directory at /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/43008abe74604666b330d78ec0f51c7c
I20240503 01:59:22.190867 2245 catalog_manager.cc:4830] TS b63928177fa54227a5ff9929265ab9dd (127.0.99.129:36425): tablet 43008abe74604666b330d78ec0f51c7c (table rebalance_test_table_0 [id=2de29504ce33484ba12dd9985ab742a9]) successfully deleted
I20240503 01:59:22.407775 5519 raft_consensus.cc:1050] T c9787dcf6d674611a2f06c143a291bb5 P 6568f69a78c44bb985162d2b7d0a9376: attempting to promote NON_VOTER c4a1b28608a6413caaf5980d39b99223 to VOTER
I20240503 01:59:22.408418 5519 consensus_queue.cc:237] T c9787dcf6d674611a2f06c143a291bb5 P 6568f69a78c44bb985162d2b7d0a9376 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 263, Committed index: 263, Last appended: 1.263, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 264 OBSOLETE_local: false peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { replace: true } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } }
I20240503 01:59:22.410425 2924 raft_consensus.cc:1261] T c9787dcf6d674611a2f06c143a291bb5 P 327aa26a8e434a609ebcb382aa665fce [term 1 FOLLOWER]: Refusing update from remote peer 6568f69a78c44bb985162d2b7d0a9376: Log matching property violated. Preceding OpId in replica: term: 1 index: 263. Preceding OpId from leader: term: 1 index: 264. (index mismatch)
I20240503 01:59:22.410501 5060 raft_consensus.cc:1261] T c9787dcf6d674611a2f06c143a291bb5 P c4a1b28608a6413caaf5980d39b99223 [term 1 LEARNER]: Refusing update from remote peer 6568f69a78c44bb985162d2b7d0a9376: Log matching property violated. Preceding OpId in replica: term: 1 index: 263. Preceding OpId from leader: term: 1 index: 264. (index mismatch)
I20240503 01:59:22.410593 2523 raft_consensus.cc:1261] T c9787dcf6d674611a2f06c143a291bb5 P a1a844ecd71145e4a362f4c42500fbcb [term 1 FOLLOWER]: Refusing update from remote peer 6568f69a78c44bb985162d2b7d0a9376: Log matching property violated. Preceding OpId in replica: term: 1 index: 263. Preceding OpId from leader: term: 1 index: 264. (index mismatch)
I20240503 01:59:22.411047 2397 raft_consensus.cc:1261] T c9787dcf6d674611a2f06c143a291bb5 P b63928177fa54227a5ff9929265ab9dd [term 1 FOLLOWER]: Refusing update from remote peer 6568f69a78c44bb985162d2b7d0a9376: Log matching property violated. Preceding OpId in replica: term: 1 index: 263. Preceding OpId from leader: term: 1 index: 264. (index mismatch)
I20240503 01:59:22.411041 5994 consensus_queue.cc:1035] T c9787dcf6d674611a2f06c143a291bb5 P 6568f69a78c44bb985162d2b7d0a9376 [LEADER]: Connected to new peer: Peer: permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 264, Last known committed idx: 263, Time since last communication: 0.000s
I20240503 01:59:22.411129 2789 raft_consensus.cc:1261] T c9787dcf6d674611a2f06c143a291bb5 P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Refusing update from remote peer 6568f69a78c44bb985162d2b7d0a9376: Log matching property violated. Preceding OpId in replica: term: 1 index: 263. Preceding OpId from leader: term: 1 index: 264. (index mismatch)
I20240503 01:59:22.411425 5996 consensus_queue.cc:1035] T c9787dcf6d674611a2f06c143a291bb5 P 6568f69a78c44bb985162d2b7d0a9376 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 264, Last known committed idx: 263, Time since last communication: 0.000s
I20240503 01:59:22.411686 5838 consensus_queue.cc:1035] T c9787dcf6d674611a2f06c143a291bb5 P 6568f69a78c44bb985162d2b7d0a9376 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 264, Last known committed idx: 263, Time since last communication: 0.000s
I20240503 01:59:22.411789 5994 consensus_queue.cc:1035] T c9787dcf6d674611a2f06c143a291bb5 P 6568f69a78c44bb985162d2b7d0a9376 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 264, Last known committed idx: 263, Time since last communication: 0.000s
I20240503 01:59:22.412056 5519 consensus_queue.cc:1035] T c9787dcf6d674611a2f06c143a291bb5 P 6568f69a78c44bb985162d2b7d0a9376 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 264, Last known committed idx: 263, Time since last communication: 0.000s
I20240503 01:59:22.414451 5994 raft_consensus.cc:2932] T c9787dcf6d674611a2f06c143a291bb5 P 6568f69a78c44bb985162d2b7d0a9376 [term 1 LEADER]: Committing config change with OpId 1.264: config changed from index 263 to 264, c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 264 OBSOLETE_local: false peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { replace: true } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:22.414944 5060 raft_consensus.cc:2932] T c9787dcf6d674611a2f06c143a291bb5 P c4a1b28608a6413caaf5980d39b99223 [term 1 FOLLOWER]: Committing config change with OpId 1.264: config changed from index 263 to 264, c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 264 OBSOLETE_local: false peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { replace: true } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:22.416031 2789 raft_consensus.cc:2932] T c9787dcf6d674611a2f06c143a291bb5 P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Committing config change with OpId 1.264: config changed from index 263 to 264, c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 264 OBSOLETE_local: false peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { replace: true } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:22.416013 2397 raft_consensus.cc:2932] T c9787dcf6d674611a2f06c143a291bb5 P b63928177fa54227a5ff9929265ab9dd [term 1 FOLLOWER]: Committing config change with OpId 1.264: config changed from index 263 to 264, c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 264 OBSOLETE_local: false peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { replace: true } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:22.418259 2250 catalog_manager.cc:5484] T c9787dcf6d674611a2f06c143a291bb5 P c4a1b28608a6413caaf5980d39b99223 reported cstate change: config changed from index 263 to 264, c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6568f69a78c44bb985162d2b7d0a9376" committed_config { opid_index: 264 OBSOLETE_local: false peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { replace: true } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:22.419148 2924 raft_consensus.cc:2932] T c9787dcf6d674611a2f06c143a291bb5 P 327aa26a8e434a609ebcb382aa665fce [term 1 FOLLOWER]: Committing config change with OpId 1.264: config changed from index 263 to 264, c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 264 OBSOLETE_local: false peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { replace: true } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:22.419437 2523 raft_consensus.cc:2932] T c9787dcf6d674611a2f06c143a291bb5 P a1a844ecd71145e4a362f4c42500fbcb [term 1 FOLLOWER]: Committing config change with OpId 1.264: config changed from index 263 to 264, c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 264 OBSOLETE_local: false peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } attrs { replace: true } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:22.424875 3052 consensus_queue.cc:237] T c9787dcf6d674611a2f06c143a291bb5 P 6568f69a78c44bb985162d2b7d0a9376 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 264, Committed index: 264, Last appended: 1.264, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 265 OBSOLETE_local: false peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } }
I20240503 01:59:22.426359 2397 raft_consensus.cc:1261] T c9787dcf6d674611a2f06c143a291bb5 P b63928177fa54227a5ff9929265ab9dd [term 1 FOLLOWER]: Refusing update from remote peer 6568f69a78c44bb985162d2b7d0a9376: Log matching property violated. Preceding OpId in replica: term: 1 index: 264. Preceding OpId from leader: term: 1 index: 265. (index mismatch)
I20240503 01:59:22.426474 5060 raft_consensus.cc:1261] T c9787dcf6d674611a2f06c143a291bb5 P c4a1b28608a6413caaf5980d39b99223 [term 1 FOLLOWER]: Refusing update from remote peer 6568f69a78c44bb985162d2b7d0a9376: Log matching property violated. Preceding OpId in replica: term: 1 index: 264. Preceding OpId from leader: term: 1 index: 265. (index mismatch)
I20240503 01:59:22.426474 2523 raft_consensus.cc:1261] T c9787dcf6d674611a2f06c143a291bb5 P a1a844ecd71145e4a362f4c42500fbcb [term 1 FOLLOWER]: Refusing update from remote peer 6568f69a78c44bb985162d2b7d0a9376: Log matching property violated. Preceding OpId in replica: term: 1 index: 264. Preceding OpId from leader: term: 1 index: 265. (index mismatch)
I20240503 01:59:22.426764 5995 consensus_queue.cc:1035] T c9787dcf6d674611a2f06c143a291bb5 P 6568f69a78c44bb985162d2b7d0a9376 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 265, Last known committed idx: 264, Time since last communication: 0.000s
I20240503 01:59:22.426887 2789 raft_consensus.cc:1261] T c9787dcf6d674611a2f06c143a291bb5 P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Refusing update from remote peer 6568f69a78c44bb985162d2b7d0a9376: Log matching property violated. Preceding OpId in replica: term: 1 index: 264. Preceding OpId from leader: term: 1 index: 265. (index mismatch)
I20240503 01:59:22.426978 5996 consensus_queue.cc:1035] T c9787dcf6d674611a2f06c143a291bb5 P 6568f69a78c44bb985162d2b7d0a9376 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 265, Last known committed idx: 264, Time since last communication: 0.000s
I20240503 01:59:22.427323 5837 consensus_queue.cc:1035] T c9787dcf6d674611a2f06c143a291bb5 P 6568f69a78c44bb985162d2b7d0a9376 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 265, Last known committed idx: 264, Time since last communication: 0.000s
I20240503 01:59:22.427963 5837 consensus_queue.cc:1035] T c9787dcf6d674611a2f06c143a291bb5 P 6568f69a78c44bb985162d2b7d0a9376 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 265, Last known committed idx: 264, Time since last communication: 0.000s
I20240503 01:59:22.429087 5996 raft_consensus.cc:2932] T c9787dcf6d674611a2f06c143a291bb5 P 6568f69a78c44bb985162d2b7d0a9376 [term 1 LEADER]: Committing config change with OpId 1.265: config changed from index 264 to 265, VOTER 327aa26a8e434a609ebcb382aa665fce (127.0.99.133) evicted. New config: { opid_index: 265 OBSOLETE_local: false peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:22.430173 5060 raft_consensus.cc:2932] T c9787dcf6d674611a2f06c143a291bb5 P c4a1b28608a6413caaf5980d39b99223 [term 1 FOLLOWER]: Committing config change with OpId 1.265: config changed from index 264 to 265, VOTER 327aa26a8e434a609ebcb382aa665fce (127.0.99.133) evicted. New config: { opid_index: 265 OBSOLETE_local: false peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:22.430891 2243 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c9787dcf6d674611a2f06c143a291bb5 with cas_config_opid_index 264: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:22.430999 2523 raft_consensus.cc:2932] T c9787dcf6d674611a2f06c143a291bb5 P a1a844ecd71145e4a362f4c42500fbcb [term 1 FOLLOWER]: Committing config change with OpId 1.265: config changed from index 264 to 265, VOTER 327aa26a8e434a609ebcb382aa665fce (127.0.99.133) evicted. New config: { opid_index: 265 OBSOLETE_local: false peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:22.431546 2257 catalog_manager.cc:5484] T c9787dcf6d674611a2f06c143a291bb5 P 6568f69a78c44bb985162d2b7d0a9376 reported cstate change: config changed from index 264 to 265, VOTER 327aa26a8e434a609ebcb382aa665fce (127.0.99.133) evicted. New cstate: current_term: 1 leader_uuid: "6568f69a78c44bb985162d2b7d0a9376" committed_config { opid_index: 265 OBSOLETE_local: false peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:22.432308 2789 raft_consensus.cc:2932] T c9787dcf6d674611a2f06c143a291bb5 P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Committing config change with OpId 1.265: config changed from index 264 to 265, VOTER 327aa26a8e434a609ebcb382aa665fce (127.0.99.133) evicted. New config: { opid_index: 265 OBSOLETE_local: false peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:22.434038 2904 tablet_service.cc:1514] Processing DeleteTablet for tablet c9787dcf6d674611a2f06c143a291bb5 with delete_type TABLET_DATA_TOMBSTONED (TS 327aa26a8e434a609ebcb382aa665fce not found in new config with opid_index 265) from {username='slave'} at 127.0.0.1:60782
I20240503 01:59:22.432404 2397 raft_consensus.cc:2932] T c9787dcf6d674611a2f06c143a291bb5 P b63928177fa54227a5ff9929265ab9dd [term 1 FOLLOWER]: Committing config change with OpId 1.265: config changed from index 264 to 265, VOTER 327aa26a8e434a609ebcb382aa665fce (127.0.99.133) evicted. New config: { opid_index: 265 OBSOLETE_local: false peers { permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:22.434540 6003 tablet_replica.cc:330] T c9787dcf6d674611a2f06c143a291bb5 P 327aa26a8e434a609ebcb382aa665fce: stopping tablet replica
I20240503 01:59:22.434751 6003 raft_consensus.cc:2229] T c9787dcf6d674611a2f06c143a291bb5 P 327aa26a8e434a609ebcb382aa665fce [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:22.435329 6003 raft_consensus.cc:2258] T c9787dcf6d674611a2f06c143a291bb5 P 327aa26a8e434a609ebcb382aa665fce [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:22.437939 6003 ts_tablet_manager.cc:1875] T c9787dcf6d674611a2f06c143a291bb5 P 327aa26a8e434a609ebcb382aa665fce: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:22.440021 6003 ts_tablet_manager.cc:1888] T c9787dcf6d674611a2f06c143a291bb5 P 327aa26a8e434a609ebcb382aa665fce: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.264
I20240503 01:59:22.440100 6003 log.cc:1196] T c9787dcf6d674611a2f06c143a291bb5 P 327aa26a8e434a609ebcb382aa665fce: Deleting WAL directory at /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/c9787dcf6d674611a2f06c143a291bb5
I20240503 01:59:22.440591 2243 catalog_manager.cc:4830] TS 327aa26a8e434a609ebcb382aa665fce (127.0.99.133:38853): tablet c9787dcf6d674611a2f06c143a291bb5 (table rebalance_test_table_0 [id=2de29504ce33484ba12dd9985ab742a9]) successfully deleted
I20240503 01:59:22.453696 3052 raft_consensus.cc:1261] T 8f4c3f70f2714cc299bc0d595f838c85 P 6568f69a78c44bb985162d2b7d0a9376 [term 2 FOLLOWER]: Refusing update from remote peer c4a1b28608a6413caaf5980d39b99223: Log matching property violated. Preceding OpId in replica: term: 1 index: 219. Preceding OpId from leader: term: 2 index: 220. (index mismatch)
I20240503 01:59:22.454265 5357 consensus_queue.cc:1035] T 8f4c3f70f2714cc299bc0d595f838c85 P c4a1b28608a6413caaf5980d39b99223 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6568f69a78c44bb985162d2b7d0a9376" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43603 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 220, Last known committed idx: 219, Time since last communication: 0.000s
I20240503 01:59:22.457597 2523 raft_consensus.cc:1261] T 8f4c3f70f2714cc299bc0d595f838c85 P a1a844ecd71145e4a362f4c42500fbcb [term 2 FOLLOWER]: Refusing update from remote peer c4a1b28608a6413caaf5980d39b99223: Log matching property violated. Preceding OpId in replica: term: 1 index: 219. Preceding OpId from leader: term: 2 index: 220. (index mismatch)
I20240503 01:59:22.458060 5357 consensus_queue.cc:1035] T 8f4c3f70f2714cc299bc0d595f838c85 P c4a1b28608a6413caaf5980d39b99223 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 220, Last known committed idx: 219, Time since last communication: 0.000s
I20240503 01:59:22.463132 2924 raft_consensus.cc:1261] T 8f4c3f70f2714cc299bc0d595f838c85 P 327aa26a8e434a609ebcb382aa665fce [term 2 FOLLOWER]: Refusing update from remote peer c4a1b28608a6413caaf5980d39b99223: Log matching property violated. Preceding OpId in replica: term: 1 index: 219. Preceding OpId from leader: term: 2 index: 220. (index mismatch)
I20240503 01:59:22.463541 5357 consensus_queue.cc:1035] T 8f4c3f70f2714cc299bc0d595f838c85 P c4a1b28608a6413caaf5980d39b99223 [LEADER]: Connected to new peer: Peer: permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 220, Last known committed idx: 219, Time since last communication: 0.000s
I20240503 01:59:22.465834 6005 raft_consensus.cc:1050] T c10343409a2a423eb0b21759bfc9ba61 P b63928177fa54227a5ff9929265ab9dd: attempting to promote NON_VOTER c4a1b28608a6413caaf5980d39b99223 to VOTER
I20240503 01:59:22.466698 6005 consensus_queue.cc:237] T c10343409a2a423eb0b21759bfc9ba61 P b63928177fa54227a5ff9929265ab9dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 180, Committed index: 180, Last appended: 1.180, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 181 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } attrs { replace: true } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } }
I20240503 01:59:22.468408 2789 raft_consensus.cc:1261] T 8f4c3f70f2714cc299bc0d595f838c85 P 52c196c15e3348b985c689652bc49918 [term 2 FOLLOWER]: Refusing update from remote peer c4a1b28608a6413caaf5980d39b99223: Log matching property violated. Preceding OpId in replica: term: 1 index: 219. Preceding OpId from leader: term: 2 index: 220. (index mismatch)
I20240503 01:59:22.469282 2523 raft_consensus.cc:1261] T c10343409a2a423eb0b21759bfc9ba61 P a1a844ecd71145e4a362f4c42500fbcb [term 1 FOLLOWER]: Refusing update from remote peer b63928177fa54227a5ff9929265ab9dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 180. Preceding OpId from leader: term: 1 index: 181. (index mismatch)
I20240503 01:59:22.469324 5060 raft_consensus.cc:1261] T c10343409a2a423eb0b21759bfc9ba61 P c4a1b28608a6413caaf5980d39b99223 [term 1 LEARNER]: Refusing update from remote peer b63928177fa54227a5ff9929265ab9dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 180. Preceding OpId from leader: term: 1 index: 181. (index mismatch)
I20240503 01:59:22.469677 2661 raft_consensus.cc:1261] T c10343409a2a423eb0b21759bfc9ba61 P b40a9a7b30d34a6db007ed8febd8a759 [term 1 FOLLOWER]: Refusing update from remote peer b63928177fa54227a5ff9929265ab9dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 180. Preceding OpId from leader: term: 1 index: 181. (index mismatch)
I20240503 01:59:22.469717 3954 raft_consensus.cc:1261] T c10343409a2a423eb0b21759bfc9ba61 P 4125b1a70e864e129bd54bdc2720d7fa [term 1 FOLLOWER]: Refusing update from remote peer b63928177fa54227a5ff9929265ab9dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 180. Preceding OpId from leader: term: 1 index: 181. (index mismatch)
I20240503 01:59:22.469852 6011 consensus_queue.cc:1035] T c10343409a2a423eb0b21759bfc9ba61 P b63928177fa54227a5ff9929265ab9dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 181, Last known committed idx: 180, Time since last communication: 0.000s
I20240503 01:59:22.470211 5378 consensus_queue.cc:1035] T c10343409a2a423eb0b21759bfc9ba61 P b63928177fa54227a5ff9929265ab9dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 181, Last known committed idx: 180, Time since last communication: 0.000s
I20240503 01:59:22.470947 6012 consensus_queue.cc:1035] T c10343409a2a423eb0b21759bfc9ba61 P b63928177fa54227a5ff9929265ab9dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 181, Last known committed idx: 180, Time since last communication: 0.000s
I20240503 01:59:22.471320 2397 raft_consensus.cc:1261] T 8f4c3f70f2714cc299bc0d595f838c85 P b63928177fa54227a5ff9929265ab9dd [term 2 FOLLOWER]: Refusing update from remote peer c4a1b28608a6413caaf5980d39b99223: Log matching property violated. Preceding OpId in replica: term: 1 index: 219. Preceding OpId from leader: term: 2 index: 220. (index mismatch)
I20240503 01:59:22.471377 6013 consensus_queue.cc:1035] T c10343409a2a423eb0b21759bfc9ba61 P b63928177fa54227a5ff9929265ab9dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 181, Last known committed idx: 180, Time since last communication: 0.000s
I20240503 01:59:22.471611 5357 consensus_queue.cc:1035] T 8f4c3f70f2714cc299bc0d595f838c85 P c4a1b28608a6413caaf5980d39b99223 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 220, Last known committed idx: 219, Time since last communication: 0.000s
I20240503 01:59:22.472942 2789 raft_consensus.cc:1261] T c10343409a2a423eb0b21759bfc9ba61 P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Refusing update from remote peer b63928177fa54227a5ff9929265ab9dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 180. Preceding OpId from leader: term: 1 index: 181. (index mismatch)
I20240503 01:59:22.473371 6012 consensus_queue.cc:1035] T c10343409a2a423eb0b21759bfc9ba61 P b63928177fa54227a5ff9929265ab9dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 181, Last known committed idx: 180, Time since last communication: 0.000s
I20240503 01:59:22.474910 6016 consensus_queue.cc:1035] T 8f4c3f70f2714cc299bc0d595f838c85 P c4a1b28608a6413caaf5980d39b99223 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 220, Last known committed idx: 219, Time since last communication: 0.000s
I20240503 01:59:22.475419 6012 raft_consensus.cc:2932] T c10343409a2a423eb0b21759bfc9ba61 P b63928177fa54227a5ff9929265ab9dd [term 1 LEADER]: Committing config change with OpId 1.181: config changed from index 180 to 181, c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 181 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } attrs { replace: true } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:22.475842 2661 raft_consensus.cc:2932] T c10343409a2a423eb0b21759bfc9ba61 P b40a9a7b30d34a6db007ed8febd8a759 [term 1 FOLLOWER]: Committing config change with OpId 1.181: config changed from index 180 to 181, c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 181 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } attrs { replace: true } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:22.477306 5060 raft_consensus.cc:2932] T c10343409a2a423eb0b21759bfc9ba61 P c4a1b28608a6413caaf5980d39b99223 [term 1 FOLLOWER]: Committing config change with OpId 1.181: config changed from index 180 to 181, c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 181 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } attrs { replace: true } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:22.479975 2789 raft_consensus.cc:2932] T c10343409a2a423eb0b21759bfc9ba61 P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Committing config change with OpId 1.181: config changed from index 180 to 181, c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 181 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } attrs { replace: true } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:22.480340 2257 catalog_manager.cc:5484] T c10343409a2a423eb0b21759bfc9ba61 P c4a1b28608a6413caaf5980d39b99223 reported cstate change: config changed from index 180 to 181, c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b63928177fa54227a5ff9929265ab9dd" committed_config { opid_index: 181 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } attrs { replace: true } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:22.485311 3954 raft_consensus.cc:2932] T c10343409a2a423eb0b21759bfc9ba61 P 4125b1a70e864e129bd54bdc2720d7fa [term 1 FOLLOWER]: Committing config change with OpId 1.181: config changed from index 180 to 181, c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 181 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } attrs { replace: true } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:22.485399 2523 raft_consensus.cc:2932] T c10343409a2a423eb0b21759bfc9ba61 P a1a844ecd71145e4a362f4c42500fbcb [term 1 FOLLOWER]: Committing config change with OpId 1.181: config changed from index 180 to 181, c4a1b28608a6413caaf5980d39b99223 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 181 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } attrs { replace: true } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:22.492144 2397 consensus_queue.cc:237] T c10343409a2a423eb0b21759bfc9ba61 P b63928177fa54227a5ff9929265ab9dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 181, Committed index: 181, Last appended: 1.181, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 182 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } }
I20240503 01:59:22.494170 5060 raft_consensus.cc:1261] T c10343409a2a423eb0b21759bfc9ba61 P c4a1b28608a6413caaf5980d39b99223 [term 1 FOLLOWER]: Refusing update from remote peer b63928177fa54227a5ff9929265ab9dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 181. Preceding OpId from leader: term: 1 index: 182. (index mismatch)
I20240503 01:59:22.494303 2661 raft_consensus.cc:1261] T c10343409a2a423eb0b21759bfc9ba61 P b40a9a7b30d34a6db007ed8febd8a759 [term 1 FOLLOWER]: Refusing update from remote peer b63928177fa54227a5ff9929265ab9dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 181. Preceding OpId from leader: term: 1 index: 182. (index mismatch)
I20240503 01:59:22.494340 2789 raft_consensus.cc:1261] T c10343409a2a423eb0b21759bfc9ba61 P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Refusing update from remote peer b63928177fa54227a5ff9929265ab9dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 181. Preceding OpId from leader: term: 1 index: 182. (index mismatch)
I20240503 01:59:22.494724 3954 raft_consensus.cc:1261] T c10343409a2a423eb0b21759bfc9ba61 P 4125b1a70e864e129bd54bdc2720d7fa [term 1 FOLLOWER]: Refusing update from remote peer b63928177fa54227a5ff9929265ab9dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 181. Preceding OpId from leader: term: 1 index: 182. (index mismatch)
I20240503 01:59:22.494829 6011 consensus_queue.cc:1035] T c10343409a2a423eb0b21759bfc9ba61 P b63928177fa54227a5ff9929265ab9dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 182, Last known committed idx: 181, Time since last communication: 0.000s
I20240503 01:59:22.495134 6013 consensus_queue.cc:1035] T c10343409a2a423eb0b21759bfc9ba61 P b63928177fa54227a5ff9929265ab9dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 182, Last known committed idx: 181, Time since last communication: 0.000s
I20240503 01:59:22.495313 5378 consensus_queue.cc:1035] T c10343409a2a423eb0b21759bfc9ba61 P b63928177fa54227a5ff9929265ab9dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 182, Last known committed idx: 181, Time since last communication: 0.000s
I20240503 01:59:22.495446 6011 consensus_queue.cc:1035] T c10343409a2a423eb0b21759bfc9ba61 P b63928177fa54227a5ff9929265ab9dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 182, Last known committed idx: 181, Time since last communication: 0.000s
I20240503 01:59:22.499369 5378 raft_consensus.cc:2932] T c10343409a2a423eb0b21759bfc9ba61 P b63928177fa54227a5ff9929265ab9dd [term 1 LEADER]: Committing config change with OpId 1.182: config changed from index 181 to 182, VOTER a1a844ecd71145e4a362f4c42500fbcb (127.0.99.130) evicted. New config: { opid_index: 182 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:22.501075 3954 raft_consensus.cc:2932] T c10343409a2a423eb0b21759bfc9ba61 P 4125b1a70e864e129bd54bdc2720d7fa [term 1 FOLLOWER]: Committing config change with OpId 1.182: config changed from index 181 to 182, VOTER a1a844ecd71145e4a362f4c42500fbcb (127.0.99.130) evicted. New config: { opid_index: 182 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:22.501441 2245 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c10343409a2a423eb0b21759bfc9ba61 with cas_config_opid_index 181: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:22.502039 5060 raft_consensus.cc:2932] T c10343409a2a423eb0b21759bfc9ba61 P c4a1b28608a6413caaf5980d39b99223 [term 1 FOLLOWER]: Committing config change with OpId 1.182: config changed from index 181 to 182, VOTER a1a844ecd71145e4a362f4c42500fbcb (127.0.99.130) evicted. New config: { opid_index: 182 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:22.502243 2661 raft_consensus.cc:2932] T c10343409a2a423eb0b21759bfc9ba61 P b40a9a7b30d34a6db007ed8febd8a759 [term 1 FOLLOWER]: Committing config change with OpId 1.182: config changed from index 181 to 182, VOTER a1a844ecd71145e4a362f4c42500fbcb (127.0.99.130) evicted. New config: { opid_index: 182 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:22.504559 2253 catalog_manager.cc:5484] T c10343409a2a423eb0b21759bfc9ba61 P b63928177fa54227a5ff9929265ab9dd reported cstate change: config changed from index 181 to 182, VOTER a1a844ecd71145e4a362f4c42500fbcb (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "b63928177fa54227a5ff9929265ab9dd" committed_config { opid_index: 182 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:22.505318 2789 raft_consensus.cc:2932] T c10343409a2a423eb0b21759bfc9ba61 P 52c196c15e3348b985c689652bc49918 [term 1 FOLLOWER]: Committing config change with OpId 1.182: config changed from index 181 to 182, VOTER a1a844ecd71145e4a362f4c42500fbcb (127.0.99.130) evicted. New config: { opid_index: 182 OBSOLETE_local: false peers { permanent_uuid: "b40a9a7b30d34a6db007ed8febd8a759" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34079 } } peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "4125b1a70e864e129bd54bdc2720d7fa" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39501 } attrs { promote: false } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:22.508148 2510 tablet_service.cc:1514] Processing DeleteTablet for tablet c10343409a2a423eb0b21759bfc9ba61 with delete_type TABLET_DATA_TOMBSTONED (TS a1a844ecd71145e4a362f4c42500fbcb not found in new config with opid_index 182) from {username='slave'} at 127.0.0.1:59126
I20240503 01:59:22.508703 6031 tablet_replica.cc:330] T c10343409a2a423eb0b21759bfc9ba61 P a1a844ecd71145e4a362f4c42500fbcb: stopping tablet replica
I20240503 01:59:22.509004 6031 raft_consensus.cc:2229] T c10343409a2a423eb0b21759bfc9ba61 P a1a844ecd71145e4a362f4c42500fbcb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:22.509248 6031 raft_consensus.cc:2258] T c10343409a2a423eb0b21759bfc9ba61 P a1a844ecd71145e4a362f4c42500fbcb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:22.511158 6031 ts_tablet_manager.cc:1875] T c10343409a2a423eb0b21759bfc9ba61 P a1a844ecd71145e4a362f4c42500fbcb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:22.513531 6031 ts_tablet_manager.cc:1888] T c10343409a2a423eb0b21759bfc9ba61 P a1a844ecd71145e4a362f4c42500fbcb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.181
I20240503 01:59:22.513612 6031 log.cc:1196] T c10343409a2a423eb0b21759bfc9ba61 P a1a844ecd71145e4a362f4c42500fbcb: Deleting WAL directory at /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/c10343409a2a423eb0b21759bfc9ba61
I20240503 01:59:22.514252 2245 catalog_manager.cc:4830] TS a1a844ecd71145e4a362f4c42500fbcb (127.0.99.130:46033): tablet c10343409a2a423eb0b21759bfc9ba61 (table rebalance_test_table_1 [id=d0846ac337c64bf1ac218add4c796fff]) successfully deleted
I20240503 01:59:23.099910 5060 consensus_queue.cc:237] T 8f4c3f70f2714cc299bc0d595f838c85 P c4a1b28608a6413caaf5980d39b99223 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 220, Committed index: 220, Last appended: 2.220, Last appended by leader: 219, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 221 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } }
I20240503 01:59:23.101568 2789 raft_consensus.cc:1261] T 8f4c3f70f2714cc299bc0d595f838c85 P 52c196c15e3348b985c689652bc49918 [term 2 FOLLOWER]: Refusing update from remote peer c4a1b28608a6413caaf5980d39b99223: Log matching property violated. Preceding OpId in replica: term: 2 index: 220. Preceding OpId from leader: term: 2 index: 221. (index mismatch)
I20240503 01:59:23.101614 2924 raft_consensus.cc:1261] T 8f4c3f70f2714cc299bc0d595f838c85 P 327aa26a8e434a609ebcb382aa665fce [term 2 FOLLOWER]: Refusing update from remote peer c4a1b28608a6413caaf5980d39b99223: Log matching property violated. Preceding OpId in replica: term: 2 index: 220. Preceding OpId from leader: term: 2 index: 221. (index mismatch)
I20240503 01:59:23.101853 2523 raft_consensus.cc:1261] T 8f4c3f70f2714cc299bc0d595f838c85 P a1a844ecd71145e4a362f4c42500fbcb [term 2 FOLLOWER]: Refusing update from remote peer c4a1b28608a6413caaf5980d39b99223: Log matching property violated. Preceding OpId in replica: term: 2 index: 220. Preceding OpId from leader: term: 2 index: 221. (index mismatch)
I20240503 01:59:23.102254 2397 raft_consensus.cc:1261] T 8f4c3f70f2714cc299bc0d595f838c85 P b63928177fa54227a5ff9929265ab9dd [term 2 FOLLOWER]: Refusing update from remote peer c4a1b28608a6413caaf5980d39b99223: Log matching property violated. Preceding OpId in replica: term: 2 index: 220. Preceding OpId from leader: term: 2 index: 221. (index mismatch)
I20240503 01:59:23.102283 6016 consensus_queue.cc:1035] T 8f4c3f70f2714cc299bc0d595f838c85 P c4a1b28608a6413caaf5980d39b99223 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 221, Last known committed idx: 220, Time since last communication: 0.000s
I20240503 01:59:23.102728 6016 consensus_queue.cc:1035] T 8f4c3f70f2714cc299bc0d595f838c85 P c4a1b28608a6413caaf5980d39b99223 [LEADER]: Connected to new peer: Peer: permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 221, Last known committed idx: 220, Time since last communication: 0.000s
I20240503 01:59:23.103019 6057 consensus_queue.cc:1035] T 8f4c3f70f2714cc299bc0d595f838c85 P c4a1b28608a6413caaf5980d39b99223 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 221, Last known committed idx: 220, Time since last communication: 0.000s
I20240503 01:59:23.104386 6058 consensus_queue.cc:1035] T 8f4c3f70f2714cc299bc0d595f838c85 P c4a1b28608a6413caaf5980d39b99223 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 221, Last known committed idx: 220, Time since last communication: 0.000s
I20240503 01:59:23.105001 6057 raft_consensus.cc:2932] T 8f4c3f70f2714cc299bc0d595f838c85 P c4a1b28608a6413caaf5980d39b99223 [term 2 LEADER]: Committing config change with OpId 2.221: config changed from index 219 to 221, VOTER 6568f69a78c44bb985162d2b7d0a9376 (127.0.99.134) evicted. New config: { opid_index: 221 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:23.105329 2924 raft_consensus.cc:2932] T 8f4c3f70f2714cc299bc0d595f838c85 P 327aa26a8e434a609ebcb382aa665fce [term 2 FOLLOWER]: Committing config change with OpId 2.221: config changed from index 219 to 221, VOTER 6568f69a78c44bb985162d2b7d0a9376 (127.0.99.134) evicted. New config: { opid_index: 221 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:23.105906 2523 raft_consensus.cc:2932] T 8f4c3f70f2714cc299bc0d595f838c85 P a1a844ecd71145e4a362f4c42500fbcb [term 2 FOLLOWER]: Committing config change with OpId 2.221: config changed from index 219 to 221, VOTER 6568f69a78c44bb985162d2b7d0a9376 (127.0.99.134) evicted. New config: { opid_index: 221 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:23.107618 2789 raft_consensus.cc:2932] T 8f4c3f70f2714cc299bc0d595f838c85 P 52c196c15e3348b985c689652bc49918 [term 2 FOLLOWER]: Committing config change with OpId 2.221: config changed from index 219 to 221, VOTER 6568f69a78c44bb985162d2b7d0a9376 (127.0.99.134) evicted. New config: { opid_index: 221 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:23.107977 2242 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8f4c3f70f2714cc299bc0d595f838c85 with cas_config_opid_index 219: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:23.109108 2256 catalog_manager.cc:5484] T 8f4c3f70f2714cc299bc0d595f838c85 P 327aa26a8e434a609ebcb382aa665fce reported cstate change: config changed from index 219 to 221, VOTER 6568f69a78c44bb985162d2b7d0a9376 (127.0.99.134) evicted. New cstate: current_term: 2 leader_uuid: "c4a1b28608a6413caaf5980d39b99223" committed_config { opid_index: 221 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:23.109439 2397 raft_consensus.cc:2932] T 8f4c3f70f2714cc299bc0d595f838c85 P b63928177fa54227a5ff9929265ab9dd [term 2 FOLLOWER]: Committing config change with OpId 2.221: config changed from index 219 to 221, VOTER 6568f69a78c44bb985162d2b7d0a9376 (127.0.99.134) evicted. New config: { opid_index: 221 OBSOLETE_local: false peers { permanent_uuid: "52c196c15e3348b985c689652bc49918" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42887 } } peers { permanent_uuid: "b63928177fa54227a5ff9929265ab9dd" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36425 } } peers { permanent_uuid: "327aa26a8e434a609ebcb382aa665fce" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38853 } } peers { permanent_uuid: "a1a844ecd71145e4a362f4c42500fbcb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46033 } } peers { permanent_uuid: "c4a1b28608a6413caaf5980d39b99223" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33113 } attrs { promote: false } } }
I20240503 01:59:23.112995 3035 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f4c3f70f2714cc299bc0d595f838c85 with delete_type TABLET_DATA_TOMBSTONED (TS 6568f69a78c44bb985162d2b7d0a9376 not found in new config with opid_index 221) from {username='slave'} at 127.0.0.1:53118
I20240503 01:59:23.116861 6065 tablet_replica.cc:330] T 8f4c3f70f2714cc299bc0d595f838c85 P 6568f69a78c44bb985162d2b7d0a9376: stopping tablet replica
I20240503 01:59:23.117015 6065 raft_consensus.cc:2229] T 8f4c3f70f2714cc299bc0d595f838c85 P 6568f69a78c44bb985162d2b7d0a9376 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:23.117121 6065 raft_consensus.cc:2258] T 8f4c3f70f2714cc299bc0d595f838c85 P 6568f69a78c44bb985162d2b7d0a9376 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:23.118707 6065 ts_tablet_manager.cc:1875] T 8f4c3f70f2714cc299bc0d595f838c85 P 6568f69a78c44bb985162d2b7d0a9376: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:23.120548 6065 ts_tablet_manager.cc:1888] T 8f4c3f70f2714cc299bc0d595f838c85 P 6568f69a78c44bb985162d2b7d0a9376: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.220
I20240503 01:59:23.120615 6065 log.cc:1196] T 8f4c3f70f2714cc299bc0d595f838c85 P 6568f69a78c44bb985162d2b7d0a9376: Deleting WAL directory at /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/8f4c3f70f2714cc299bc0d595f838c85
I20240503 01:59:23.121116 2243 catalog_manager.cc:4830] TS 6568f69a78c44bb985162d2b7d0a9376 (127.0.99.134:43603): tablet 8f4c3f70f2714cc299bc0d595f838c85 (table rebalance_test_table_2 [id=18b5c04f80684606bc736d3ca6780b16]) successfully deleted
I20240503 01:59:23.216089 3035 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:23.219785 2904 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 01:59:23.222622 2772 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:23.228674 5047 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:23.228813 2641 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:23.230953 2378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:23.231572 2510 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:23.248172 3938 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:23.350155 2772 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:23.351284 2510 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:23.356952 2641 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:23.358613 3035 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:23.362321 3938 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:23.364118 5047 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:23.368849 2378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:23.376456 2904 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 01:59:23.497505 3938 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:23.498431 2904 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 01:59:23.509657 2772 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:23.509641 3035 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:23.533427 2641 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:23.542870 2378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:23.544565 5047 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:23.572837 2510 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
ab13918717ec4184908d11fa45b8012a | 127.0.99.190:38489 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.99.148:35375 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
327aa26a8e434a609ebcb382aa665fce | 127.0.99.133:38853 | HEALTHY | <none> | 5 | 0
4125b1a70e864e129bd54bdc2720d7fa | 127.0.99.135:39501 | HEALTHY | <none> | 0 | 0
52c196c15e3348b985c689652bc49918 | 127.0.99.132:42887 | HEALTHY | <none> | 0 | 0
6568f69a78c44bb985162d2b7d0a9376 | 127.0.99.134:43603 | HEALTHY | <none> | 3 | 0
a1a844ecd71145e4a362f4c42500fbcb | 127.0.99.130:46033 | HEALTHY | <none> | 1 | 0
b40a9a7b30d34a6db007ed8febd8a759 | 127.0.99.131:34079 | HEALTHY | <none> | 3 | 0
b63928177fa54227a5ff9929265ab9dd | 127.0.99.129:36425 | HEALTHY | <none> | 3 | 0
c4a1b28608a6413caaf5980d39b99223 | 127.0.99.136:33113 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 8
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 8 server(s) checked
local_ip_for_outbound_sockets | 127.0.99.129 | experimental | 127.0.99.129:36425
local_ip_for_outbound_sockets | 127.0.99.130 | experimental | 127.0.99.130:46033
local_ip_for_outbound_sockets | 127.0.99.131 | experimental | 127.0.99.131:34079
local_ip_for_outbound_sockets | 127.0.99.132 | experimental | 127.0.99.132:42887
local_ip_for_outbound_sockets | 127.0.99.133 | experimental | 127.0.99.133:38853
local_ip_for_outbound_sockets | 127.0.99.134 | experimental | 127.0.99.134:43603
local_ip_for_outbound_sockets | 127.0.99.135 | experimental | 127.0.99.135:39501
local_ip_for_outbound_sockets | 127.0.99.136 | experimental | 127.0.99.136:33113
never_fsync | true | unsafe,advanced | all 8 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 8 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 8 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 8 server(s) checked
server_dump_info_format | pb | hidden | all 8 server(s) checked
server_dump_info_path | /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.99.129:36425
server_dump_info_path | /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.0.99.130:46033
server_dump_info_path | /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.99.131:34079
server_dump_info_path | /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.0.99.132:42887
server_dump_info_path | /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.0.99.133:38853
server_dump_info_path | /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.0.99.134:43603
server_dump_info_path | /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.0.99.135:39501
server_dump_info_path | /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714701523524339-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb | hidden | 127.0.99.136:33113
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.99.148:35375 | all 8 server(s) checked
time_source | builtin | all 8 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 9 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 5 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 5 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 5 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 5 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 5 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 5 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 10
First Quartile | 10
Median | 10
Third Quartile | 10
Maximum | 10
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 8
Tables | 6
Tablets | 16
Replicas | 80
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240503 01:59:23.599303 398 log_verifier.cc:126] Checking tablet 0b669a47f1d14ce98ccab24479d39304
I20240503 01:59:23.712200 398 log_verifier.cc:177] Verified matching terms for 223 ops in tablet 0b669a47f1d14ce98ccab24479d39304
I20240503 01:59:23.712432 398 log_verifier.cc:126] Checking tablet 10ecacdfb9a44d068b0df24e991612ee
I20240503 01:59:23.810844 398 log_verifier.cc:177] Verified matching terms for 151 ops in tablet 10ecacdfb9a44d068b0df24e991612ee
I20240503 01:59:23.811009 398 log_verifier.cc:126] Checking tablet 19b3a90e13194997bde2b9a806f8f2b5
I20240503 01:59:23.931391 398 log_verifier.cc:177] Verified matching terms for 240 ops in tablet 19b3a90e13194997bde2b9a806f8f2b5
I20240503 01:59:23.931677 398 log_verifier.cc:126] Checking tablet 240749b0904f4bb2a4c79b7713473411
I20240503 01:59:23.960201 398 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 240749b0904f4bb2a4c79b7713473411
I20240503 01:59:23.960286 398 log_verifier.cc:126] Checking tablet 43008abe74604666b330d78ec0f51c7c
I20240503 01:59:24.091691 398 log_verifier.cc:177] Verified matching terms for 281 ops in tablet 43008abe74604666b330d78ec0f51c7c
I20240503 01:59:24.091928 398 log_verifier.cc:126] Checking tablet 568a05972208414c90f7f4e2ea2c10a9
I20240503 01:59:24.221416 398 log_verifier.cc:177] Verified matching terms for 293 ops in tablet 568a05972208414c90f7f4e2ea2c10a9
I20240503 01:59:24.221617 398 log_verifier.cc:126] Checking tablet 5fd1010447524b6695abb98192dc54f1
I20240503 01:59:24.299500 398 log_verifier.cc:177] Verified matching terms for 154 ops in tablet 5fd1010447524b6695abb98192dc54f1
I20240503 01:59:24.299618 398 log_verifier.cc:126] Checking tablet 8682bff56e604cc3b7e9b07c8ba3fca5
I20240503 01:59:24.382748 398 log_verifier.cc:177] Verified matching terms for 158 ops in tablet 8682bff56e604cc3b7e9b07c8ba3fca5
I20240503 01:59:24.382877 398 log_verifier.cc:126] Checking tablet 8f4c3f70f2714cc299bc0d595f838c85
I20240503 01:59:24.491845 398 log_verifier.cc:177] Verified matching terms for 221 ops in tablet 8f4c3f70f2714cc299bc0d595f838c85
I20240503 01:59:24.492018 398 log_verifier.cc:126] Checking tablet 9d84cbdf02c14873b8838be296977aaf
I20240503 01:59:24.585861 398 log_verifier.cc:177] Verified matching terms for 176 ops in tablet 9d84cbdf02c14873b8838be296977aaf
I20240503 01:59:24.586064 398 log_verifier.cc:126] Checking tablet a6e17a2c4ba0482e96d04eb9d3ba854a
I20240503 01:59:24.682895 398 log_verifier.cc:177] Verified matching terms for 199 ops in tablet a6e17a2c4ba0482e96d04eb9d3ba854a
I20240503 01:59:24.683032 398 log_verifier.cc:126] Checking tablet c10343409a2a423eb0b21759bfc9ba61
I20240503 01:59:24.777158 398 log_verifier.cc:177] Verified matching terms for 182 ops in tablet c10343409a2a423eb0b21759bfc9ba61
I20240503 01:59:24.777308 398 log_verifier.cc:126] Checking tablet c2911cf48f384bf09074f7bea2a4c173
I20240503 01:59:24.891592 398 log_verifier.cc:177] Verified matching terms for 251 ops in tablet c2911cf48f384bf09074f7bea2a4c173
I20240503 01:59:24.891772 398 log_verifier.cc:126] Checking tablet c9787dcf6d674611a2f06c143a291bb5
I20240503 01:59:25.016157 398 log_verifier.cc:177] Verified matching terms for 265 ops in tablet c9787dcf6d674611a2f06c143a291bb5
I20240503 01:59:25.016345 398 log_verifier.cc:126] Checking tablet d4d07c9621134780bc7ece7bb061d934
I20240503 01:59:25.118330 398 log_verifier.cc:177] Verified matching terms for 216 ops in tablet d4d07c9621134780bc7ece7bb061d934
I20240503 01:59:25.118607 398 log_verifier.cc:126] Checking tablet f40f77238d1e4a34a3f7bea387c6406c
I20240503 01:59:25.218930 398 log_verifier.cc:177] Verified matching terms for 209 ops in tablet f40f77238d1e4a34a3f7bea387c6406c
I20240503 01:59:25.220443 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task42rE7p/build/debug/bin/kudu with pid 2318
I20240503 01:59:25.229503 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task42rE7p/build/debug/bin/kudu with pid 2449
W20240503 01:59:25.235966 2860 connection.cc:535] client connection to 127.0.99.130:46033 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 01:59:25.236251 2860 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 01:59:25.236363 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task42rE7p/build/debug/bin/kudu with pid 2580
W20240503 01:59:25.238711 2860 consensus_peers.cc:485] T d4d07c9621134780bc7ece7bb061d934 P 327aa26a8e434a609ebcb382aa665fce -> Peer a1a844ecd71145e4a362f4c42500fbcb (127.0.99.130:46033): Couldn't send request to peer a1a844ecd71145e4a362f4c42500fbcb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.130:46033: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:25.243458 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task42rE7p/build/debug/bin/kudu with pid 2711
I20240503 01:59:25.249985 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task42rE7p/build/debug/bin/kudu with pid 2842
I20240503 01:59:25.256549 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task42rE7p/build/debug/bin/kudu with pid 2973
I20240503 01:59:25.262960 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task42rE7p/build/debug/bin/kudu with pid 3875
I20240503 01:59:25.269016 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task42rE7p/build/debug/bin/kudu with pid 4985
I20240503 01:59:25.274699 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task42rE7p/build/debug/bin/kudu with pid 2227
2024-05-03T01:59:25Z chronyd exiting
[ OK ] RebalanceParamTest.Rebalance/7 (33070 ms)
[----------] 1 test from RebalanceParamTest (33070 ms total)
[----------] 1 test from TserverAddedDuringRebalancingTest
[ RUN ] TserverAddedDuringRebalancingTest.TserverStarts/1
I20240503 01:59:25.373483 398 test_util.cc:268] Using random seed: 296954918
I20240503 01:59:25.374277 398 ts_itest-base.cc:112] Starting cluster with:
I20240503 01:59:25.374311 398 ts_itest-base.cc:113] --------------
I20240503 01:59:25.374331 398 ts_itest-base.cc:114] 8 tablet servers
I20240503 01:59:25.374348 398 ts_itest-base.cc:115] 3 replicas per TS
I20240503 01:59:25.374375 398 ts_itest-base.cc:116] --------------
2024-05-03T01:59:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T01:59:25Z Disabled control of system clock
I20240503 01:59:25.382496 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task42rE7p/build/debug/bin/kudu
/tmp/dist-test-task42rE7p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:38597
--webserver_interface=127.0.99.190
--webserver_port=0
--builtin_ntp_servers=127.0.99.148:34293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.99.190:38597
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240503 01:59:25.497510 6192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:25.497731 6192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:59:25.497803 6192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:25.500706 6192 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:59:25.500774 6192 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 01:59:25.500820 6192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:59:25.500856 6192 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 01:59:25.500895 6192 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 01:59:25.504420 6192 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:34293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.0.99.190:38597
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:38597
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.99.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.6192
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:59:25.505296 6192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:59:25.506395 6192 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 01:59:25.514992 6192 server_base.cc:1034] running on GCE node
W20240503 01:59:25.516052 6200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:25.516932 6197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:25.517966 6198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:25.518431 6192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:59:25.519326 6192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:59:25.520493 6192 hybrid_clock.cc:648] HybridClock initialized: now 1714701565520482 us; error 47 us; skew 500 ppm
I20240503 01:59:25.522346 6192 webserver.cc:460] Webserver started at http://127.0.99.190:40617/ using document root <none> and password file <none>
I20240503 01:59:25.522886 6192 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:25.522951 6192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:25.523123 6192 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:59:25.524739 6192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "6a09095f39a24e218f23f2963783bca2"
format_stamp: "Formatted at 2024-05-03 01:59:25 on dist-test-slave-dtz3"
I20240503 01:59:25.525197 6192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "6a09095f39a24e218f23f2963783bca2"
format_stamp: "Formatted at 2024-05-03 01:59:25 on dist-test-slave-dtz3"
I20240503 01:59:25.528427 6192 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20240503 01:59:25.530644 6206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:25.531616 6192 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:25.531736 6192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/wal
uuid: "6a09095f39a24e218f23f2963783bca2"
format_stamp: "Formatted at 2024-05-03 01:59:25 on dist-test-slave-dtz3"
I20240503 01:59:25.531816 6192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:25.556012 6192 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:25.556591 6192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:59:25.556759 6192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:25.563091 6192 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.190:38597
I20240503 01:59:25.563105 6258 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.190:38597 every 8 connection(s)
I20240503 01:59:25.564016 6192 server_base.cc:1166] Dumped server information to /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 01:59:25.566537 6259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:25.567243 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task42rE7p/build/debug/bin/kudu as pid 6192
I20240503 01:59:25.567353 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 01:59:25.571879 6259 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a09095f39a24e218f23f2963783bca2: Bootstrap starting.
I20240503 01:59:25.574096 6259 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6a09095f39a24e218f23f2963783bca2: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:25.574889 6259 log.cc:824] T 00000000000000000000000000000000 P 6a09095f39a24e218f23f2963783bca2: Log is configured to *not* fsync() on all Append() calls
I20240503 01:59:25.576462 6259 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a09095f39a24e218f23f2963783bca2: No bootstrap required, opened a new log
I20240503 01:59:25.578763 6259 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a09095f39a24e218f23f2963783bca2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a09095f39a24e218f23f2963783bca2" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 38597 } }
I20240503 01:59:25.578946 6259 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6a09095f39a24e218f23f2963783bca2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:25.578979 6259 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a09095f39a24e218f23f2963783bca2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a09095f39a24e218f23f2963783bca2, State: Initialized, Role: FOLLOWER
I20240503 01:59:25.579301 6259 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a09095f39a24e218f23f2963783bca2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a09095f39a24e218f23f2963783bca2" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 38597 } }
I20240503 01:59:25.579396 6259 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6a09095f39a24e218f23f2963783bca2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 01:59:25.579437 6259 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6a09095f39a24e218f23f2963783bca2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 01:59:25.579551 6259 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a09095f39a24e218f23f2963783bca2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:25.580283 6259 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6a09095f39a24e218f23f2963783bca2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a09095f39a24e218f23f2963783bca2" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 38597 } }
I20240503 01:59:25.580593 6259 leader_election.cc:302] T 00000000000000000000000000000000 P 6a09095f39a24e218f23f2963783bca2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6a09095f39a24e218f23f2963783bca2; no voters:
I20240503 01:59:25.580871 6259 leader_election.cc:288] T 00000000000000000000000000000000 P 6a09095f39a24e218f23f2963783bca2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 01:59:25.580996 6264 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6a09095f39a24e218f23f2963783bca2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:25.581189 6264 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6a09095f39a24e218f23f2963783bca2 [term 1 LEADER]: Becoming Leader. State: Replica: 6a09095f39a24e218f23f2963783bca2, State: Running, Role: LEADER
I20240503 01:59:25.581511 6264 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6a09095f39a24e218f23f2963783bca2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a09095f39a24e218f23f2963783bca2" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 38597 } }
I20240503 01:59:25.581857 6259 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a09095f39a24e218f23f2963783bca2 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 01:59:25.583035 6266 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a09095f39a24e218f23f2963783bca2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a09095f39a24e218f23f2963783bca2. Latest consensus state: current_term: 1 leader_uuid: "6a09095f39a24e218f23f2963783bca2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a09095f39a24e218f23f2963783bca2" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 38597 } } }
I20240503 01:59:25.583161 6266 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a09095f39a24e218f23f2963783bca2 [sys.catalog]: This master's current role is: LEADER
I20240503 01:59:25.583393 6265 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a09095f39a24e218f23f2963783bca2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6a09095f39a24e218f23f2963783bca2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a09095f39a24e218f23f2963783bca2" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 38597 } } }
I20240503 01:59:25.583565 6265 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a09095f39a24e218f23f2963783bca2 [sys.catalog]: This master's current role is: LEADER
I20240503 01:59:25.584049 6273 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 01:59:25.586795 6273 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 01:59:25.591621 6273 catalog_manager.cc:1347] Generated new cluster ID: 70664f0c5e2c41bbb6f20b0d8cf8a731
I20240503 01:59:25.591694 6273 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 01:59:25.599841 6273 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 01:59:25.600965 6273 catalog_manager.cc:1504] Loading token signing keys...
I20240503 01:59:25.607694 6273 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 6a09095f39a24e218f23f2963783bca2: Generated new TSK 0
I20240503 01:59:25.608290 6273 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 01:59:25.617213 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task42rE7p/build/debug/bin/kudu
/tmp/dist-test-task42rE7p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.129:0
--local_ip_for_outbound_sockets=127.0.99.129
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:38597
--builtin_ntp_servers=127.0.99.148:34293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
W20240503 01:59:25.718454 6283 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:25.718657 6283 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:59:25.718727 6283 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:25.721599 6283 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:59:25.721666 6283 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:59:25.721796 6283 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.129
I20240503 01:59:25.725258 6283 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:34293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.99.129
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.0.99.190:38597
--never_fsync=true
--heap_profile_path=/tmp/kudu.6283
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.129
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:59:25.726061 6283 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:59:25.727082 6283 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:25.736354 6288 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:25.737402 6291 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:25.738334 6289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:25.739446 6283 server_base.cc:1034] running on GCE node
I20240503 01:59:25.739850 6283 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:59:25.740366 6283 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:59:25.741531 6283 hybrid_clock.cc:648] HybridClock initialized: now 1714701565741522 us; error 23 us; skew 500 ppm
I20240503 01:59:25.743250 6283 webserver.cc:460] Webserver started at http://127.0.99.129:33001/ using document root <none> and password file <none>
I20240503 01:59:25.743813 6283 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:25.743880 6283 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:25.744062 6283 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:59:25.745656 6283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "4127249ba42644388d225d614c935a25"
format_stamp: "Formatted at 2024-05-03 01:59:25 on dist-test-slave-dtz3"
I20240503 01:59:25.746101 6283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "4127249ba42644388d225d614c935a25"
format_stamp: "Formatted at 2024-05-03 01:59:25 on dist-test-slave-dtz3"
I20240503 01:59:25.749154 6283 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240503 01:59:25.751149 6297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:25.752008 6283 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:25.752104 6283 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "4127249ba42644388d225d614c935a25"
format_stamp: "Formatted at 2024-05-03 01:59:25 on dist-test-slave-dtz3"
I20240503 01:59:25.752193 6283 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:25.776178 6283 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:25.776752 6283 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:59:25.776878 6283 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:25.777323 6283 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:25.778131 6283 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:59:25.778175 6283 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:25.778213 6283 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:59:25.778232 6283 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:25.787886 6410 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.129:35613 every 8 connection(s)
I20240503 01:59:25.787869 6283 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.129:35613
I20240503 01:59:25.788861 6283 server_base.cc:1166] Dumped server information to /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 01:59:25.792809 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task42rE7p/build/debug/bin/kudu as pid 6283
I20240503 01:59:25.792919 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 01:59:25.794834 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task42rE7p/build/debug/bin/kudu
/tmp/dist-test-task42rE7p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.130:0
--local_ip_for_outbound_sockets=127.0.99.130
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:38597
--builtin_ntp_servers=127.0.99.148:34293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240503 01:59:25.797369 6411 heartbeater.cc:347] Connected to a master server at 127.0.99.190:38597
I20240503 01:59:25.797609 6411 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:25.798236 6411 heartbeater.cc:511] Master 127.0.99.190:38597 requested a full tablet report, sending...
I20240503 01:59:25.799757 6223 ts_manager.cc:194] Registered new tserver with Master: 4127249ba42644388d225d614c935a25 (127.0.99.129:35613)
I20240503 01:59:25.800908 6223 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.129:53213
W20240503 01:59:25.898895 6414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:25.899075 6414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:59:25.899118 6414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:25.902144 6414 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:59:25.902199 6414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:59:25.902284 6414 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.130
I20240503 01:59:25.905720 6414 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:34293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.99.130
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.0.99.190:38597
--never_fsync=true
--heap_profile_path=/tmp/kudu.6414
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.130
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:59:25.906464 6414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:59:25.907452 6414 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:25.915797 6420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:25.916671 6419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:25.917878 6422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:25.918843 6414 server_base.cc:1034] running on GCE node
I20240503 01:59:25.919201 6414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:59:25.919673 6414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:59:25.920823 6414 hybrid_clock.cc:648] HybridClock initialized: now 1714701565920807 us; error 32 us; skew 500 ppm
I20240503 01:59:25.922715 6414 webserver.cc:460] Webserver started at http://127.0.99.130:36829/ using document root <none> and password file <none>
I20240503 01:59:25.923234 6414 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:25.923295 6414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:25.923451 6414 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:59:25.925144 6414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "0c4821d9898d42799c5789d4d2da3f63"
format_stamp: "Formatted at 2024-05-03 01:59:25 on dist-test-slave-dtz3"
I20240503 01:59:25.925597 6414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "0c4821d9898d42799c5789d4d2da3f63"
format_stamp: "Formatted at 2024-05-03 01:59:25 on dist-test-slave-dtz3"
I20240503 01:59:25.928633 6414 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240503 01:59:25.930665 6428 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:25.931640 6414 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240503 01:59:25.931753 6414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "0c4821d9898d42799c5789d4d2da3f63"
format_stamp: "Formatted at 2024-05-03 01:59:25 on dist-test-slave-dtz3"
I20240503 01:59:25.931835 6414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:25.940395 6414 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:25.940943 6414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:59:25.941087 6414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:25.941526 6414 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:25.942261 6414 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:59:25.942317 6414 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:25.942369 6414 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:59:25.942395 6414 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:25.950371 6414 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.130:34189
I20240503 01:59:25.950412 6541 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.130:34189 every 8 connection(s)
I20240503 01:59:25.951169 6414 server_base.cc:1166] Dumped server information to /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 01:59:25.959333 6542 heartbeater.cc:347] Connected to a master server at 127.0.99.190:38597
I20240503 01:59:25.959551 6542 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:25.960009 6542 heartbeater.cc:511] Master 127.0.99.190:38597 requested a full tablet report, sending...
I20240503 01:59:25.960598 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task42rE7p/build/debug/bin/kudu as pid 6414
I20240503 01:59:25.960762 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 01:59:25.960987 6223 ts_manager.cc:194] Registered new tserver with Master: 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130:34189)
I20240503 01:59:25.961630 6223 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.130:34245
I20240503 01:59:25.962962 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task42rE7p/build/debug/bin/kudu
/tmp/dist-test-task42rE7p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.131:0
--local_ip_for_outbound_sockets=127.0.99.131
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:38597
--builtin_ntp_servers=127.0.99.148:34293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
W20240503 01:59:26.064014 6545 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:26.064222 6545 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:59:26.064270 6545 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:26.067317 6545 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:59:26.067396 6545 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:59:26.067507 6545 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.131
I20240503 01:59:26.071039 6545 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:34293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.99.131
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.0.99.190:38597
--never_fsync=true
--heap_profile_path=/tmp/kudu.6545
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.131
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:59:26.071885 6545 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:59:26.073012 6545 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:26.082145 6553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:26.082962 6550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:26.084173 6551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:26.085166 6545 server_base.cc:1034] running on GCE node
I20240503 01:59:26.085525 6545 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:59:26.086040 6545 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:59:26.087188 6545 hybrid_clock.cc:648] HybridClock initialized: now 1714701566087181 us; error 27 us; skew 500 ppm
I20240503 01:59:26.089046 6545 webserver.cc:460] Webserver started at http://127.0.99.131:36227/ using document root <none> and password file <none>
I20240503 01:59:26.089583 6545 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:26.089648 6545 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:26.089838 6545 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:59:26.091400 6545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "79163c69f6f04a9abd0208d74e1c61be"
format_stamp: "Formatted at 2024-05-03 01:59:26 on dist-test-slave-dtz3"
I20240503 01:59:26.091853 6545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "79163c69f6f04a9abd0208d74e1c61be"
format_stamp: "Formatted at 2024-05-03 01:59:26 on dist-test-slave-dtz3"
I20240503 01:59:26.094959 6545 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 01:59:26.096987 6559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:26.097893 6545 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:26.097994 6545 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "79163c69f6f04a9abd0208d74e1c61be"
format_stamp: "Formatted at 2024-05-03 01:59:26 on dist-test-slave-dtz3"
I20240503 01:59:26.098073 6545 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:26.128090 6545 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:26.128677 6545 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:59:26.128829 6545 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:26.129308 6545 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:26.130132 6545 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:59:26.130182 6545 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:26.130240 6545 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:59:26.130272 6545 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:26.138948 6545 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.131:44005
I20240503 01:59:26.138979 6672 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.131:44005 every 8 connection(s)
I20240503 01:59:26.139807 6545 server_base.cc:1166] Dumped server information to /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 01:59:26.147853 6673 heartbeater.cc:347] Connected to a master server at 127.0.99.190:38597
I20240503 01:59:26.148054 6673 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:26.148296 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task42rE7p/build/debug/bin/kudu as pid 6545
I20240503 01:59:26.148425 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 01:59:26.148566 6673 heartbeater.cc:511] Master 127.0.99.190:38597 requested a full tablet report, sending...
I20240503 01:59:26.149477 6223 ts_manager.cc:194] Registered new tserver with Master: 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131:44005)
I20240503 01:59:26.150092 6223 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.131:54403
I20240503 01:59:26.150485 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task42rE7p/build/debug/bin/kudu
/tmp/dist-test-task42rE7p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.132:0
--local_ip_for_outbound_sockets=127.0.99.132
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:38597
--builtin_ntp_servers=127.0.99.148:34293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
W20240503 01:59:26.252175 6676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:26.252362 6676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:59:26.252404 6676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:26.255352 6676 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:59:26.255407 6676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:59:26.255474 6676 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.132
I20240503 01:59:26.258777 6676 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:34293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.99.132
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.0.99.190:38597
--never_fsync=true
--heap_profile_path=/tmp/kudu.6676
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.132
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:59:26.259577 6676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:59:26.260552 6676 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:26.269619 6684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:26.270546 6681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:26.271384 6682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:26.272598 6676 server_base.cc:1034] running on GCE node
I20240503 01:59:26.272986 6676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:59:26.273576 6676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:59:26.274740 6676 hybrid_clock.cc:648] HybridClock initialized: now 1714701566274718 us; error 41 us; skew 500 ppm
I20240503 01:59:26.276609 6676 webserver.cc:460] Webserver started at http://127.0.99.132:43047/ using document root <none> and password file <none>
I20240503 01:59:26.277172 6676 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:26.277232 6676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:26.277397 6676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:59:26.278978 6676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "b6b1d5677b8d47788b7968f64b3dac3c"
format_stamp: "Formatted at 2024-05-03 01:59:26 on dist-test-slave-dtz3"
I20240503 01:59:26.279420 6676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "b6b1d5677b8d47788b7968f64b3dac3c"
format_stamp: "Formatted at 2024-05-03 01:59:26 on dist-test-slave-dtz3"
I20240503 01:59:26.282469 6676 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20240503 01:59:26.284555 6690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:26.285487 6676 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:26.285589 6676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "b6b1d5677b8d47788b7968f64b3dac3c"
format_stamp: "Formatted at 2024-05-03 01:59:26 on dist-test-slave-dtz3"
I20240503 01:59:26.285681 6676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:26.329411 6676 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:26.329973 6676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:59:26.330122 6676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:26.330600 6676 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:26.331445 6676 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:59:26.331507 6676 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:26.331571 6676 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:59:26.331597 6676 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:26.339632 6676 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.132:46067
I20240503 01:59:26.339671 6803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.132:46067 every 8 connection(s)
I20240503 01:59:26.340466 6676 server_base.cc:1166] Dumped server information to /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 01:59:26.346164 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task42rE7p/build/debug/bin/kudu as pid 6676
I20240503 01:59:26.346282 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 01:59:26.348358 6804 heartbeater.cc:347] Connected to a master server at 127.0.99.190:38597
I20240503 01:59:26.348470 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task42rE7p/build/debug/bin/kudu
/tmp/dist-test-task42rE7p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:0
--local_ip_for_outbound_sockets=127.0.99.133
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:38597
--builtin_ntp_servers=127.0.99.148:34293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240503 01:59:26.348570 6804 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:26.349152 6804 heartbeater.cc:511] Master 127.0.99.190:38597 requested a full tablet report, sending...
I20240503 01:59:26.350209 6223 ts_manager.cc:194] Registered new tserver with Master: b6b1d5677b8d47788b7968f64b3dac3c (127.0.99.132:46067)
I20240503 01:59:26.350880 6223 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.132:45083
W20240503 01:59:26.455683 6807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:26.455889 6807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:59:26.455957 6807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:26.458909 6807 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:59:26.458976 6807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:59:26.459065 6807 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240503 01:59:26.462487 6807 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:34293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.0.99.190:38597
--never_fsync=true
--heap_profile_path=/tmp/kudu.6807
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:59:26.463306 6807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:59:26.464390 6807 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:26.473182 6813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:26.474251 6815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:26.475134 6812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:26.476346 6807 server_base.cc:1034] running on GCE node
I20240503 01:59:26.476784 6807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:59:26.477326 6807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:59:26.478511 6807 hybrid_clock.cc:648] HybridClock initialized: now 1714701566478475 us; error 57 us; skew 500 ppm
I20240503 01:59:26.480408 6807 webserver.cc:460] Webserver started at http://127.0.99.133:40469/ using document root <none> and password file <none>
I20240503 01:59:26.480981 6807 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:26.481052 6807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:26.481213 6807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:59:26.482833 6807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "9264f24d028347d98f7e4e2d1aedeb79"
format_stamp: "Formatted at 2024-05-03 01:59:26 on dist-test-slave-dtz3"
I20240503 01:59:26.483328 6807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "9264f24d028347d98f7e4e2d1aedeb79"
format_stamp: "Formatted at 2024-05-03 01:59:26 on dist-test-slave-dtz3"
I20240503 01:59:26.486466 6807 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 01:59:26.488417 6821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:26.489429 6807 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:26.489544 6807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "9264f24d028347d98f7e4e2d1aedeb79"
format_stamp: "Formatted at 2024-05-03 01:59:26 on dist-test-slave-dtz3"
I20240503 01:59:26.489635 6807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:26.502131 6807 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:26.502671 6807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:59:26.502808 6807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:26.503242 6807 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:26.504043 6807 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:59:26.504101 6807 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:26.504155 6807 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:59:26.504169 6807 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:26.512881 6807 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:41713
I20240503 01:59:26.512933 6934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:41713 every 8 connection(s)
I20240503 01:59:26.513734 6807 server_base.cc:1166] Dumped server information to /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 01:59:26.514272 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task42rE7p/build/debug/bin/kudu as pid 6807
I20240503 01:59:26.514410 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 01:59:26.516633 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task42rE7p/build/debug/bin/kudu
/tmp/dist-test-task42rE7p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.134:0
--local_ip_for_outbound_sockets=127.0.99.134
--webserver_interface=127.0.99.134
--webserver_port=0
--tserver_master_addrs=127.0.99.190:38597
--builtin_ntp_servers=127.0.99.148:34293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240503 01:59:26.522403 6935 heartbeater.cc:347] Connected to a master server at 127.0.99.190:38597
I20240503 01:59:26.522629 6935 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:26.523113 6935 heartbeater.cc:511] Master 127.0.99.190:38597 requested a full tablet report, sending...
I20240503 01:59:26.524084 6223 ts_manager.cc:194] Registered new tserver with Master: 9264f24d028347d98f7e4e2d1aedeb79 (127.0.99.133:41713)
I20240503 01:59:26.524606 6223 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:41535
W20240503 01:59:26.624878 6938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:26.625084 6938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:59:26.625129 6938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:26.627995 6938 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:59:26.628047 6938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:59:26.628110 6938 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.134
I20240503 01:59:26.631645 6938 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:34293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.134:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.0.99.134
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.0.99.190:38597
--never_fsync=true
--heap_profile_path=/tmp/kudu.6938
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.134
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:59:26.632369 6938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:59:26.633394 6938 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:26.642469 6943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:26.643643 6946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:26.644551 6944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:26.645639 6938 server_base.cc:1034] running on GCE node
I20240503 01:59:26.645965 6938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:59:26.646425 6938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:59:26.647565 6938 hybrid_clock.cc:648] HybridClock initialized: now 1714701566647547 us; error 35 us; skew 500 ppm
I20240503 01:59:26.649313 6938 webserver.cc:460] Webserver started at http://127.0.99.134:43753/ using document root <none> and password file <none>
I20240503 01:59:26.649806 6938 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:26.649859 6938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:26.650015 6938 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:59:26.651523 6938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "65d4632bcdd54ba596acdbbe1b5ef8af"
format_stamp: "Formatted at 2024-05-03 01:59:26 on dist-test-slave-dtz3"
I20240503 01:59:26.651963 6938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "65d4632bcdd54ba596acdbbe1b5ef8af"
format_stamp: "Formatted at 2024-05-03 01:59:26 on dist-test-slave-dtz3"
I20240503 01:59:26.655107 6938 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20240503 01:59:26.657147 6952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:26.658015 6938 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:26.658104 6938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "65d4632bcdd54ba596acdbbe1b5ef8af"
format_stamp: "Formatted at 2024-05-03 01:59:26 on dist-test-slave-dtz3"
I20240503 01:59:26.658190 6938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:26.685211 6938 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:26.685736 6938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:59:26.685882 6938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:26.686349 6938 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:26.687179 6938 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:59:26.687225 6938 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:26.687264 6938 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:59:26.687296 6938 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:26.695636 6938 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.134:35795
I20240503 01:59:26.695741 7065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.134:35795 every 8 connection(s)
I20240503 01:59:26.696491 6938 server_base.cc:1166] Dumped server information to /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 01:59:26.702181 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task42rE7p/build/debug/bin/kudu as pid 6938
I20240503 01:59:26.702311 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240503 01:59:26.704521 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task42rE7p/build/debug/bin/kudu
/tmp/dist-test-task42rE7p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.135:0
--local_ip_for_outbound_sockets=127.0.99.135
--webserver_interface=127.0.99.135
--webserver_port=0
--tserver_master_addrs=127.0.99.190:38597
--builtin_ntp_servers=127.0.99.148:34293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240503 01:59:26.704536 7066 heartbeater.cc:347] Connected to a master server at 127.0.99.190:38597
I20240503 01:59:26.704764 7066 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:26.705227 7066 heartbeater.cc:511] Master 127.0.99.190:38597 requested a full tablet report, sending...
I20240503 01:59:26.706106 6223 ts_manager.cc:194] Registered new tserver with Master: 65d4632bcdd54ba596acdbbe1b5ef8af (127.0.99.134:35795)
I20240503 01:59:26.706633 6223 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.134:45123
I20240503 01:59:26.803411 6411 heartbeater.cc:503] Master 127.0.99.190:38597 was elected leader, sending a full tablet report...
W20240503 01:59:26.808450 7069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:26.808669 7069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:59:26.808751 7069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:26.811718 7069 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:59:26.811775 7069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:59:26.811868 7069 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.135
I20240503 01:59:26.815356 7069 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:34293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.135:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.0.99.135
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.0.99.190:38597
--never_fsync=true
--heap_profile_path=/tmp/kudu.7069
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.135
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:59:26.816126 7069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:59:26.817184 7069 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:26.825884 7077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:26.827003 7074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:26.828007 7075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:26.828979 7069 server_base.cc:1034] running on GCE node
I20240503 01:59:26.829340 7069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:59:26.829890 7069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:59:26.831048 7069 hybrid_clock.cc:648] HybridClock initialized: now 1714701566831038 us; error 24 us; skew 500 ppm
I20240503 01:59:26.832893 7069 webserver.cc:460] Webserver started at http://127.0.99.135:37065/ using document root <none> and password file <none>
I20240503 01:59:26.833402 7069 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:26.833461 7069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:26.833655 7069 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:59:26.835372 7069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "de2f076398ee44008622fb5db2a75952"
format_stamp: "Formatted at 2024-05-03 01:59:26 on dist-test-slave-dtz3"
I20240503 01:59:26.835853 7069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "de2f076398ee44008622fb5db2a75952"
format_stamp: "Formatted at 2024-05-03 01:59:26 on dist-test-slave-dtz3"
I20240503 01:59:26.838990 7069 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20240503 01:59:26.841054 7083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:26.841979 7069 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:26.842082 7069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "de2f076398ee44008622fb5db2a75952"
format_stamp: "Formatted at 2024-05-03 01:59:26 on dist-test-slave-dtz3"
I20240503 01:59:26.842155 7069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:26.870692 7069 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:26.871276 7069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:59:26.871424 7069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:26.871932 7069 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:26.872833 7069 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:59:26.872916 7069 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:26.873026 7069 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:59:26.873055 7069 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:26.881721 7069 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.135:34035
I20240503 01:59:26.881752 7196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.135:34035 every 8 connection(s)
I20240503 01:59:26.882699 7069 server_base.cc:1166] Dumped server information to /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 01:59:26.889899 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task42rE7p/build/debug/bin/kudu as pid 7069
I20240503 01:59:26.890055 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240503 01:59:26.891012 7197 heartbeater.cc:347] Connected to a master server at 127.0.99.190:38597
I20240503 01:59:26.891209 7197 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:26.891760 7197 heartbeater.cc:511] Master 127.0.99.190:38597 requested a full tablet report, sending...
I20240503 01:59:26.892210 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task42rE7p/build/debug/bin/kudu
/tmp/dist-test-task42rE7p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.136:0
--local_ip_for_outbound_sockets=127.0.99.136
--webserver_interface=127.0.99.136
--webserver_port=0
--tserver_master_addrs=127.0.99.190:38597
--builtin_ntp_servers=127.0.99.148:34293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240503 01:59:26.892781 6223 ts_manager.cc:194] Registered new tserver with Master: de2f076398ee44008622fb5db2a75952 (127.0.99.135:34035)
I20240503 01:59:26.893368 6223 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.135:58149
I20240503 01:59:26.963661 6542 heartbeater.cc:503] Master 127.0.99.190:38597 was elected leader, sending a full tablet report...
W20240503 01:59:26.996647 7200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:26.996907 7200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:59:26.996984 7200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:26.999809 7200 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:59:26.999871 7200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:59:26.999962 7200 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.136
I20240503 01:59:27.003360 7200 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:34293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.136:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.0.99.136
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.0.99.190:38597
--never_fsync=true
--heap_profile_path=/tmp/kudu.7200
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.136
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:59:27.004173 7200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:59:27.005275 7200 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:27.013980 7206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:27.014791 7208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:27.016047 7205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:27.017040 7200 server_base.cc:1034] running on GCE node
I20240503 01:59:27.017375 7200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:59:27.017920 7200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:59:27.019075 7200 hybrid_clock.cc:648] HybridClock initialized: now 1714701567019058 us; error 31 us; skew 500 ppm
I20240503 01:59:27.020895 7200 webserver.cc:460] Webserver started at http://127.0.99.136:43353/ using document root <none> and password file <none>
I20240503 01:59:27.021415 7200 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:27.021466 7200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:27.021641 7200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:59:27.023252 7200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "7a1404fe1c1b4b388b821c8261b53104"
format_stamp: "Formatted at 2024-05-03 01:59:27 on dist-test-slave-dtz3"
I20240503 01:59:27.023699 7200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "7a1404fe1c1b4b388b821c8261b53104"
format_stamp: "Formatted at 2024-05-03 01:59:27 on dist-test-slave-dtz3"
I20240503 01:59:27.026865 7200 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 01:59:27.028991 7214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:27.029930 7200 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:27.030050 7200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "7a1404fe1c1b4b388b821c8261b53104"
format_stamp: "Formatted at 2024-05-03 01:59:27 on dist-test-slave-dtz3"
I20240503 01:59:27.030134 7200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:27.060398 7200 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:27.060964 7200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:59:27.061110 7200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:27.061601 7200 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:27.062443 7200 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:59:27.062508 7200 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:27.062558 7200 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:59:27.062580 7200 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:27.071027 7200 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.136:46759
I20240503 01:59:27.071069 7327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.136:46759 every 8 connection(s)
I20240503 01:59:27.072039 7200 server_base.cc:1166] Dumped server information to /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 01:59:27.077894 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task42rE7p/build/debug/bin/kudu as pid 7200
I20240503 01:59:27.078024 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240503 01:59:27.080948 7328 heartbeater.cc:347] Connected to a master server at 127.0.99.190:38597
I20240503 01:59:27.081130 7328 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:27.081614 7328 heartbeater.cc:511] Master 127.0.99.190:38597 requested a full tablet report, sending...
I20240503 01:59:27.082576 6223 ts_manager.cc:194] Registered new tserver with Master: 7a1404fe1c1b4b388b821c8261b53104 (127.0.99.136:46759)
I20240503 01:59:27.083177 6223 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.136:38503
I20240503 01:59:27.092288 398 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240503 01:59:27.101619 398 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240503 01:59:27.101691 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task42rE7p/build/debug/bin/kudu with pid 6807
I20240503 01:59:27.107053 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task42rE7p/build/debug/bin/kudu with pid 6938
I20240503 01:59:27.112396 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task42rE7p/build/debug/bin/kudu with pid 7069
I20240503 01:59:27.117367 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task42rE7p/build/debug/bin/kudu with pid 7200
I20240503 01:59:27.152235 6673 heartbeater.cc:503] Master 127.0.99.190:38597 was elected leader, sending a full tablet report...
I20240503 01:59:27.353114 6804 heartbeater.cc:503] Master 127.0.99.190:38597 was elected leader, sending a full tablet report...
I20240503 01:59:30.874174 6223 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44084:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:59:30.898180 6476 tablet_service.cc:1467] Processing CreateTablet for tablet ff8f06c5ad8442ee942bf900212fe858 (DEFAULT_TABLE table=rebalance_test_table_0 [id=2e2540d8e10e43848a2e486f767d1576]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:30.898172 6607 tablet_service.cc:1467] Processing CreateTablet for tablet a655d914858447b59fe5763e43c0807d (DEFAULT_TABLE table=rebalance_test_table_0 [id=2e2540d8e10e43848a2e486f767d1576]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:30.898274 6738 tablet_service.cc:1467] Processing CreateTablet for tablet a655d914858447b59fe5763e43c0807d (DEFAULT_TABLE table=rebalance_test_table_0 [id=2e2540d8e10e43848a2e486f767d1576]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:30.898383 6343 tablet_service.cc:1467] Processing CreateTablet for tablet e5ed3e729bc64ebeac216ce5076b9fd9 (DEFAULT_TABLE table=rebalance_test_table_0 [id=2e2540d8e10e43848a2e486f767d1576]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:30.899312 6343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5ed3e729bc64ebeac216ce5076b9fd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:30.899312 6607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a655d914858447b59fe5763e43c0807d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:30.899313 6738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a655d914858447b59fe5763e43c0807d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:30.899070 6475 tablet_service.cc:1467] Processing CreateTablet for tablet e5ed3e729bc64ebeac216ce5076b9fd9 (DEFAULT_TABLE table=rebalance_test_table_0 [id=2e2540d8e10e43848a2e486f767d1576]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:30.899914 6476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff8f06c5ad8442ee942bf900212fe858. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:30.900341 6475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5ed3e729bc64ebeac216ce5076b9fd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:30.900916 6344 tablet_service.cc:1467] Processing CreateTablet for tablet ff8f06c5ad8442ee942bf900212fe858 (DEFAULT_TABLE table=rebalance_test_table_0 [id=2e2540d8e10e43848a2e486f767d1576]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:30.901124 6344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff8f06c5ad8442ee942bf900212fe858. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:30.904912 6345 tablet_service.cc:1467] Processing CreateTablet for tablet a655d914858447b59fe5763e43c0807d (DEFAULT_TABLE table=rebalance_test_table_0 [id=2e2540d8e10e43848a2e486f767d1576]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:30.905148 6345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a655d914858447b59fe5763e43c0807d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:30.907258 6737 tablet_service.cc:1467] Processing CreateTablet for tablet e5ed3e729bc64ebeac216ce5076b9fd9 (DEFAULT_TABLE table=rebalance_test_table_0 [id=2e2540d8e10e43848a2e486f767d1576]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:30.907438 7348 tablet_bootstrap.cc:492] T a655d914858447b59fe5763e43c0807d P 79163c69f6f04a9abd0208d74e1c61be: Bootstrap starting.
I20240503 01:59:30.907547 6737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5ed3e729bc64ebeac216ce5076b9fd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:30.907678 7350 tablet_bootstrap.cc:492] T e5ed3e729bc64ebeac216ce5076b9fd9 P 0c4821d9898d42799c5789d4d2da3f63: Bootstrap starting.
I20240503 01:59:30.908862 6606 tablet_service.cc:1467] Processing CreateTablet for tablet ff8f06c5ad8442ee942bf900212fe858 (DEFAULT_TABLE table=rebalance_test_table_0 [id=2e2540d8e10e43848a2e486f767d1576]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:30.909140 6606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff8f06c5ad8442ee942bf900212fe858. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:30.909849 7348 tablet_bootstrap.cc:654] T a655d914858447b59fe5763e43c0807d P 79163c69f6f04a9abd0208d74e1c61be: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:30.909850 7350 tablet_bootstrap.cc:654] T e5ed3e729bc64ebeac216ce5076b9fd9 P 0c4821d9898d42799c5789d4d2da3f63: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:30.909972 7349 tablet_bootstrap.cc:492] T a655d914858447b59fe5763e43c0807d P b6b1d5677b8d47788b7968f64b3dac3c: Bootstrap starting.
I20240503 01:59:30.910732 7348 log.cc:824] T a655d914858447b59fe5763e43c0807d P 79163c69f6f04a9abd0208d74e1c61be: Log is configured to *not* fsync() on all Append() calls
I20240503 01:59:30.911473 7350 log.cc:824] T e5ed3e729bc64ebeac216ce5076b9fd9 P 0c4821d9898d42799c5789d4d2da3f63: Log is configured to *not* fsync() on all Append() calls
I20240503 01:59:30.911823 7349 tablet_bootstrap.cc:654] T a655d914858447b59fe5763e43c0807d P b6b1d5677b8d47788b7968f64b3dac3c: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:30.912698 7349 log.cc:824] T a655d914858447b59fe5763e43c0807d P b6b1d5677b8d47788b7968f64b3dac3c: Log is configured to *not* fsync() on all Append() calls
I20240503 01:59:30.912909 7348 tablet_bootstrap.cc:492] T a655d914858447b59fe5763e43c0807d P 79163c69f6f04a9abd0208d74e1c61be: No bootstrap required, opened a new log
I20240503 01:59:30.913095 7348 ts_tablet_manager.cc:1367] T a655d914858447b59fe5763e43c0807d P 79163c69f6f04a9abd0208d74e1c61be: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20240503 01:59:30.913132 7350 tablet_bootstrap.cc:492] T e5ed3e729bc64ebeac216ce5076b9fd9 P 0c4821d9898d42799c5789d4d2da3f63: No bootstrap required, opened a new log
I20240503 01:59:30.913286 7350 ts_tablet_manager.cc:1367] T e5ed3e729bc64ebeac216ce5076b9fd9 P 0c4821d9898d42799c5789d4d2da3f63: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240503 01:59:30.914264 7349 tablet_bootstrap.cc:492] T a655d914858447b59fe5763e43c0807d P b6b1d5677b8d47788b7968f64b3dac3c: No bootstrap required, opened a new log
I20240503 01:59:30.914402 7349 ts_tablet_manager.cc:1367] T a655d914858447b59fe5763e43c0807d P b6b1d5677b8d47788b7968f64b3dac3c: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 01:59:30.915161 7354 tablet_bootstrap.cc:492] T a655d914858447b59fe5763e43c0807d P 4127249ba42644388d225d614c935a25: Bootstrap starting.
I20240503 01:59:30.917260 7354 tablet_bootstrap.cc:654] T a655d914858447b59fe5763e43c0807d P 4127249ba42644388d225d614c935a25: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:30.917323 7348 raft_consensus.cc:348] T a655d914858447b59fe5763e43c0807d P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:30.917542 7348 raft_consensus.cc:374] T a655d914858447b59fe5763e43c0807d P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:30.917603 7348 raft_consensus.cc:729] T a655d914858447b59fe5763e43c0807d P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79163c69f6f04a9abd0208d74e1c61be, State: Initialized, Role: FOLLOWER
I20240503 01:59:30.918035 7354 log.cc:824] T a655d914858447b59fe5763e43c0807d P 4127249ba42644388d225d614c935a25: Log is configured to *not* fsync() on all Append() calls
I20240503 01:59:30.917270 7350 raft_consensus.cc:348] T e5ed3e729bc64ebeac216ce5076b9fd9 P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:30.918282 7349 raft_consensus.cc:348] T a655d914858447b59fe5763e43c0807d P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:30.918449 7350 raft_consensus.cc:374] T e5ed3e729bc64ebeac216ce5076b9fd9 P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:30.918385 7348 consensus_queue.cc:260] T a655d914858447b59fe5763e43c0807d P 79163c69f6f04a9abd0208d74e1c61be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:30.918545 7350 raft_consensus.cc:729] T e5ed3e729bc64ebeac216ce5076b9fd9 P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c4821d9898d42799c5789d4d2da3f63, State: Initialized, Role: FOLLOWER
I20240503 01:59:30.919008 7350 consensus_queue.cc:260] T e5ed3e729bc64ebeac216ce5076b9fd9 P 0c4821d9898d42799c5789d4d2da3f63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:30.919121 7349 raft_consensus.cc:374] T a655d914858447b59fe5763e43c0807d P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:30.919209 7349 raft_consensus.cc:729] T a655d914858447b59fe5763e43c0807d P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6b1d5677b8d47788b7968f64b3dac3c, State: Initialized, Role: FOLLOWER
I20240503 01:59:30.919335 7348 ts_tablet_manager.cc:1398] T a655d914858447b59fe5763e43c0807d P 79163c69f6f04a9abd0208d74e1c61be: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240503 01:59:30.919695 7349 consensus_queue.cc:260] T a655d914858447b59fe5763e43c0807d P b6b1d5677b8d47788b7968f64b3dac3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:30.919781 7354 tablet_bootstrap.cc:492] T a655d914858447b59fe5763e43c0807d P 4127249ba42644388d225d614c935a25: No bootstrap required, opened a new log
I20240503 01:59:30.919900 7350 ts_tablet_manager.cc:1398] T e5ed3e729bc64ebeac216ce5076b9fd9 P 0c4821d9898d42799c5789d4d2da3f63: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20240503 01:59:30.919929 7354 ts_tablet_manager.cc:1367] T a655d914858447b59fe5763e43c0807d P 4127249ba42644388d225d614c935a25: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 01:59:30.920303 7350 tablet_bootstrap.cc:492] T ff8f06c5ad8442ee942bf900212fe858 P 0c4821d9898d42799c5789d4d2da3f63: Bootstrap starting.
I20240503 01:59:30.920531 7349 ts_tablet_manager.cc:1398] T a655d914858447b59fe5763e43c0807d P b6b1d5677b8d47788b7968f64b3dac3c: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240503 01:59:30.921355 7350 tablet_bootstrap.cc:654] T ff8f06c5ad8442ee942bf900212fe858 P 0c4821d9898d42799c5789d4d2da3f63: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:30.922112 7349 tablet_bootstrap.cc:492] T e5ed3e729bc64ebeac216ce5076b9fd9 P b6b1d5677b8d47788b7968f64b3dac3c: Bootstrap starting.
I20240503 01:59:30.922616 7350 tablet_bootstrap.cc:492] T ff8f06c5ad8442ee942bf900212fe858 P 0c4821d9898d42799c5789d4d2da3f63: No bootstrap required, opened a new log
I20240503 01:59:30.922745 7350 ts_tablet_manager.cc:1367] T ff8f06c5ad8442ee942bf900212fe858 P 0c4821d9898d42799c5789d4d2da3f63: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:30.923125 7349 tablet_bootstrap.cc:654] T e5ed3e729bc64ebeac216ce5076b9fd9 P b6b1d5677b8d47788b7968f64b3dac3c: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:30.923372 7350 raft_consensus.cc:348] T ff8f06c5ad8442ee942bf900212fe858 P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:30.923542 7350 raft_consensus.cc:374] T ff8f06c5ad8442ee942bf900212fe858 P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:30.923622 7350 raft_consensus.cc:729] T ff8f06c5ad8442ee942bf900212fe858 P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c4821d9898d42799c5789d4d2da3f63, State: Initialized, Role: FOLLOWER
I20240503 01:59:30.923835 7350 consensus_queue.cc:260] T ff8f06c5ad8442ee942bf900212fe858 P 0c4821d9898d42799c5789d4d2da3f63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:30.924008 7354 raft_consensus.cc:348] T a655d914858447b59fe5763e43c0807d P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:30.924196 7354 raft_consensus.cc:374] T a655d914858447b59fe5763e43c0807d P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:30.924257 7354 raft_consensus.cc:729] T a655d914858447b59fe5763e43c0807d P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4127249ba42644388d225d614c935a25, State: Initialized, Role: FOLLOWER
I20240503 01:59:30.924350 7349 tablet_bootstrap.cc:492] T e5ed3e729bc64ebeac216ce5076b9fd9 P b6b1d5677b8d47788b7968f64b3dac3c: No bootstrap required, opened a new log
I20240503 01:59:30.924573 7349 ts_tablet_manager.cc:1367] T e5ed3e729bc64ebeac216ce5076b9fd9 P b6b1d5677b8d47788b7968f64b3dac3c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:30.924579 7350 ts_tablet_manager.cc:1398] T ff8f06c5ad8442ee942bf900212fe858 P 0c4821d9898d42799c5789d4d2da3f63: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:30.924716 7354 consensus_queue.cc:260] T a655d914858447b59fe5763e43c0807d P 4127249ba42644388d225d614c935a25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:30.925375 7348 tablet_bootstrap.cc:492] T ff8f06c5ad8442ee942bf900212fe858 P 79163c69f6f04a9abd0208d74e1c61be: Bootstrap starting.
I20240503 01:59:30.925472 7349 raft_consensus.cc:348] T e5ed3e729bc64ebeac216ce5076b9fd9 P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:30.925648 7349 raft_consensus.cc:374] T e5ed3e729bc64ebeac216ce5076b9fd9 P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:30.925678 7354 ts_tablet_manager.cc:1398] T a655d914858447b59fe5763e43c0807d P 4127249ba42644388d225d614c935a25: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20240503 01:59:30.925757 7349 raft_consensus.cc:729] T e5ed3e729bc64ebeac216ce5076b9fd9 P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6b1d5677b8d47788b7968f64b3dac3c, State: Initialized, Role: FOLLOWER
I20240503 01:59:30.925964 7349 consensus_queue.cc:260] T e5ed3e729bc64ebeac216ce5076b9fd9 P b6b1d5677b8d47788b7968f64b3dac3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:30.926398 7349 ts_tablet_manager.cc:1398] T e5ed3e729bc64ebeac216ce5076b9fd9 P b6b1d5677b8d47788b7968f64b3dac3c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:30.926860 7348 tablet_bootstrap.cc:654] T ff8f06c5ad8442ee942bf900212fe858 P 79163c69f6f04a9abd0208d74e1c61be: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:30.928772 7348 tablet_bootstrap.cc:492] T ff8f06c5ad8442ee942bf900212fe858 P 79163c69f6f04a9abd0208d74e1c61be: No bootstrap required, opened a new log
I20240503 01:59:30.928903 7348 ts_tablet_manager.cc:1367] T ff8f06c5ad8442ee942bf900212fe858 P 79163c69f6f04a9abd0208d74e1c61be: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20240503 01:59:30.928962 7354 tablet_bootstrap.cc:492] T ff8f06c5ad8442ee942bf900212fe858 P 4127249ba42644388d225d614c935a25: Bootstrap starting.
I20240503 01:59:30.929658 7348 raft_consensus.cc:348] T ff8f06c5ad8442ee942bf900212fe858 P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:30.929800 7348 raft_consensus.cc:374] T ff8f06c5ad8442ee942bf900212fe858 P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:30.929858 7348 raft_consensus.cc:729] T ff8f06c5ad8442ee942bf900212fe858 P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79163c69f6f04a9abd0208d74e1c61be, State: Initialized, Role: FOLLOWER
I20240503 01:59:30.929988 7348 consensus_queue.cc:260] T ff8f06c5ad8442ee942bf900212fe858 P 79163c69f6f04a9abd0208d74e1c61be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:30.930050 7354 tablet_bootstrap.cc:654] T ff8f06c5ad8442ee942bf900212fe858 P 4127249ba42644388d225d614c935a25: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:30.930330 7348 ts_tablet_manager.cc:1398] T ff8f06c5ad8442ee942bf900212fe858 P 79163c69f6f04a9abd0208d74e1c61be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:30.931672 7354 tablet_bootstrap.cc:492] T ff8f06c5ad8442ee942bf900212fe858 P 4127249ba42644388d225d614c935a25: No bootstrap required, opened a new log
I20240503 01:59:30.931761 7354 ts_tablet_manager.cc:1367] T ff8f06c5ad8442ee942bf900212fe858 P 4127249ba42644388d225d614c935a25: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:30.932225 7354 raft_consensus.cc:348] T ff8f06c5ad8442ee942bf900212fe858 P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:30.932320 7354 raft_consensus.cc:374] T ff8f06c5ad8442ee942bf900212fe858 P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:30.932343 7354 raft_consensus.cc:729] T ff8f06c5ad8442ee942bf900212fe858 P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4127249ba42644388d225d614c935a25, State: Initialized, Role: FOLLOWER
I20240503 01:59:30.932458 7354 consensus_queue.cc:260] T ff8f06c5ad8442ee942bf900212fe858 P 4127249ba42644388d225d614c935a25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:30.932786 7354 ts_tablet_manager.cc:1398] T ff8f06c5ad8442ee942bf900212fe858 P 4127249ba42644388d225d614c935a25: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:30.932996 7354 tablet_bootstrap.cc:492] T e5ed3e729bc64ebeac216ce5076b9fd9 P 4127249ba42644388d225d614c935a25: Bootstrap starting.
I20240503 01:59:30.933728 7354 tablet_bootstrap.cc:654] T e5ed3e729bc64ebeac216ce5076b9fd9 P 4127249ba42644388d225d614c935a25: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:30.934954 7354 tablet_bootstrap.cc:492] T e5ed3e729bc64ebeac216ce5076b9fd9 P 4127249ba42644388d225d614c935a25: No bootstrap required, opened a new log
I20240503 01:59:30.935042 7354 ts_tablet_manager.cc:1367] T e5ed3e729bc64ebeac216ce5076b9fd9 P 4127249ba42644388d225d614c935a25: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:30.935444 7354 raft_consensus.cc:348] T e5ed3e729bc64ebeac216ce5076b9fd9 P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:30.935528 7354 raft_consensus.cc:374] T e5ed3e729bc64ebeac216ce5076b9fd9 P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:30.935549 7354 raft_consensus.cc:729] T e5ed3e729bc64ebeac216ce5076b9fd9 P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4127249ba42644388d225d614c935a25, State: Initialized, Role: FOLLOWER
I20240503 01:59:30.935649 7354 consensus_queue.cc:260] T e5ed3e729bc64ebeac216ce5076b9fd9 P 4127249ba42644388d225d614c935a25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:30.935907 7354 ts_tablet_manager.cc:1398] T e5ed3e729bc64ebeac216ce5076b9fd9 P 4127249ba42644388d225d614c935a25: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:30.940322 7361 raft_consensus.cc:482] T ff8f06c5ad8442ee942bf900212fe858 P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:30.940511 7361 raft_consensus.cc:504] T ff8f06c5ad8442ee942bf900212fe858 P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:30.941784 7361 leader_election.cc:288] T ff8f06c5ad8442ee942bf900212fe858 P 0c4821d9898d42799c5789d4d2da3f63 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131:44005), 4127249ba42644388d225d614c935a25 (127.0.99.129:35613)
I20240503 01:59:30.946395 6365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff8f06c5ad8442ee942bf900212fe858" candidate_uuid: "0c4821d9898d42799c5789d4d2da3f63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4127249ba42644388d225d614c935a25" is_pre_election: true
I20240503 01:59:30.946393 6627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff8f06c5ad8442ee942bf900212fe858" candidate_uuid: "0c4821d9898d42799c5789d4d2da3f63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79163c69f6f04a9abd0208d74e1c61be" is_pre_election: true
I20240503 01:59:30.946727 6365 raft_consensus.cc:2454] T ff8f06c5ad8442ee942bf900212fe858 P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c4821d9898d42799c5789d4d2da3f63 in term 0.
I20240503 01:59:30.946720 6627 raft_consensus.cc:2454] T ff8f06c5ad8442ee942bf900212fe858 P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c4821d9898d42799c5789d4d2da3f63 in term 0.
I20240503 01:59:30.947201 6430 leader_election.cc:302] T ff8f06c5ad8442ee942bf900212fe858 P 0c4821d9898d42799c5789d4d2da3f63 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c4821d9898d42799c5789d4d2da3f63, 4127249ba42644388d225d614c935a25; no voters:
I20240503 01:59:30.947443 7361 raft_consensus.cc:2781] T ff8f06c5ad8442ee942bf900212fe858 P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:30.947521 7361 raft_consensus.cc:482] T ff8f06c5ad8442ee942bf900212fe858 P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:30.947575 7361 raft_consensus.cc:3037] T ff8f06c5ad8442ee942bf900212fe858 P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:30.948598 7361 raft_consensus.cc:504] T ff8f06c5ad8442ee942bf900212fe858 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:30.948963 7361 leader_election.cc:288] T ff8f06c5ad8442ee942bf900212fe858 P 0c4821d9898d42799c5789d4d2da3f63 [CANDIDATE]: Term 1 election: Requested vote from peers 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131:44005), 4127249ba42644388d225d614c935a25 (127.0.99.129:35613)
I20240503 01:59:30.949298 6627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff8f06c5ad8442ee942bf900212fe858" candidate_uuid: "0c4821d9898d42799c5789d4d2da3f63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79163c69f6f04a9abd0208d74e1c61be"
I20240503 01:59:30.949393 6365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff8f06c5ad8442ee942bf900212fe858" candidate_uuid: "0c4821d9898d42799c5789d4d2da3f63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4127249ba42644388d225d614c935a25"
I20240503 01:59:30.949451 6627 raft_consensus.cc:3037] T ff8f06c5ad8442ee942bf900212fe858 P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:30.949523 6365 raft_consensus.cc:3037] T ff8f06c5ad8442ee942bf900212fe858 P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:30.950788 6627 raft_consensus.cc:2454] T ff8f06c5ad8442ee942bf900212fe858 P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c4821d9898d42799c5789d4d2da3f63 in term 1.
I20240503 01:59:30.951078 6430 leader_election.cc:302] T ff8f06c5ad8442ee942bf900212fe858 P 0c4821d9898d42799c5789d4d2da3f63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c4821d9898d42799c5789d4d2da3f63, 79163c69f6f04a9abd0208d74e1c61be; no voters:
I20240503 01:59:30.951187 6365 raft_consensus.cc:2454] T ff8f06c5ad8442ee942bf900212fe858 P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c4821d9898d42799c5789d4d2da3f63 in term 1.
I20240503 01:59:30.951282 7361 raft_consensus.cc:2781] T ff8f06c5ad8442ee942bf900212fe858 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:30.951380 7361 raft_consensus.cc:686] T ff8f06c5ad8442ee942bf900212fe858 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 LEADER]: Becoming Leader. State: Replica: 0c4821d9898d42799c5789d4d2da3f63, State: Running, Role: LEADER
I20240503 01:59:30.951754 7361 consensus_queue.cc:237] T ff8f06c5ad8442ee942bf900212fe858 P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:30.955446 6221 catalog_manager.cc:5484] T ff8f06c5ad8442ee942bf900212fe858 P 0c4821d9898d42799c5789d4d2da3f63 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "0c4821d9898d42799c5789d4d2da3f63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } health_report { overall_health: UNKNOWN } } }
W20240503 01:59:30.955854 6543 tablet.cc:2369] T e5ed3e729bc64ebeac216ce5076b9fd9 P 0c4821d9898d42799c5789d4d2da3f63: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 01:59:30.959791 7361 raft_consensus.cc:482] T e5ed3e729bc64ebeac216ce5076b9fd9 P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:30.959867 7361 raft_consensus.cc:504] T e5ed3e729bc64ebeac216ce5076b9fd9 P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:30.960263 7361 leader_election.cc:288] T e5ed3e729bc64ebeac216ce5076b9fd9 P 0c4821d9898d42799c5789d4d2da3f63 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b6b1d5677b8d47788b7968f64b3dac3c (127.0.99.132:46067), 4127249ba42644388d225d614c935a25 (127.0.99.129:35613)
I20240503 01:59:30.960693 6365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ed3e729bc64ebeac216ce5076b9fd9" candidate_uuid: "0c4821d9898d42799c5789d4d2da3f63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4127249ba42644388d225d614c935a25" is_pre_election: true
I20240503 01:59:30.960837 6365 raft_consensus.cc:2454] T e5ed3e729bc64ebeac216ce5076b9fd9 P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c4821d9898d42799c5789d4d2da3f63 in term 0.
I20240503 01:59:30.961102 6430 leader_election.cc:302] T e5ed3e729bc64ebeac216ce5076b9fd9 P 0c4821d9898d42799c5789d4d2da3f63 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c4821d9898d42799c5789d4d2da3f63, 4127249ba42644388d225d614c935a25; no voters:
I20240503 01:59:30.961266 7361 raft_consensus.cc:2781] T e5ed3e729bc64ebeac216ce5076b9fd9 P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:30.961320 7361 raft_consensus.cc:482] T e5ed3e729bc64ebeac216ce5076b9fd9 P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:30.961354 7361 raft_consensus.cc:3037] T e5ed3e729bc64ebeac216ce5076b9fd9 P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:30.962131 7361 raft_consensus.cc:504] T e5ed3e729bc64ebeac216ce5076b9fd9 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:30.962450 7361 leader_election.cc:288] T e5ed3e729bc64ebeac216ce5076b9fd9 P 0c4821d9898d42799c5789d4d2da3f63 [CANDIDATE]: Term 1 election: Requested vote from peers b6b1d5677b8d47788b7968f64b3dac3c (127.0.99.132:46067), 4127249ba42644388d225d614c935a25 (127.0.99.129:35613)
I20240503 01:59:30.962747 6365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ed3e729bc64ebeac216ce5076b9fd9" candidate_uuid: "0c4821d9898d42799c5789d4d2da3f63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4127249ba42644388d225d614c935a25"
I20240503 01:59:30.962839 6365 raft_consensus.cc:3037] T e5ed3e729bc64ebeac216ce5076b9fd9 P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:30.963640 6365 raft_consensus.cc:2454] T e5ed3e729bc64ebeac216ce5076b9fd9 P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c4821d9898d42799c5789d4d2da3f63 in term 1.
I20240503 01:59:30.963877 6430 leader_election.cc:302] T e5ed3e729bc64ebeac216ce5076b9fd9 P 0c4821d9898d42799c5789d4d2da3f63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c4821d9898d42799c5789d4d2da3f63, 4127249ba42644388d225d614c935a25; no voters:
I20240503 01:59:30.964038 7361 raft_consensus.cc:2781] T e5ed3e729bc64ebeac216ce5076b9fd9 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:30.964182 7361 raft_consensus.cc:686] T e5ed3e729bc64ebeac216ce5076b9fd9 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 LEADER]: Becoming Leader. State: Replica: 0c4821d9898d42799c5789d4d2da3f63, State: Running, Role: LEADER
I20240503 01:59:30.964259 6757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ed3e729bc64ebeac216ce5076b9fd9" candidate_uuid: "0c4821d9898d42799c5789d4d2da3f63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6b1d5677b8d47788b7968f64b3dac3c"
I20240503 01:59:30.964325 7361 consensus_queue.cc:237] T e5ed3e729bc64ebeac216ce5076b9fd9 P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:30.964258 6758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ed3e729bc64ebeac216ce5076b9fd9" candidate_uuid: "0c4821d9898d42799c5789d4d2da3f63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" is_pre_election: true
I20240503 01:59:30.964504 6757 raft_consensus.cc:3037] T e5ed3e729bc64ebeac216ce5076b9fd9 P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:30.965837 6757 raft_consensus.cc:2454] T e5ed3e729bc64ebeac216ce5076b9fd9 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c4821d9898d42799c5789d4d2da3f63 in term 1.
I20240503 01:59:30.965853 6221 catalog_manager.cc:5484] T e5ed3e729bc64ebeac216ce5076b9fd9 P 0c4821d9898d42799c5789d4d2da3f63 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "0c4821d9898d42799c5789d4d2da3f63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } health_report { overall_health: UNKNOWN } } }
W20240503 01:59:31.043931 6412 tablet.cc:2369] T a655d914858447b59fe5763e43c0807d P 4127249ba42644388d225d614c935a25: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 01:59:31.095098 6805 tablet.cc:2369] T a655d914858447b59fe5763e43c0807d P b6b1d5677b8d47788b7968f64b3dac3c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 01:59:31.144723 6674 tablet.cc:2369] T a655d914858447b59fe5763e43c0807d P 79163c69f6f04a9abd0208d74e1c61be: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 01:59:31.283102 7356 raft_consensus.cc:482] T a655d914858447b59fe5763e43c0807d P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:31.283349 7356 raft_consensus.cc:504] T a655d914858447b59fe5763e43c0807d P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:31.284929 7356 leader_election.cc:288] T a655d914858447b59fe5763e43c0807d P 79163c69f6f04a9abd0208d74e1c61be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b6b1d5677b8d47788b7968f64b3dac3c (127.0.99.132:46067), 4127249ba42644388d225d614c935a25 (127.0.99.129:35613)
I20240503 01:59:31.290340 6365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a655d914858447b59fe5763e43c0807d" candidate_uuid: "79163c69f6f04a9abd0208d74e1c61be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4127249ba42644388d225d614c935a25" is_pre_election: true
I20240503 01:59:31.290561 6365 raft_consensus.cc:2454] T a655d914858447b59fe5763e43c0807d P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79163c69f6f04a9abd0208d74e1c61be in term 0.
I20240503 01:59:31.290555 6757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a655d914858447b59fe5763e43c0807d" candidate_uuid: "79163c69f6f04a9abd0208d74e1c61be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" is_pre_election: true
I20240503 01:59:31.290702 6757 raft_consensus.cc:2454] T a655d914858447b59fe5763e43c0807d P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79163c69f6f04a9abd0208d74e1c61be in term 0.
I20240503 01:59:31.291369 6561 leader_election.cc:302] T a655d914858447b59fe5763e43c0807d P 79163c69f6f04a9abd0208d74e1c61be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4127249ba42644388d225d614c935a25, 79163c69f6f04a9abd0208d74e1c61be; no voters:
I20240503 01:59:31.291697 7356 raft_consensus.cc:2781] T a655d914858447b59fe5763e43c0807d P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:31.291774 7356 raft_consensus.cc:482] T a655d914858447b59fe5763e43c0807d P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:31.291819 7356 raft_consensus.cc:3037] T a655d914858447b59fe5763e43c0807d P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:31.292819 7356 raft_consensus.cc:504] T a655d914858447b59fe5763e43c0807d P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:31.293228 7356 leader_election.cc:288] T a655d914858447b59fe5763e43c0807d P 79163c69f6f04a9abd0208d74e1c61be [CANDIDATE]: Term 1 election: Requested vote from peers b6b1d5677b8d47788b7968f64b3dac3c (127.0.99.132:46067), 4127249ba42644388d225d614c935a25 (127.0.99.129:35613)
I20240503 01:59:31.293618 6365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a655d914858447b59fe5763e43c0807d" candidate_uuid: "79163c69f6f04a9abd0208d74e1c61be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4127249ba42644388d225d614c935a25"
I20240503 01:59:31.293717 6365 raft_consensus.cc:3037] T a655d914858447b59fe5763e43c0807d P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:31.294545 6365 raft_consensus.cc:2454] T a655d914858447b59fe5763e43c0807d P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79163c69f6f04a9abd0208d74e1c61be in term 1.
I20240503 01:59:31.294757 6757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a655d914858447b59fe5763e43c0807d" candidate_uuid: "79163c69f6f04a9abd0208d74e1c61be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6b1d5677b8d47788b7968f64b3dac3c"
I20240503 01:59:31.294782 6561 leader_election.cc:302] T a655d914858447b59fe5763e43c0807d P 79163c69f6f04a9abd0208d74e1c61be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4127249ba42644388d225d614c935a25, 79163c69f6f04a9abd0208d74e1c61be; no voters:
I20240503 01:59:31.294867 6757 raft_consensus.cc:3037] T a655d914858447b59fe5763e43c0807d P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:31.295742 6757 raft_consensus.cc:2454] T a655d914858447b59fe5763e43c0807d P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79163c69f6f04a9abd0208d74e1c61be in term 1.
I20240503 01:59:31.295944 7356 raft_consensus.cc:2781] T a655d914858447b59fe5763e43c0807d P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:31.296190 7356 raft_consensus.cc:686] T a655d914858447b59fe5763e43c0807d P 79163c69f6f04a9abd0208d74e1c61be [term 1 LEADER]: Becoming Leader. State: Replica: 79163c69f6f04a9abd0208d74e1c61be, State: Running, Role: LEADER
I20240503 01:59:31.296531 7356 consensus_queue.cc:237] T a655d914858447b59fe5763e43c0807d P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:31.299827 6221 catalog_manager.cc:5484] T a655d914858447b59fe5763e43c0807d P 79163c69f6f04a9abd0208d74e1c61be reported cstate change: term changed from 0 to 1, leader changed from <none> to 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131). New cstate: current_term: 1 leader_uuid: "79163c69f6f04a9abd0208d74e1c61be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:31.348232 7361 consensus_queue.cc:1035] T ff8f06c5ad8442ee942bf900212fe858 P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:31.353804 7378 consensus_queue.cc:1035] T ff8f06c5ad8442ee942bf900212fe858 P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:31.375517 7378 consensus_queue.cc:1035] T e5ed3e729bc64ebeac216ce5076b9fd9 P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:31.379161 7357 consensus_queue.cc:1035] T e5ed3e729bc64ebeac216ce5076b9fd9 P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 01:59:31.449717 7375 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:31.449919 7375 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:31.452869 7375 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240503 01:59:31.547070 6364 raft_consensus.cc:1261] T a655d914858447b59fe5763e43c0807d P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:31.550102 6757 raft_consensus.cc:1261] T a655d914858447b59fe5763e43c0807d P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:31.550617 7356 consensus_queue.cc:1035] T a655d914858447b59fe5763e43c0807d P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:31.550860 7373 consensus_queue.cc:1035] T a655d914858447b59fe5763e43c0807d P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:31.579372 7384 mvcc.cc:205] Tried to move back new op lower bound from 7023417637049995264 to 7023417636033155072. Current Snapshot: MvccSnapshot[applied={T|T < 7023417637049995264}]
I20240503 01:59:31.588714 7381 mvcc.cc:205] Tried to move back new op lower bound from 7023417637049995264 to 7023417636033155072. Current Snapshot: MvccSnapshot[applied={T|T < 7023417637049995264}]
I20240503 01:59:31.597421 7379 mvcc.cc:205] Tried to move back new op lower bound from 7023417637151350784 to 7023417636033155072. Current Snapshot: MvccSnapshot[applied={T|T < 7023417637049995264}]
INSERT report
rows total: 2000
time total: 124.797 ms
time per row: 0.0623987 ms
I20240503 01:59:31.620199 6217 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44084:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:59:31.625838 6606 tablet_service.cc:1467] Processing CreateTablet for tablet d6ee8175874c4063986033b0a80ccb82 (DEFAULT_TABLE table=rebalance_test_table_1 [id=91bdb83bc45b4b61b094a0180eacf8e3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:31.626154 6606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6ee8175874c4063986033b0a80ccb82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:31.626092 6737 tablet_service.cc:1467] Processing CreateTablet for tablet d6ee8175874c4063986033b0a80ccb82 (DEFAULT_TABLE table=rebalance_test_table_1 [id=91bdb83bc45b4b61b094a0180eacf8e3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:31.626379 6737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6ee8175874c4063986033b0a80ccb82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:31.626945 6607 tablet_service.cc:1467] Processing CreateTablet for tablet e5a00d4ce3a64ca5a7c5bd65c556f429 (DEFAULT_TABLE table=rebalance_test_table_1 [id=91bdb83bc45b4b61b094a0180eacf8e3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:31.626953 6605 tablet_service.cc:1467] Processing CreateTablet for tablet a388d59f6ba94e91958f55b340343fa6 (DEFAULT_TABLE table=rebalance_test_table_1 [id=91bdb83bc45b4b61b094a0180eacf8e3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:31.627171 6607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5a00d4ce3a64ca5a7c5bd65c556f429. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:31.627305 6605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a388d59f6ba94e91958f55b340343fa6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:31.628727 6475 tablet_service.cc:1467] Processing CreateTablet for tablet d6ee8175874c4063986033b0a80ccb82 (DEFAULT_TABLE table=rebalance_test_table_1 [id=91bdb83bc45b4b61b094a0180eacf8e3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:31.629010 6475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6ee8175874c4063986033b0a80ccb82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:31.629428 7433 tablet_bootstrap.cc:492] T d6ee8175874c4063986033b0a80ccb82 P b6b1d5677b8d47788b7968f64b3dac3c: Bootstrap starting.
I20240503 01:59:31.629741 6737 tablet_service.cc:1467] Processing CreateTablet for tablet e5a00d4ce3a64ca5a7c5bd65c556f429 (DEFAULT_TABLE table=rebalance_test_table_1 [id=91bdb83bc45b4b61b094a0180eacf8e3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:31.629937 6737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5a00d4ce3a64ca5a7c5bd65c556f429. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:31.630288 7433 tablet_bootstrap.cc:654] T d6ee8175874c4063986033b0a80ccb82 P b6b1d5677b8d47788b7968f64b3dac3c: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:31.630899 7434 tablet_bootstrap.cc:492] T d6ee8175874c4063986033b0a80ccb82 P 0c4821d9898d42799c5789d4d2da3f63: Bootstrap starting.
I20240503 01:59:31.631042 6476 tablet_service.cc:1467] Processing CreateTablet for tablet e5a00d4ce3a64ca5a7c5bd65c556f429 (DEFAULT_TABLE table=rebalance_test_table_1 [id=91bdb83bc45b4b61b094a0180eacf8e3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:31.631242 6476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5a00d4ce3a64ca5a7c5bd65c556f429. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:31.631923 7434 tablet_bootstrap.cc:654] T d6ee8175874c4063986033b0a80ccb82 P 0c4821d9898d42799c5789d4d2da3f63: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:31.632200 7433 tablet_bootstrap.cc:492] T d6ee8175874c4063986033b0a80ccb82 P b6b1d5677b8d47788b7968f64b3dac3c: No bootstrap required, opened a new log
I20240503 01:59:31.632292 7433 ts_tablet_manager.cc:1367] T d6ee8175874c4063986033b0a80ccb82 P b6b1d5677b8d47788b7968f64b3dac3c: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 01:59:31.632675 7433 raft_consensus.cc:348] T d6ee8175874c4063986033b0a80ccb82 P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } }
I20240503 01:59:31.632792 7433 raft_consensus.cc:374] T d6ee8175874c4063986033b0a80ccb82 P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:31.632830 7433 raft_consensus.cc:729] T d6ee8175874c4063986033b0a80ccb82 P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6b1d5677b8d47788b7968f64b3dac3c, State: Initialized, Role: FOLLOWER
I20240503 01:59:31.632961 7433 consensus_queue.cc:260] T d6ee8175874c4063986033b0a80ccb82 P b6b1d5677b8d47788b7968f64b3dac3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } }
I20240503 01:59:31.633189 7434 tablet_bootstrap.cc:492] T d6ee8175874c4063986033b0a80ccb82 P 0c4821d9898d42799c5789d4d2da3f63: No bootstrap required, opened a new log
I20240503 01:59:31.633201 7433 ts_tablet_manager.cc:1398] T d6ee8175874c4063986033b0a80ccb82 P b6b1d5677b8d47788b7968f64b3dac3c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:31.633286 7434 ts_tablet_manager.cc:1367] T d6ee8175874c4063986033b0a80ccb82 P 0c4821d9898d42799c5789d4d2da3f63: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:31.633325 7433 tablet_bootstrap.cc:492] T e5a00d4ce3a64ca5a7c5bd65c556f429 P b6b1d5677b8d47788b7968f64b3dac3c: Bootstrap starting.
I20240503 01:59:31.633275 6474 tablet_service.cc:1467] Processing CreateTablet for tablet a388d59f6ba94e91958f55b340343fa6 (DEFAULT_TABLE table=rebalance_test_table_1 [id=91bdb83bc45b4b61b094a0180eacf8e3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:31.633519 6474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a388d59f6ba94e91958f55b340343fa6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:31.633772 7434 raft_consensus.cc:348] T d6ee8175874c4063986033b0a80ccb82 P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } }
I20240503 01:59:31.633858 7434 raft_consensus.cc:374] T d6ee8175874c4063986033b0a80ccb82 P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:31.633893 7434 raft_consensus.cc:729] T d6ee8175874c4063986033b0a80ccb82 P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c4821d9898d42799c5789d4d2da3f63, State: Initialized, Role: FOLLOWER
I20240503 01:59:31.634037 7434 consensus_queue.cc:260] T d6ee8175874c4063986033b0a80ccb82 P 0c4821d9898d42799c5789d4d2da3f63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } }
I20240503 01:59:31.634164 7433 tablet_bootstrap.cc:654] T e5a00d4ce3a64ca5a7c5bd65c556f429 P b6b1d5677b8d47788b7968f64b3dac3c: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:31.634713 7434 ts_tablet_manager.cc:1398] T d6ee8175874c4063986033b0a80ccb82 P 0c4821d9898d42799c5789d4d2da3f63: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:31.634862 7434 tablet_bootstrap.cc:492] T e5a00d4ce3a64ca5a7c5bd65c556f429 P 0c4821d9898d42799c5789d4d2da3f63: Bootstrap starting.
I20240503 01:59:31.635380 7433 tablet_bootstrap.cc:492] T e5a00d4ce3a64ca5a7c5bd65c556f429 P b6b1d5677b8d47788b7968f64b3dac3c: No bootstrap required, opened a new log
I20240503 01:59:31.635452 7433 ts_tablet_manager.cc:1367] T e5a00d4ce3a64ca5a7c5bd65c556f429 P b6b1d5677b8d47788b7968f64b3dac3c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:31.635612 6343 tablet_service.cc:1467] Processing CreateTablet for tablet a388d59f6ba94e91958f55b340343fa6 (DEFAULT_TABLE table=rebalance_test_table_1 [id=91bdb83bc45b4b61b094a0180eacf8e3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:31.635912 6343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a388d59f6ba94e91958f55b340343fa6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:31.635916 7433 raft_consensus.cc:348] T e5a00d4ce3a64ca5a7c5bd65c556f429 P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } }
I20240503 01:59:31.635987 7434 tablet_bootstrap.cc:654] T e5a00d4ce3a64ca5a7c5bd65c556f429 P 0c4821d9898d42799c5789d4d2da3f63: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:31.636003 7433 raft_consensus.cc:374] T e5a00d4ce3a64ca5a7c5bd65c556f429 P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:31.636039 7433 raft_consensus.cc:729] T e5a00d4ce3a64ca5a7c5bd65c556f429 P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6b1d5677b8d47788b7968f64b3dac3c, State: Initialized, Role: FOLLOWER
I20240503 01:59:31.636154 7433 consensus_queue.cc:260] T e5a00d4ce3a64ca5a7c5bd65c556f429 P b6b1d5677b8d47788b7968f64b3dac3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } }
I20240503 01:59:31.636410 7433 ts_tablet_manager.cc:1398] T e5a00d4ce3a64ca5a7c5bd65c556f429 P b6b1d5677b8d47788b7968f64b3dac3c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:31.637892 7434 tablet_bootstrap.cc:492] T e5a00d4ce3a64ca5a7c5bd65c556f429 P 0c4821d9898d42799c5789d4d2da3f63: No bootstrap required, opened a new log
I20240503 01:59:31.638005 7434 ts_tablet_manager.cc:1367] T e5a00d4ce3a64ca5a7c5bd65c556f429 P 0c4821d9898d42799c5789d4d2da3f63: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:31.638214 7439 tablet_bootstrap.cc:492] T a388d59f6ba94e91958f55b340343fa6 P 4127249ba42644388d225d614c935a25: Bootstrap starting.
I20240503 01:59:31.638660 7432 tablet_bootstrap.cc:492] T d6ee8175874c4063986033b0a80ccb82 P 79163c69f6f04a9abd0208d74e1c61be: Bootstrap starting.
I20240503 01:59:31.638659 7434 raft_consensus.cc:348] T e5a00d4ce3a64ca5a7c5bd65c556f429 P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } }
I20240503 01:59:31.638864 7434 raft_consensus.cc:374] T e5a00d4ce3a64ca5a7c5bd65c556f429 P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:31.638960 7434 raft_consensus.cc:729] T e5a00d4ce3a64ca5a7c5bd65c556f429 P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c4821d9898d42799c5789d4d2da3f63, State: Initialized, Role: FOLLOWER
I20240503 01:59:31.639183 7434 consensus_queue.cc:260] T e5a00d4ce3a64ca5a7c5bd65c556f429 P 0c4821d9898d42799c5789d4d2da3f63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } }
I20240503 01:59:31.639544 7439 tablet_bootstrap.cc:654] T a388d59f6ba94e91958f55b340343fa6 P 4127249ba42644388d225d614c935a25: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:31.639593 7434 ts_tablet_manager.cc:1398] T e5a00d4ce3a64ca5a7c5bd65c556f429 P 0c4821d9898d42799c5789d4d2da3f63: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:31.639782 7434 tablet_bootstrap.cc:492] T a388d59f6ba94e91958f55b340343fa6 P 0c4821d9898d42799c5789d4d2da3f63: Bootstrap starting.
I20240503 01:59:31.639942 7432 tablet_bootstrap.cc:654] T d6ee8175874c4063986033b0a80ccb82 P 79163c69f6f04a9abd0208d74e1c61be: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:31.640738 7434 tablet_bootstrap.cc:654] T a388d59f6ba94e91958f55b340343fa6 P 0c4821d9898d42799c5789d4d2da3f63: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:31.640791 7439 tablet_bootstrap.cc:492] T a388d59f6ba94e91958f55b340343fa6 P 4127249ba42644388d225d614c935a25: No bootstrap required, opened a new log
I20240503 01:59:31.640877 7439 ts_tablet_manager.cc:1367] T a388d59f6ba94e91958f55b340343fa6 P 4127249ba42644388d225d614c935a25: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:31.641343 7439 raft_consensus.cc:348] T a388d59f6ba94e91958f55b340343fa6 P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:31.641470 7439 raft_consensus.cc:374] T a388d59f6ba94e91958f55b340343fa6 P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:31.641517 7439 raft_consensus.cc:729] T a388d59f6ba94e91958f55b340343fa6 P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4127249ba42644388d225d614c935a25, State: Initialized, Role: FOLLOWER
I20240503 01:59:31.641651 7439 consensus_queue.cc:260] T a388d59f6ba94e91958f55b340343fa6 P 4127249ba42644388d225d614c935a25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:31.641840 7434 tablet_bootstrap.cc:492] T a388d59f6ba94e91958f55b340343fa6 P 0c4821d9898d42799c5789d4d2da3f63: No bootstrap required, opened a new log
I20240503 01:59:31.641914 7439 ts_tablet_manager.cc:1398] T a388d59f6ba94e91958f55b340343fa6 P 4127249ba42644388d225d614c935a25: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:31.641937 7434 ts_tablet_manager.cc:1367] T a388d59f6ba94e91958f55b340343fa6 P 0c4821d9898d42799c5789d4d2da3f63: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240503 01:59:31.642275 7434 raft_consensus.cc:348] T a388d59f6ba94e91958f55b340343fa6 P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:31.642371 7434 raft_consensus.cc:374] T a388d59f6ba94e91958f55b340343fa6 P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:31.642416 7432 tablet_bootstrap.cc:492] T d6ee8175874c4063986033b0a80ccb82 P 79163c69f6f04a9abd0208d74e1c61be: No bootstrap required, opened a new log
I20240503 01:59:31.642436 7434 raft_consensus.cc:729] T a388d59f6ba94e91958f55b340343fa6 P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c4821d9898d42799c5789d4d2da3f63, State: Initialized, Role: FOLLOWER
I20240503 01:59:31.642503 7432 ts_tablet_manager.cc:1367] T d6ee8175874c4063986033b0a80ccb82 P 79163c69f6f04a9abd0208d74e1c61be: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 01:59:31.642562 7434 consensus_queue.cc:260] T a388d59f6ba94e91958f55b340343fa6 P 0c4821d9898d42799c5789d4d2da3f63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:31.642866 7434 ts_tablet_manager.cc:1398] T a388d59f6ba94e91958f55b340343fa6 P 0c4821d9898d42799c5789d4d2da3f63: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:31.643507 7432 raft_consensus.cc:348] T d6ee8175874c4063986033b0a80ccb82 P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } }
I20240503 01:59:31.643595 7432 raft_consensus.cc:374] T d6ee8175874c4063986033b0a80ccb82 P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:31.643628 7432 raft_consensus.cc:729] T d6ee8175874c4063986033b0a80ccb82 P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79163c69f6f04a9abd0208d74e1c61be, State: Initialized, Role: FOLLOWER
I20240503 01:59:31.643774 7432 consensus_queue.cc:260] T d6ee8175874c4063986033b0a80ccb82 P 79163c69f6f04a9abd0208d74e1c61be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } }
I20240503 01:59:31.644845 7432 ts_tablet_manager.cc:1398] T d6ee8175874c4063986033b0a80ccb82 P 79163c69f6f04a9abd0208d74e1c61be: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:31.645025 7432 tablet_bootstrap.cc:492] T a388d59f6ba94e91958f55b340343fa6 P 79163c69f6f04a9abd0208d74e1c61be: Bootstrap starting.
I20240503 01:59:31.646102 7432 tablet_bootstrap.cc:654] T a388d59f6ba94e91958f55b340343fa6 P 79163c69f6f04a9abd0208d74e1c61be: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:31.647007 7432 tablet_bootstrap.cc:492] T a388d59f6ba94e91958f55b340343fa6 P 79163c69f6f04a9abd0208d74e1c61be: No bootstrap required, opened a new log
I20240503 01:59:31.647073 7432 ts_tablet_manager.cc:1367] T a388d59f6ba94e91958f55b340343fa6 P 79163c69f6f04a9abd0208d74e1c61be: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:31.647362 7432 raft_consensus.cc:348] T a388d59f6ba94e91958f55b340343fa6 P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:31.647433 7432 raft_consensus.cc:374] T a388d59f6ba94e91958f55b340343fa6 P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:31.647469 7432 raft_consensus.cc:729] T a388d59f6ba94e91958f55b340343fa6 P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79163c69f6f04a9abd0208d74e1c61be, State: Initialized, Role: FOLLOWER
I20240503 01:59:31.647552 7432 consensus_queue.cc:260] T a388d59f6ba94e91958f55b340343fa6 P 79163c69f6f04a9abd0208d74e1c61be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:31.647727 7432 ts_tablet_manager.cc:1398] T a388d59f6ba94e91958f55b340343fa6 P 79163c69f6f04a9abd0208d74e1c61be: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:31.647873 7432 tablet_bootstrap.cc:492] T e5a00d4ce3a64ca5a7c5bd65c556f429 P 79163c69f6f04a9abd0208d74e1c61be: Bootstrap starting.
I20240503 01:59:31.648613 7432 tablet_bootstrap.cc:654] T e5a00d4ce3a64ca5a7c5bd65c556f429 P 79163c69f6f04a9abd0208d74e1c61be: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:31.649688 7432 tablet_bootstrap.cc:492] T e5a00d4ce3a64ca5a7c5bd65c556f429 P 79163c69f6f04a9abd0208d74e1c61be: No bootstrap required, opened a new log
I20240503 01:59:31.649773 7432 ts_tablet_manager.cc:1367] T e5a00d4ce3a64ca5a7c5bd65c556f429 P 79163c69f6f04a9abd0208d74e1c61be: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 01:59:31.650223 7432 raft_consensus.cc:348] T e5a00d4ce3a64ca5a7c5bd65c556f429 P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } }
I20240503 01:59:31.650297 7432 raft_consensus.cc:374] T e5a00d4ce3a64ca5a7c5bd65c556f429 P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:31.650318 7432 raft_consensus.cc:729] T e5a00d4ce3a64ca5a7c5bd65c556f429 P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79163c69f6f04a9abd0208d74e1c61be, State: Initialized, Role: FOLLOWER
I20240503 01:59:31.650415 7432 consensus_queue.cc:260] T e5a00d4ce3a64ca5a7c5bd65c556f429 P 79163c69f6f04a9abd0208d74e1c61be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } }
I20240503 01:59:31.650641 7432 ts_tablet_manager.cc:1398] T e5a00d4ce3a64ca5a7c5bd65c556f429 P 79163c69f6f04a9abd0208d74e1c61be: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:31.740397 7373 raft_consensus.cc:482] T d6ee8175874c4063986033b0a80ccb82 P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:31.740526 7373 raft_consensus.cc:504] T d6ee8175874c4063986033b0a80ccb82 P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } }
I20240503 01:59:31.741082 7373 leader_election.cc:288] T d6ee8175874c4063986033b0a80ccb82 P 79163c69f6f04a9abd0208d74e1c61be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b6b1d5677b8d47788b7968f64b3dac3c (127.0.99.132:46067), 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130:34189)
I20240503 01:59:31.741550 6757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6ee8175874c4063986033b0a80ccb82" candidate_uuid: "79163c69f6f04a9abd0208d74e1c61be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" is_pre_election: true
I20240503 01:59:31.741704 6757 raft_consensus.cc:2454] T d6ee8175874c4063986033b0a80ccb82 P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79163c69f6f04a9abd0208d74e1c61be in term 0.
I20240503 01:59:31.742009 6562 leader_election.cc:302] T d6ee8175874c4063986033b0a80ccb82 P 79163c69f6f04a9abd0208d74e1c61be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79163c69f6f04a9abd0208d74e1c61be, b6b1d5677b8d47788b7968f64b3dac3c; no voters:
I20240503 01:59:31.742156 7373 raft_consensus.cc:2781] T d6ee8175874c4063986033b0a80ccb82 P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:31.742216 7373 raft_consensus.cc:482] T d6ee8175874c4063986033b0a80ccb82 P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:31.742260 7373 raft_consensus.cc:3037] T d6ee8175874c4063986033b0a80ccb82 P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:31.743152 7373 raft_consensus.cc:504] T d6ee8175874c4063986033b0a80ccb82 P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } }
I20240503 01:59:31.743520 7373 leader_election.cc:288] T d6ee8175874c4063986033b0a80ccb82 P 79163c69f6f04a9abd0208d74e1c61be [CANDIDATE]: Term 1 election: Requested vote from peers b6b1d5677b8d47788b7968f64b3dac3c (127.0.99.132:46067), 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130:34189)
I20240503 01:59:31.743826 6757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6ee8175874c4063986033b0a80ccb82" candidate_uuid: "79163c69f6f04a9abd0208d74e1c61be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6b1d5677b8d47788b7968f64b3dac3c"
I20240503 01:59:31.743932 6757 raft_consensus.cc:3037] T d6ee8175874c4063986033b0a80ccb82 P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:31.744619 6496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6ee8175874c4063986033b0a80ccb82" candidate_uuid: "79163c69f6f04a9abd0208d74e1c61be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c4821d9898d42799c5789d4d2da3f63" is_pre_election: true
I20240503 01:59:31.744673 6495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6ee8175874c4063986033b0a80ccb82" candidate_uuid: "79163c69f6f04a9abd0208d74e1c61be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c4821d9898d42799c5789d4d2da3f63"
I20240503 01:59:31.744755 6757 raft_consensus.cc:2454] T d6ee8175874c4063986033b0a80ccb82 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79163c69f6f04a9abd0208d74e1c61be in term 1.
I20240503 01:59:31.744832 6495 raft_consensus.cc:3037] T d6ee8175874c4063986033b0a80ccb82 P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:31.745043 6562 leader_election.cc:302] T d6ee8175874c4063986033b0a80ccb82 P 79163c69f6f04a9abd0208d74e1c61be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79163c69f6f04a9abd0208d74e1c61be, b6b1d5677b8d47788b7968f64b3dac3c; no voters:
I20240503 01:59:31.745213 7373 raft_consensus.cc:2781] T d6ee8175874c4063986033b0a80ccb82 P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:31.745290 7373 raft_consensus.cc:686] T d6ee8175874c4063986033b0a80ccb82 P 79163c69f6f04a9abd0208d74e1c61be [term 1 LEADER]: Becoming Leader. State: Replica: 79163c69f6f04a9abd0208d74e1c61be, State: Running, Role: LEADER
I20240503 01:59:31.745429 7373 consensus_queue.cc:237] T d6ee8175874c4063986033b0a80ccb82 P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } }
I20240503 01:59:31.745868 6495 raft_consensus.cc:2454] T d6ee8175874c4063986033b0a80ccb82 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79163c69f6f04a9abd0208d74e1c61be in term 1.
I20240503 01:59:31.747288 6218 catalog_manager.cc:5484] T d6ee8175874c4063986033b0a80ccb82 P 79163c69f6f04a9abd0208d74e1c61be reported cstate change: term changed from 0 to 1, leader changed from <none> to 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131). New cstate: current_term: 1 leader_uuid: "79163c69f6f04a9abd0208d74e1c61be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:31.846421 7363 raft_consensus.cc:482] T e5a00d4ce3a64ca5a7c5bd65c556f429 P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:31.846571 7363 raft_consensus.cc:504] T e5a00d4ce3a64ca5a7c5bd65c556f429 P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } }
I20240503 01:59:31.847644 7363 leader_election.cc:288] T e5a00d4ce3a64ca5a7c5bd65c556f429 P b6b1d5677b8d47788b7968f64b3dac3c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130:34189), 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131:44005)
I20240503 01:59:31.851027 6495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5a00d4ce3a64ca5a7c5bd65c556f429" candidate_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c4821d9898d42799c5789d4d2da3f63" is_pre_election: true
I20240503 01:59:31.851037 6627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5a00d4ce3a64ca5a7c5bd65c556f429" candidate_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79163c69f6f04a9abd0208d74e1c61be" is_pre_election: true
I20240503 01:59:31.851167 6495 raft_consensus.cc:2454] T e5a00d4ce3a64ca5a7c5bd65c556f429 P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6b1d5677b8d47788b7968f64b3dac3c in term 0.
I20240503 01:59:31.851182 6627 raft_consensus.cc:2454] T e5a00d4ce3a64ca5a7c5bd65c556f429 P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6b1d5677b8d47788b7968f64b3dac3c in term 0.
I20240503 01:59:31.851497 6694 leader_election.cc:302] T e5a00d4ce3a64ca5a7c5bd65c556f429 P b6b1d5677b8d47788b7968f64b3dac3c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c4821d9898d42799c5789d4d2da3f63, b6b1d5677b8d47788b7968f64b3dac3c; no voters:
I20240503 01:59:31.851712 7363 raft_consensus.cc:2781] T e5a00d4ce3a64ca5a7c5bd65c556f429 P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:31.851769 7363 raft_consensus.cc:482] T e5a00d4ce3a64ca5a7c5bd65c556f429 P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:31.851827 7363 raft_consensus.cc:3037] T e5a00d4ce3a64ca5a7c5bd65c556f429 P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:31.852962 7363 raft_consensus.cc:504] T e5a00d4ce3a64ca5a7c5bd65c556f429 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } }
I20240503 01:59:31.853276 7363 leader_election.cc:288] T e5a00d4ce3a64ca5a7c5bd65c556f429 P b6b1d5677b8d47788b7968f64b3dac3c [CANDIDATE]: Term 1 election: Requested vote from peers 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130:34189), 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131:44005)
I20240503 01:59:31.853611 6627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5a00d4ce3a64ca5a7c5bd65c556f429" candidate_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79163c69f6f04a9abd0208d74e1c61be"
I20240503 01:59:31.853729 6627 raft_consensus.cc:3037] T e5a00d4ce3a64ca5a7c5bd65c556f429 P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:31.853829 6495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5a00d4ce3a64ca5a7c5bd65c556f429" candidate_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c4821d9898d42799c5789d4d2da3f63"
I20240503 01:59:31.853935 6495 raft_consensus.cc:3037] T e5a00d4ce3a64ca5a7c5bd65c556f429 P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:31.854485 6627 raft_consensus.cc:2454] T e5a00d4ce3a64ca5a7c5bd65c556f429 P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6b1d5677b8d47788b7968f64b3dac3c in term 1.
I20240503 01:59:31.854732 6495 raft_consensus.cc:2454] T e5a00d4ce3a64ca5a7c5bd65c556f429 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6b1d5677b8d47788b7968f64b3dac3c in term 1.
I20240503 01:59:31.854755 6692 leader_election.cc:302] T e5a00d4ce3a64ca5a7c5bd65c556f429 P b6b1d5677b8d47788b7968f64b3dac3c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79163c69f6f04a9abd0208d74e1c61be, b6b1d5677b8d47788b7968f64b3dac3c; no voters:
I20240503 01:59:31.854892 7363 raft_consensus.cc:2781] T e5a00d4ce3a64ca5a7c5bd65c556f429 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:31.855088 7363 raft_consensus.cc:686] T e5a00d4ce3a64ca5a7c5bd65c556f429 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 LEADER]: Becoming Leader. State: Replica: b6b1d5677b8d47788b7968f64b3dac3c, State: Running, Role: LEADER
I20240503 01:59:31.855371 7363 consensus_queue.cc:237] T e5a00d4ce3a64ca5a7c5bd65c556f429 P b6b1d5677b8d47788b7968f64b3dac3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } }
I20240503 01:59:31.857640 6218 catalog_manager.cc:5484] T e5a00d4ce3a64ca5a7c5bd65c556f429 P b6b1d5677b8d47788b7968f64b3dac3c reported cstate change: term changed from 0 to 1, leader changed from <none> to b6b1d5677b8d47788b7968f64b3dac3c (127.0.99.132). New cstate: current_term: 1 leader_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:31.975970 7356 raft_consensus.cc:482] T a388d59f6ba94e91958f55b340343fa6 P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:31.976102 7356 raft_consensus.cc:504] T a388d59f6ba94e91958f55b340343fa6 P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:31.976519 7356 leader_election.cc:288] T a388d59f6ba94e91958f55b340343fa6 P 79163c69f6f04a9abd0208d74e1c61be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130:34189), 4127249ba42644388d225d614c935a25 (127.0.99.129:35613)
I20240503 01:59:31.976949 6495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a388d59f6ba94e91958f55b340343fa6" candidate_uuid: "79163c69f6f04a9abd0208d74e1c61be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c4821d9898d42799c5789d4d2da3f63" is_pre_election: true
I20240503 01:59:31.977020 6365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a388d59f6ba94e91958f55b340343fa6" candidate_uuid: "79163c69f6f04a9abd0208d74e1c61be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4127249ba42644388d225d614c935a25" is_pre_election: true
I20240503 01:59:31.977087 6495 raft_consensus.cc:2454] T a388d59f6ba94e91958f55b340343fa6 P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79163c69f6f04a9abd0208d74e1c61be in term 0.
I20240503 01:59:31.977157 6365 raft_consensus.cc:2454] T a388d59f6ba94e91958f55b340343fa6 P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79163c69f6f04a9abd0208d74e1c61be in term 0.
I20240503 01:59:31.977438 6563 leader_election.cc:302] T a388d59f6ba94e91958f55b340343fa6 P 79163c69f6f04a9abd0208d74e1c61be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c4821d9898d42799c5789d4d2da3f63, 79163c69f6f04a9abd0208d74e1c61be; no voters:
I20240503 01:59:31.977639 7356 raft_consensus.cc:2781] T a388d59f6ba94e91958f55b340343fa6 P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:31.977700 7356 raft_consensus.cc:482] T a388d59f6ba94e91958f55b340343fa6 P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:31.977736 7356 raft_consensus.cc:3037] T a388d59f6ba94e91958f55b340343fa6 P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:31.978576 7356 raft_consensus.cc:504] T a388d59f6ba94e91958f55b340343fa6 P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:31.978945 7356 leader_election.cc:288] T a388d59f6ba94e91958f55b340343fa6 P 79163c69f6f04a9abd0208d74e1c61be [CANDIDATE]: Term 1 election: Requested vote from peers 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130:34189), 4127249ba42644388d225d614c935a25 (127.0.99.129:35613)
I20240503 01:59:31.979279 6365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a388d59f6ba94e91958f55b340343fa6" candidate_uuid: "79163c69f6f04a9abd0208d74e1c61be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4127249ba42644388d225d614c935a25"
I20240503 01:59:31.979302 6495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a388d59f6ba94e91958f55b340343fa6" candidate_uuid: "79163c69f6f04a9abd0208d74e1c61be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c4821d9898d42799c5789d4d2da3f63"
I20240503 01:59:31.979408 6495 raft_consensus.cc:3037] T a388d59f6ba94e91958f55b340343fa6 P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:31.979408 6365 raft_consensus.cc:3037] T a388d59f6ba94e91958f55b340343fa6 P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:31.980324 6365 raft_consensus.cc:2454] T a388d59f6ba94e91958f55b340343fa6 P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79163c69f6f04a9abd0208d74e1c61be in term 1.
I20240503 01:59:31.980335 6495 raft_consensus.cc:2454] T a388d59f6ba94e91958f55b340343fa6 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79163c69f6f04a9abd0208d74e1c61be in term 1.
I20240503 01:59:31.980631 6561 leader_election.cc:302] T a388d59f6ba94e91958f55b340343fa6 P 79163c69f6f04a9abd0208d74e1c61be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4127249ba42644388d225d614c935a25, 79163c69f6f04a9abd0208d74e1c61be; no voters:
I20240503 01:59:31.980840 7356 raft_consensus.cc:2781] T a388d59f6ba94e91958f55b340343fa6 P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:31.980923 7356 raft_consensus.cc:686] T a388d59f6ba94e91958f55b340343fa6 P 79163c69f6f04a9abd0208d74e1c61be [term 1 LEADER]: Becoming Leader. State: Replica: 79163c69f6f04a9abd0208d74e1c61be, State: Running, Role: LEADER
I20240503 01:59:31.981061 7356 consensus_queue.cc:237] T a388d59f6ba94e91958f55b340343fa6 P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:31.982606 6218 catalog_manager.cc:5484] T a388d59f6ba94e91958f55b340343fa6 P 79163c69f6f04a9abd0208d74e1c61be reported cstate change: term changed from 0 to 1, leader changed from <none> to 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131). New cstate: current_term: 1 leader_uuid: "79163c69f6f04a9abd0208d74e1c61be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } health_report { overall_health: UNKNOWN } } }
W20240503 01:59:32.091368 7451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:32.091588 7451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:32.094605 7451 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240503 01:59:32.180514 6365 raft_consensus.cc:1261] T a388d59f6ba94e91958f55b340343fa6 P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:32.180755 6495 raft_consensus.cc:1261] T a388d59f6ba94e91958f55b340343fa6 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:32.181232 7356 consensus_queue.cc:1035] T a388d59f6ba94e91958f55b340343fa6 P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:32.181479 7373 consensus_queue.cc:1035] T a388d59f6ba94e91958f55b340343fa6 P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:32.181569 6627 raft_consensus.cc:1261] T e5a00d4ce3a64ca5a7c5bd65c556f429 P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Refusing update from remote peer b6b1d5677b8d47788b7968f64b3dac3c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:32.182011 7363 consensus_queue.cc:1035] T e5a00d4ce3a64ca5a7c5bd65c556f429 P b6b1d5677b8d47788b7968f64b3dac3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:32.183909 6757 raft_consensus.cc:1261] T d6ee8175874c4063986033b0a80ccb82 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:32.184283 7356 consensus_queue.cc:1035] T d6ee8175874c4063986033b0a80ccb82 P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:32.184376 6496 raft_consensus.cc:1261] T d6ee8175874c4063986033b0a80ccb82 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:32.185889 6496 raft_consensus.cc:1261] T e5a00d4ce3a64ca5a7c5bd65c556f429 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Refusing update from remote peer b6b1d5677b8d47788b7968f64b3dac3c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:32.186331 7373 consensus_queue.cc:1035] T d6ee8175874c4063986033b0a80ccb82 P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:32.188614 7363 consensus_queue.cc:1035] T e5a00d4ce3a64ca5a7c5bd65c556f429 P b6b1d5677b8d47788b7968f64b3dac3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:32.203426 7485 mvcc.cc:205] Tried to move back new op lower bound from 7023417639646257152 to 7023417638835507200. Current Snapshot: MvccSnapshot[applied={T|T < 7023417639646257152}]
INSERT report
rows total: 2000
time total: 106.318 ms
time per row: 0.0531591 ms
I20240503 01:59:32.242771 6214 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44084:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:59:32.247787 6737 tablet_service.cc:1467] Processing CreateTablet for tablet 885b787d0cbb4ddabd47cb7c42a78c74 (DEFAULT_TABLE table=rebalance_test_table_2 [id=4a248d80763944458620bf40775e2098]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:32.247972 6343 tablet_service.cc:1467] Processing CreateTablet for tablet 885b787d0cbb4ddabd47cb7c42a78c74 (DEFAULT_TABLE table=rebalance_test_table_2 [id=4a248d80763944458620bf40775e2098]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:32.248077 6737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 885b787d0cbb4ddabd47cb7c42a78c74. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:32.248229 6343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 885b787d0cbb4ddabd47cb7c42a78c74. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:32.248709 6476 tablet_service.cc:1467] Processing CreateTablet for tablet 470b553976064bdd8c321799ac718274 (DEFAULT_TABLE table=rebalance_test_table_2 [id=4a248d80763944458620bf40775e2098]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:32.248952 6476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 470b553976064bdd8c321799ac718274. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:32.248972 6607 tablet_service.cc:1467] Processing CreateTablet for tablet a9a74e04ce3745b2b479e96a3f3dc86e (DEFAULT_TABLE table=rebalance_test_table_2 [id=4a248d80763944458620bf40775e2098]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:32.249233 6607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9a74e04ce3745b2b479e96a3f3dc86e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:32.250476 6474 tablet_service.cc:1467] Processing CreateTablet for tablet 885b787d0cbb4ddabd47cb7c42a78c74 (DEFAULT_TABLE table=rebalance_test_table_2 [id=4a248d80763944458620bf40775e2098]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:32.250559 7514 tablet_bootstrap.cc:492] T 885b787d0cbb4ddabd47cb7c42a78c74 P 4127249ba42644388d225d614c935a25: Bootstrap starting.
I20240503 01:59:32.250695 6474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 885b787d0cbb4ddabd47cb7c42a78c74. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:32.251475 7514 tablet_bootstrap.cc:654] T 885b787d0cbb4ddabd47cb7c42a78c74 P 4127249ba42644388d225d614c935a25: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:32.251729 6343 tablet_service.cc:1467] Processing CreateTablet for tablet a9a74e04ce3745b2b479e96a3f3dc86e (DEFAULT_TABLE table=rebalance_test_table_2 [id=4a248d80763944458620bf40775e2098]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:32.251756 6344 tablet_service.cc:1467] Processing CreateTablet for tablet 470b553976064bdd8c321799ac718274 (DEFAULT_TABLE table=rebalance_test_table_2 [id=4a248d80763944458620bf40775e2098]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:32.251982 6343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9a74e04ce3745b2b479e96a3f3dc86e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:32.252101 6344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 470b553976064bdd8c321799ac718274. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:32.252149 7515 tablet_bootstrap.cc:492] T 470b553976064bdd8c321799ac718274 P 0c4821d9898d42799c5789d4d2da3f63: Bootstrap starting.
I20240503 01:59:32.252930 7516 tablet_bootstrap.cc:492] T a9a74e04ce3745b2b479e96a3f3dc86e P 79163c69f6f04a9abd0208d74e1c61be: Bootstrap starting.
I20240503 01:59:32.252971 7515 tablet_bootstrap.cc:654] T 470b553976064bdd8c321799ac718274 P 0c4821d9898d42799c5789d4d2da3f63: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:32.253806 7516 tablet_bootstrap.cc:654] T a9a74e04ce3745b2b479e96a3f3dc86e P 79163c69f6f04a9abd0208d74e1c61be: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:32.254076 7515 tablet_bootstrap.cc:492] T 470b553976064bdd8c321799ac718274 P 0c4821d9898d42799c5789d4d2da3f63: No bootstrap required, opened a new log
I20240503 01:59:32.254155 7515 ts_tablet_manager.cc:1367] T 470b553976064bdd8c321799ac718274 P 0c4821d9898d42799c5789d4d2da3f63: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:32.254590 7515 raft_consensus.cc:348] T 470b553976064bdd8c321799ac718274 P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } }
I20240503 01:59:32.254699 7515 raft_consensus.cc:374] T 470b553976064bdd8c321799ac718274 P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:32.254732 7515 raft_consensus.cc:729] T 470b553976064bdd8c321799ac718274 P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c4821d9898d42799c5789d4d2da3f63, State: Initialized, Role: FOLLOWER
I20240503 01:59:32.254851 7515 consensus_queue.cc:260] T 470b553976064bdd8c321799ac718274 P 0c4821d9898d42799c5789d4d2da3f63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } }
I20240503 01:59:32.254920 7516 tablet_bootstrap.cc:492] T a9a74e04ce3745b2b479e96a3f3dc86e P 79163c69f6f04a9abd0208d74e1c61be: No bootstrap required, opened a new log
I20240503 01:59:32.254974 7513 tablet_bootstrap.cc:492] T 885b787d0cbb4ddabd47cb7c42a78c74 P b6b1d5677b8d47788b7968f64b3dac3c: Bootstrap starting.
I20240503 01:59:32.255012 7516 ts_tablet_manager.cc:1367] T a9a74e04ce3745b2b479e96a3f3dc86e P 79163c69f6f04a9abd0208d74e1c61be: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:32.255060 6737 tablet_service.cc:1467] Processing CreateTablet for tablet a9a74e04ce3745b2b479e96a3f3dc86e (DEFAULT_TABLE table=rebalance_test_table_2 [id=4a248d80763944458620bf40775e2098]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:32.255153 7515 ts_tablet_manager.cc:1398] T 470b553976064bdd8c321799ac718274 P 0c4821d9898d42799c5789d4d2da3f63: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:32.255259 6737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9a74e04ce3745b2b479e96a3f3dc86e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:32.255301 7515 tablet_bootstrap.cc:492] T 885b787d0cbb4ddabd47cb7c42a78c74 P 0c4821d9898d42799c5789d4d2da3f63: Bootstrap starting.
I20240503 01:59:32.255425 7516 raft_consensus.cc:348] T a9a74e04ce3745b2b479e96a3f3dc86e P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:32.255518 7516 raft_consensus.cc:374] T a9a74e04ce3745b2b479e96a3f3dc86e P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:32.255550 7516 raft_consensus.cc:729] T a9a74e04ce3745b2b479e96a3f3dc86e P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79163c69f6f04a9abd0208d74e1c61be, State: Initialized, Role: FOLLOWER
I20240503 01:59:32.255671 7516 consensus_queue.cc:260] T a9a74e04ce3745b2b479e96a3f3dc86e P 79163c69f6f04a9abd0208d74e1c61be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:32.255767 7514 tablet_bootstrap.cc:492] T 885b787d0cbb4ddabd47cb7c42a78c74 P 4127249ba42644388d225d614c935a25: No bootstrap required, opened a new log
I20240503 01:59:32.255856 7514 ts_tablet_manager.cc:1367] T 885b787d0cbb4ddabd47cb7c42a78c74 P 4127249ba42644388d225d614c935a25: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 01:59:32.255971 7516 ts_tablet_manager.cc:1398] T a9a74e04ce3745b2b479e96a3f3dc86e P 79163c69f6f04a9abd0208d74e1c61be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:32.256198 7515 tablet_bootstrap.cc:654] T 885b787d0cbb4ddabd47cb7c42a78c74 P 0c4821d9898d42799c5789d4d2da3f63: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:32.256340 7514 raft_consensus.cc:348] T 885b787d0cbb4ddabd47cb7c42a78c74 P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:32.256455 7514 raft_consensus.cc:374] T 885b787d0cbb4ddabd47cb7c42a78c74 P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:32.256486 7514 raft_consensus.cc:729] T 885b787d0cbb4ddabd47cb7c42a78c74 P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4127249ba42644388d225d614c935a25, State: Initialized, Role: FOLLOWER
I20240503 01:59:32.256585 7514 consensus_queue.cc:260] T 885b787d0cbb4ddabd47cb7c42a78c74 P 4127249ba42644388d225d614c935a25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:32.256872 7514 ts_tablet_manager.cc:1398] T 885b787d0cbb4ddabd47cb7c42a78c74 P 4127249ba42644388d225d614c935a25: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:32.257031 7514 tablet_bootstrap.cc:492] T a9a74e04ce3745b2b479e96a3f3dc86e P 4127249ba42644388d225d614c935a25: Bootstrap starting.
I20240503 01:59:32.256332 6738 tablet_service.cc:1467] Processing CreateTablet for tablet 470b553976064bdd8c321799ac718274 (DEFAULT_TABLE table=rebalance_test_table_2 [id=4a248d80763944458620bf40775e2098]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:32.257783 6738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 470b553976064bdd8c321799ac718274. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:32.257957 7514 tablet_bootstrap.cc:654] T a9a74e04ce3745b2b479e96a3f3dc86e P 4127249ba42644388d225d614c935a25: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:32.258097 7515 tablet_bootstrap.cc:492] T 885b787d0cbb4ddabd47cb7c42a78c74 P 0c4821d9898d42799c5789d4d2da3f63: No bootstrap required, opened a new log
I20240503 01:59:32.258164 7515 ts_tablet_manager.cc:1367] T 885b787d0cbb4ddabd47cb7c42a78c74 P 0c4821d9898d42799c5789d4d2da3f63: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 01:59:32.258570 7515 raft_consensus.cc:348] T 885b787d0cbb4ddabd47cb7c42a78c74 P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:32.258657 7515 raft_consensus.cc:374] T 885b787d0cbb4ddabd47cb7c42a78c74 P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:32.258688 7515 raft_consensus.cc:729] T 885b787d0cbb4ddabd47cb7c42a78c74 P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c4821d9898d42799c5789d4d2da3f63, State: Initialized, Role: FOLLOWER
I20240503 01:59:32.258809 7515 consensus_queue.cc:260] T 885b787d0cbb4ddabd47cb7c42a78c74 P 0c4821d9898d42799c5789d4d2da3f63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:32.259102 7515 ts_tablet_manager.cc:1398] T 885b787d0cbb4ddabd47cb7c42a78c74 P 0c4821d9898d42799c5789d4d2da3f63: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:32.259238 7513 tablet_bootstrap.cc:654] T 885b787d0cbb4ddabd47cb7c42a78c74 P b6b1d5677b8d47788b7968f64b3dac3c: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:32.260334 7513 tablet_bootstrap.cc:492] T 885b787d0cbb4ddabd47cb7c42a78c74 P b6b1d5677b8d47788b7968f64b3dac3c: No bootstrap required, opened a new log
I20240503 01:59:32.260385 7514 tablet_bootstrap.cc:492] T a9a74e04ce3745b2b479e96a3f3dc86e P 4127249ba42644388d225d614c935a25: No bootstrap required, opened a new log
I20240503 01:59:32.260416 7513 ts_tablet_manager.cc:1367] T 885b787d0cbb4ddabd47cb7c42a78c74 P b6b1d5677b8d47788b7968f64b3dac3c: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240503 01:59:32.260464 7514 ts_tablet_manager.cc:1367] T a9a74e04ce3745b2b479e96a3f3dc86e P 4127249ba42644388d225d614c935a25: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:32.260885 7513 raft_consensus.cc:348] T 885b787d0cbb4ddabd47cb7c42a78c74 P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:32.260910 7514 raft_consensus.cc:348] T a9a74e04ce3745b2b479e96a3f3dc86e P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:32.260991 7513 raft_consensus.cc:374] T 885b787d0cbb4ddabd47cb7c42a78c74 P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:32.261009 7514 raft_consensus.cc:374] T a9a74e04ce3745b2b479e96a3f3dc86e P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:32.261035 7513 raft_consensus.cc:729] T 885b787d0cbb4ddabd47cb7c42a78c74 P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6b1d5677b8d47788b7968f64b3dac3c, State: Initialized, Role: FOLLOWER
I20240503 01:59:32.261044 7514 raft_consensus.cc:729] T a9a74e04ce3745b2b479e96a3f3dc86e P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4127249ba42644388d225d614c935a25, State: Initialized, Role: FOLLOWER
I20240503 01:59:32.261155 7513 consensus_queue.cc:260] T 885b787d0cbb4ddabd47cb7c42a78c74 P b6b1d5677b8d47788b7968f64b3dac3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:32.261157 7514 consensus_queue.cc:260] T a9a74e04ce3745b2b479e96a3f3dc86e P 4127249ba42644388d225d614c935a25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:32.261430 7514 ts_tablet_manager.cc:1398] T a9a74e04ce3745b2b479e96a3f3dc86e P 4127249ba42644388d225d614c935a25: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:32.261422 7513 ts_tablet_manager.cc:1398] T 885b787d0cbb4ddabd47cb7c42a78c74 P b6b1d5677b8d47788b7968f64b3dac3c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:32.261610 7513 tablet_bootstrap.cc:492] T a9a74e04ce3745b2b479e96a3f3dc86e P b6b1d5677b8d47788b7968f64b3dac3c: Bootstrap starting.
I20240503 01:59:32.261617 7514 tablet_bootstrap.cc:492] T 470b553976064bdd8c321799ac718274 P 4127249ba42644388d225d614c935a25: Bootstrap starting.
I20240503 01:59:32.262243 7361 raft_consensus.cc:482] T 470b553976064bdd8c321799ac718274 P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:32.262323 7361 raft_consensus.cc:504] T 470b553976064bdd8c321799ac718274 P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } }
I20240503 01:59:32.262475 7513 tablet_bootstrap.cc:654] T a9a74e04ce3745b2b479e96a3f3dc86e P b6b1d5677b8d47788b7968f64b3dac3c: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:32.262620 7514 tablet_bootstrap.cc:654] T 470b553976064bdd8c321799ac718274 P 4127249ba42644388d225d614c935a25: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:32.262722 7361 leader_election.cc:288] T 470b553976064bdd8c321799ac718274 P 0c4821d9898d42799c5789d4d2da3f63 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b6b1d5677b8d47788b7968f64b3dac3c (127.0.99.132:46067), 4127249ba42644388d225d614c935a25 (127.0.99.129:35613)
I20240503 01:59:32.263087 6757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "470b553976064bdd8c321799ac718274" candidate_uuid: "0c4821d9898d42799c5789d4d2da3f63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" is_pre_election: true
I20240503 01:59:32.263342 6365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "470b553976064bdd8c321799ac718274" candidate_uuid: "0c4821d9898d42799c5789d4d2da3f63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4127249ba42644388d225d614c935a25" is_pre_election: true
W20240503 01:59:32.263584 6431 leader_election.cc:341] T 470b553976064bdd8c321799ac718274 P 0c4821d9898d42799c5789d4d2da3f63 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b6b1d5677b8d47788b7968f64b3dac3c (127.0.99.132:46067): Illegal state: must be running to vote when last-logged opid is not known
I20240503 01:59:32.263847 7514 tablet_bootstrap.cc:492] T 470b553976064bdd8c321799ac718274 P 4127249ba42644388d225d614c935a25: No bootstrap required, opened a new log
I20240503 01:59:32.263932 7514 ts_tablet_manager.cc:1367] T 470b553976064bdd8c321799ac718274 P 4127249ba42644388d225d614c935a25: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
W20240503 01:59:32.264067 6430 leader_election.cc:341] T 470b553976064bdd8c321799ac718274 P 0c4821d9898d42799c5789d4d2da3f63 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4127249ba42644388d225d614c935a25 (127.0.99.129:35613): Illegal state: must be running to vote when last-logged opid is not known
I20240503 01:59:32.264147 6430 leader_election.cc:302] T 470b553976064bdd8c321799ac718274 P 0c4821d9898d42799c5789d4d2da3f63 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0c4821d9898d42799c5789d4d2da3f63; no voters: 4127249ba42644388d225d614c935a25, b6b1d5677b8d47788b7968f64b3dac3c
I20240503 01:59:32.264286 7361 raft_consensus.cc:2726] T 470b553976064bdd8c321799ac718274 P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 01:59:32.264334 7514 raft_consensus.cc:348] T 470b553976064bdd8c321799ac718274 P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } }
I20240503 01:59:32.264479 7514 raft_consensus.cc:374] T 470b553976064bdd8c321799ac718274 P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:32.264526 7514 raft_consensus.cc:729] T 470b553976064bdd8c321799ac718274 P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4127249ba42644388d225d614c935a25, State: Initialized, Role: FOLLOWER
I20240503 01:59:32.264680 7514 consensus_queue.cc:260] T 470b553976064bdd8c321799ac718274 P 4127249ba42644388d225d614c935a25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } }
I20240503 01:59:32.264941 7514 ts_tablet_manager.cc:1398] T 470b553976064bdd8c321799ac718274 P 4127249ba42644388d225d614c935a25: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:32.265482 7513 tablet_bootstrap.cc:492] T a9a74e04ce3745b2b479e96a3f3dc86e P b6b1d5677b8d47788b7968f64b3dac3c: No bootstrap required, opened a new log
I20240503 01:59:32.265558 7513 ts_tablet_manager.cc:1367] T a9a74e04ce3745b2b479e96a3f3dc86e P b6b1d5677b8d47788b7968f64b3dac3c: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 01:59:32.265977 7513 raft_consensus.cc:348] T a9a74e04ce3745b2b479e96a3f3dc86e P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:32.266073 7513 raft_consensus.cc:374] T a9a74e04ce3745b2b479e96a3f3dc86e P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:32.266106 7513 raft_consensus.cc:729] T a9a74e04ce3745b2b479e96a3f3dc86e P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6b1d5677b8d47788b7968f64b3dac3c, State: Initialized, Role: FOLLOWER
I20240503 01:59:32.266209 7513 consensus_queue.cc:260] T a9a74e04ce3745b2b479e96a3f3dc86e P b6b1d5677b8d47788b7968f64b3dac3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:32.266463 7513 ts_tablet_manager.cc:1398] T a9a74e04ce3745b2b479e96a3f3dc86e P b6b1d5677b8d47788b7968f64b3dac3c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:32.266670 7513 tablet_bootstrap.cc:492] T 470b553976064bdd8c321799ac718274 P b6b1d5677b8d47788b7968f64b3dac3c: Bootstrap starting.
I20240503 01:59:32.267483 7513 tablet_bootstrap.cc:654] T 470b553976064bdd8c321799ac718274 P b6b1d5677b8d47788b7968f64b3dac3c: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:32.268815 7513 tablet_bootstrap.cc:492] T 470b553976064bdd8c321799ac718274 P b6b1d5677b8d47788b7968f64b3dac3c: No bootstrap required, opened a new log
I20240503 01:59:32.268893 7513 ts_tablet_manager.cc:1367] T 470b553976064bdd8c321799ac718274 P b6b1d5677b8d47788b7968f64b3dac3c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:32.269266 7513 raft_consensus.cc:348] T 470b553976064bdd8c321799ac718274 P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } }
I20240503 01:59:32.269341 7513 raft_consensus.cc:374] T 470b553976064bdd8c321799ac718274 P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:32.269361 7513 raft_consensus.cc:729] T 470b553976064bdd8c321799ac718274 P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6b1d5677b8d47788b7968f64b3dac3c, State: Initialized, Role: FOLLOWER
I20240503 01:59:32.269485 7513 consensus_queue.cc:260] T 470b553976064bdd8c321799ac718274 P b6b1d5677b8d47788b7968f64b3dac3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } }
I20240503 01:59:32.269742 7513 ts_tablet_manager.cc:1398] T 470b553976064bdd8c321799ac718274 P b6b1d5677b8d47788b7968f64b3dac3c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:32.373214 7489 raft_consensus.cc:482] T a9a74e04ce3745b2b479e96a3f3dc86e P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:32.373453 7489 raft_consensus.cc:504] T a9a74e04ce3745b2b479e96a3f3dc86e P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:32.373862 7489 leader_election.cc:288] T a9a74e04ce3745b2b479e96a3f3dc86e P 79163c69f6f04a9abd0208d74e1c61be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b6b1d5677b8d47788b7968f64b3dac3c (127.0.99.132:46067), 4127249ba42644388d225d614c935a25 (127.0.99.129:35613)
I20240503 01:59:32.374310 6365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9a74e04ce3745b2b479e96a3f3dc86e" candidate_uuid: "79163c69f6f04a9abd0208d74e1c61be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4127249ba42644388d225d614c935a25" is_pre_election: true
I20240503 01:59:32.374315 6757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9a74e04ce3745b2b479e96a3f3dc86e" candidate_uuid: "79163c69f6f04a9abd0208d74e1c61be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" is_pre_election: true
I20240503 01:59:32.374464 6757 raft_consensus.cc:2454] T a9a74e04ce3745b2b479e96a3f3dc86e P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79163c69f6f04a9abd0208d74e1c61be in term 0.
I20240503 01:59:32.374470 6365 raft_consensus.cc:2454] T a9a74e04ce3745b2b479e96a3f3dc86e P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79163c69f6f04a9abd0208d74e1c61be in term 0.
I20240503 01:59:32.374760 6561 leader_election.cc:302] T a9a74e04ce3745b2b479e96a3f3dc86e P 79163c69f6f04a9abd0208d74e1c61be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4127249ba42644388d225d614c935a25, 79163c69f6f04a9abd0208d74e1c61be; no voters:
I20240503 01:59:32.374962 7489 raft_consensus.cc:2781] T a9a74e04ce3745b2b479e96a3f3dc86e P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:32.375010 7489 raft_consensus.cc:482] T a9a74e04ce3745b2b479e96a3f3dc86e P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:32.375054 7489 raft_consensus.cc:3037] T a9a74e04ce3745b2b479e96a3f3dc86e P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:32.375901 7489 raft_consensus.cc:504] T a9a74e04ce3745b2b479e96a3f3dc86e P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:32.376200 7489 leader_election.cc:288] T a9a74e04ce3745b2b479e96a3f3dc86e P 79163c69f6f04a9abd0208d74e1c61be [CANDIDATE]: Term 1 election: Requested vote from peers b6b1d5677b8d47788b7968f64b3dac3c (127.0.99.132:46067), 4127249ba42644388d225d614c935a25 (127.0.99.129:35613)
I20240503 01:59:32.376582 6365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9a74e04ce3745b2b479e96a3f3dc86e" candidate_uuid: "79163c69f6f04a9abd0208d74e1c61be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4127249ba42644388d225d614c935a25"
I20240503 01:59:32.376546 6757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9a74e04ce3745b2b479e96a3f3dc86e" candidate_uuid: "79163c69f6f04a9abd0208d74e1c61be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6b1d5677b8d47788b7968f64b3dac3c"
I20240503 01:59:32.376724 6365 raft_consensus.cc:3037] T a9a74e04ce3745b2b479e96a3f3dc86e P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:32.376724 6757 raft_consensus.cc:3037] T a9a74e04ce3745b2b479e96a3f3dc86e P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:32.377662 6365 raft_consensus.cc:2454] T a9a74e04ce3745b2b479e96a3f3dc86e P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79163c69f6f04a9abd0208d74e1c61be in term 1.
I20240503 01:59:32.377662 6757 raft_consensus.cc:2454] T a9a74e04ce3745b2b479e96a3f3dc86e P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79163c69f6f04a9abd0208d74e1c61be in term 1.
I20240503 01:59:32.377972 6561 leader_election.cc:302] T a9a74e04ce3745b2b479e96a3f3dc86e P 79163c69f6f04a9abd0208d74e1c61be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4127249ba42644388d225d614c935a25, 79163c69f6f04a9abd0208d74e1c61be; no voters:
I20240503 01:59:32.378134 7489 raft_consensus.cc:2781] T a9a74e04ce3745b2b479e96a3f3dc86e P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:32.378357 7489 raft_consensus.cc:686] T a9a74e04ce3745b2b479e96a3f3dc86e P 79163c69f6f04a9abd0208d74e1c61be [term 1 LEADER]: Becoming Leader. State: Replica: 79163c69f6f04a9abd0208d74e1c61be, State: Running, Role: LEADER
I20240503 01:59:32.378509 7489 consensus_queue.cc:237] T a9a74e04ce3745b2b479e96a3f3dc86e P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:32.380234 6214 catalog_manager.cc:5484] T a9a74e04ce3745b2b479e96a3f3dc86e P 79163c69f6f04a9abd0208d74e1c61be reported cstate change: term changed from 0 to 1, leader changed from <none> to 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131). New cstate: current_term: 1 leader_uuid: "79163c69f6f04a9abd0208d74e1c61be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:32.517037 7499 raft_consensus.cc:482] T 885b787d0cbb4ddabd47cb7c42a78c74 P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:32.517189 7499 raft_consensus.cc:504] T 885b787d0cbb4ddabd47cb7c42a78c74 P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:32.518180 7499 leader_election.cc:288] T 885b787d0cbb4ddabd47cb7c42a78c74 P 4127249ba42644388d225d614c935a25 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b6b1d5677b8d47788b7968f64b3dac3c (127.0.99.132:46067), 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130:34189)
I20240503 01:59:32.522027 6757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "885b787d0cbb4ddabd47cb7c42a78c74" candidate_uuid: "4127249ba42644388d225d614c935a25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" is_pre_election: true
I20240503 01:59:32.522048 6494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "885b787d0cbb4ddabd47cb7c42a78c74" candidate_uuid: "4127249ba42644388d225d614c935a25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c4821d9898d42799c5789d4d2da3f63" is_pre_election: true
I20240503 01:59:32.522172 6757 raft_consensus.cc:2454] T 885b787d0cbb4ddabd47cb7c42a78c74 P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4127249ba42644388d225d614c935a25 in term 0.
I20240503 01:59:32.522179 6494 raft_consensus.cc:2454] T 885b787d0cbb4ddabd47cb7c42a78c74 P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4127249ba42644388d225d614c935a25 in term 0.
I20240503 01:59:32.522523 6301 leader_election.cc:302] T 885b787d0cbb4ddabd47cb7c42a78c74 P 4127249ba42644388d225d614c935a25 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c4821d9898d42799c5789d4d2da3f63, 4127249ba42644388d225d614c935a25; no voters:
I20240503 01:59:32.522792 7499 raft_consensus.cc:2781] T 885b787d0cbb4ddabd47cb7c42a78c74 P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:32.522862 7499 raft_consensus.cc:482] T 885b787d0cbb4ddabd47cb7c42a78c74 P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:32.522889 7499 raft_consensus.cc:3037] T 885b787d0cbb4ddabd47cb7c42a78c74 P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:32.523699 7499 raft_consensus.cc:504] T 885b787d0cbb4ddabd47cb7c42a78c74 P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:32.523996 7499 leader_election.cc:288] T 885b787d0cbb4ddabd47cb7c42a78c74 P 4127249ba42644388d225d614c935a25 [CANDIDATE]: Term 1 election: Requested vote from peers b6b1d5677b8d47788b7968f64b3dac3c (127.0.99.132:46067), 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130:34189)
I20240503 01:59:32.524351 6757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "885b787d0cbb4ddabd47cb7c42a78c74" candidate_uuid: "4127249ba42644388d225d614c935a25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6b1d5677b8d47788b7968f64b3dac3c"
I20240503 01:59:32.524499 6757 raft_consensus.cc:3037] T 885b787d0cbb4ddabd47cb7c42a78c74 P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:32.524539 6494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "885b787d0cbb4ddabd47cb7c42a78c74" candidate_uuid: "4127249ba42644388d225d614c935a25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c4821d9898d42799c5789d4d2da3f63"
I20240503 01:59:32.524690 6494 raft_consensus.cc:3037] T 885b787d0cbb4ddabd47cb7c42a78c74 P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:32.525323 6757 raft_consensus.cc:2454] T 885b787d0cbb4ddabd47cb7c42a78c74 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4127249ba42644388d225d614c935a25 in term 1.
I20240503 01:59:32.525508 6494 raft_consensus.cc:2454] T 885b787d0cbb4ddabd47cb7c42a78c74 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4127249ba42644388d225d614c935a25 in term 1.
I20240503 01:59:32.525637 6300 leader_election.cc:302] T 885b787d0cbb4ddabd47cb7c42a78c74 P 4127249ba42644388d225d614c935a25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4127249ba42644388d225d614c935a25, b6b1d5677b8d47788b7968f64b3dac3c; no voters:
I20240503 01:59:32.525799 7499 raft_consensus.cc:2781] T 885b787d0cbb4ddabd47cb7c42a78c74 P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:32.525887 7499 raft_consensus.cc:686] T 885b787d0cbb4ddabd47cb7c42a78c74 P 4127249ba42644388d225d614c935a25 [term 1 LEADER]: Becoming Leader. State: Replica: 4127249ba42644388d225d614c935a25, State: Running, Role: LEADER
I20240503 01:59:32.526136 7499 consensus_queue.cc:237] T 885b787d0cbb4ddabd47cb7c42a78c74 P 4127249ba42644388d225d614c935a25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:32.528211 6214 catalog_manager.cc:5484] T 885b787d0cbb4ddabd47cb7c42a78c74 P 4127249ba42644388d225d614c935a25 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4127249ba42644388d225d614c935a25 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "4127249ba42644388d225d614c935a25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } health_report { overall_health: HEALTHY } } }
I20240503 01:59:32.578349 7499 raft_consensus.cc:482] T 470b553976064bdd8c321799ac718274 P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:32.578500 7499 raft_consensus.cc:504] T 470b553976064bdd8c321799ac718274 P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } }
I20240503 01:59:32.578953 7499 leader_election.cc:288] T 470b553976064bdd8c321799ac718274 P 4127249ba42644388d225d614c935a25 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b6b1d5677b8d47788b7968f64b3dac3c (127.0.99.132:46067), 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130:34189)
I20240503 01:59:32.579341 6757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "470b553976064bdd8c321799ac718274" candidate_uuid: "4127249ba42644388d225d614c935a25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" is_pre_election: true
I20240503 01:59:32.579391 6494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "470b553976064bdd8c321799ac718274" candidate_uuid: "4127249ba42644388d225d614c935a25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c4821d9898d42799c5789d4d2da3f63" is_pre_election: true
I20240503 01:59:32.579464 6757 raft_consensus.cc:2454] T 470b553976064bdd8c321799ac718274 P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4127249ba42644388d225d614c935a25 in term 0.
I20240503 01:59:32.579566 6494 raft_consensus.cc:2454] T 470b553976064bdd8c321799ac718274 P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4127249ba42644388d225d614c935a25 in term 0.
I20240503 01:59:32.579746 6300 leader_election.cc:302] T 470b553976064bdd8c321799ac718274 P 4127249ba42644388d225d614c935a25 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4127249ba42644388d225d614c935a25, b6b1d5677b8d47788b7968f64b3dac3c; no voters:
I20240503 01:59:32.579919 7499 raft_consensus.cc:2781] T 470b553976064bdd8c321799ac718274 P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:32.579980 7499 raft_consensus.cc:482] T 470b553976064bdd8c321799ac718274 P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:32.580006 7499 raft_consensus.cc:3037] T 470b553976064bdd8c321799ac718274 P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:32.580835 7499 raft_consensus.cc:504] T 470b553976064bdd8c321799ac718274 P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } }
I20240503 01:59:32.581135 7499 leader_election.cc:288] T 470b553976064bdd8c321799ac718274 P 4127249ba42644388d225d614c935a25 [CANDIDATE]: Term 1 election: Requested vote from peers b6b1d5677b8d47788b7968f64b3dac3c (127.0.99.132:46067), 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130:34189)
I20240503 01:59:32.581552 6757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "470b553976064bdd8c321799ac718274" candidate_uuid: "4127249ba42644388d225d614c935a25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6b1d5677b8d47788b7968f64b3dac3c"
I20240503 01:59:32.581553 6494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "470b553976064bdd8c321799ac718274" candidate_uuid: "4127249ba42644388d225d614c935a25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c4821d9898d42799c5789d4d2da3f63"
I20240503 01:59:32.581673 6757 raft_consensus.cc:3037] T 470b553976064bdd8c321799ac718274 P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:32.581673 6494 raft_consensus.cc:3037] T 470b553976064bdd8c321799ac718274 P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:32.582618 6494 raft_consensus.cc:2454] T 470b553976064bdd8c321799ac718274 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4127249ba42644388d225d614c935a25 in term 1.
I20240503 01:59:32.582618 6757 raft_consensus.cc:2454] T 470b553976064bdd8c321799ac718274 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4127249ba42644388d225d614c935a25 in term 1.
I20240503 01:59:32.582911 6301 leader_election.cc:302] T 470b553976064bdd8c321799ac718274 P 4127249ba42644388d225d614c935a25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c4821d9898d42799c5789d4d2da3f63, 4127249ba42644388d225d614c935a25; no voters:
I20240503 01:59:32.583113 7499 raft_consensus.cc:2781] T 470b553976064bdd8c321799ac718274 P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:32.583205 7499 raft_consensus.cc:686] T 470b553976064bdd8c321799ac718274 P 4127249ba42644388d225d614c935a25 [term 1 LEADER]: Becoming Leader. State: Replica: 4127249ba42644388d225d614c935a25, State: Running, Role: LEADER
I20240503 01:59:32.583356 7499 consensus_queue.cc:237] T 470b553976064bdd8c321799ac718274 P 4127249ba42644388d225d614c935a25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } }
I20240503 01:59:32.585242 6214 catalog_manager.cc:5484] T 470b553976064bdd8c321799ac718274 P 4127249ba42644388d225d614c935a25 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4127249ba42644388d225d614c935a25 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "4127249ba42644388d225d614c935a25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } health_report { overall_health: UNKNOWN } } }
W20240503 01:59:32.722824 7532 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:32.723033 7532 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:32.726050 7532 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240503 01:59:32.782441 7489 consensus_queue.cc:1035] T a9a74e04ce3745b2b479e96a3f3dc86e P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:32.786108 7373 consensus_queue.cc:1035] T a9a74e04ce3745b2b479e96a3f3dc86e P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:32.813942 6494 raft_consensus.cc:1261] T 885b787d0cbb4ddabd47cb7c42a78c74 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Refusing update from remote peer 4127249ba42644388d225d614c935a25: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:32.815754 6494 raft_consensus.cc:1261] T 470b553976064bdd8c321799ac718274 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Refusing update from remote peer 4127249ba42644388d225d614c935a25: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:32.815699 7499 consensus_queue.cc:1035] T 885b787d0cbb4ddabd47cb7c42a78c74 P 4127249ba42644388d225d614c935a25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:32.816211 6758 raft_consensus.cc:1261] T 470b553976064bdd8c321799ac718274 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Refusing update from remote peer 4127249ba42644388d225d614c935a25: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:32.816553 7499 consensus_queue.cc:1035] T 470b553976064bdd8c321799ac718274 P 4127249ba42644388d225d614c935a25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:32.821291 6757 raft_consensus.cc:1261] T 885b787d0cbb4ddabd47cb7c42a78c74 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Refusing update from remote peer 4127249ba42644388d225d614c935a25: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:32.821900 7484 consensus_queue.cc:1035] T 470b553976064bdd8c321799ac718274 P 4127249ba42644388d225d614c935a25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:32.822211 7484 consensus_queue.cc:1035] T 885b787d0cbb4ddabd47cb7c42a78c74 P 4127249ba42644388d225d614c935a25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 104.318 ms
time per row: 0.0521588 ms
I20240503 01:59:32.872684 6215 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44084:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:59:32.878134 6607 tablet_service.cc:1467] Processing CreateTablet for tablet 21d27cca81e8455e9934e1cc604fc2b2 (DEFAULT_TABLE table=rebalance_test_table_3 [id=2c07d364168a409a98aed9a7c3c1015c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:32.878264 6474 tablet_service.cc:1467] Processing CreateTablet for tablet 21d27cca81e8455e9934e1cc604fc2b2 (DEFAULT_TABLE table=rebalance_test_table_3 [id=2c07d364168a409a98aed9a7c3c1015c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:32.878458 6607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21d27cca81e8455e9934e1cc604fc2b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:32.878551 6474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21d27cca81e8455e9934e1cc604fc2b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:32.879038 6476 tablet_service.cc:1467] Processing CreateTablet for tablet 2c8c328220e64e168c74fea7619f0c3b (DEFAULT_TABLE table=rebalance_test_table_3 [id=2c07d364168a409a98aed9a7c3c1015c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:32.879139 6606 tablet_service.cc:1467] Processing CreateTablet for tablet 2c8c328220e64e168c74fea7619f0c3b (DEFAULT_TABLE table=rebalance_test_table_3 [id=2c07d364168a409a98aed9a7c3c1015c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:32.879271 6476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c8c328220e64e168c74fea7619f0c3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:32.879359 6606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c8c328220e64e168c74fea7619f0c3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:32.881587 6605 tablet_service.cc:1467] Processing CreateTablet for tablet a5926dae08184952bc693342c2c61ebf (DEFAULT_TABLE table=rebalance_test_table_3 [id=2c07d364168a409a98aed9a7c3c1015c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:32.881587 6343 tablet_service.cc:1467] Processing CreateTablet for tablet a5926dae08184952bc693342c2c61ebf (DEFAULT_TABLE table=rebalance_test_table_3 [id=2c07d364168a409a98aed9a7c3c1015c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:32.881614 6344 tablet_service.cc:1467] Processing CreateTablet for tablet 21d27cca81e8455e9934e1cc604fc2b2 (DEFAULT_TABLE table=rebalance_test_table_3 [id=2c07d364168a409a98aed9a7c3c1015c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:32.881815 6605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5926dae08184952bc693342c2c61ebf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:32.881851 6343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5926dae08184952bc693342c2c61ebf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:32.883877 6344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21d27cca81e8455e9934e1cc604fc2b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:32.884325 7594 tablet_bootstrap.cc:492] T a5926dae08184952bc693342c2c61ebf P 4127249ba42644388d225d614c935a25: Bootstrap starting.
I20240503 01:59:32.884351 6738 tablet_service.cc:1467] Processing CreateTablet for tablet a5926dae08184952bc693342c2c61ebf (DEFAULT_TABLE table=rebalance_test_table_3 [id=2c07d364168a409a98aed9a7c3c1015c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:32.884562 6738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5926dae08184952bc693342c2c61ebf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:32.885212 7594 tablet_bootstrap.cc:654] T a5926dae08184952bc693342c2c61ebf P 4127249ba42644388d225d614c935a25: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:32.886407 7596 tablet_bootstrap.cc:492] T a5926dae08184952bc693342c2c61ebf P b6b1d5677b8d47788b7968f64b3dac3c: Bootstrap starting.
I20240503 01:59:32.886559 7594 tablet_bootstrap.cc:492] T a5926dae08184952bc693342c2c61ebf P 4127249ba42644388d225d614c935a25: No bootstrap required, opened a new log
I20240503 01:59:32.886587 6737 tablet_service.cc:1467] Processing CreateTablet for tablet 2c8c328220e64e168c74fea7619f0c3b (DEFAULT_TABLE table=rebalance_test_table_3 [id=2c07d364168a409a98aed9a7c3c1015c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:32.886706 7594 ts_tablet_manager.cc:1367] T a5926dae08184952bc693342c2c61ebf P 4127249ba42644388d225d614c935a25: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:32.886823 6737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c8c328220e64e168c74fea7619f0c3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:32.887177 7594 raft_consensus.cc:348] T a5926dae08184952bc693342c2c61ebf P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:32.887292 7596 tablet_bootstrap.cc:654] T a5926dae08184952bc693342c2c61ebf P b6b1d5677b8d47788b7968f64b3dac3c: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:32.887310 7594 raft_consensus.cc:374] T a5926dae08184952bc693342c2c61ebf P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:32.887341 7594 raft_consensus.cc:729] T a5926dae08184952bc693342c2c61ebf P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4127249ba42644388d225d614c935a25, State: Initialized, Role: FOLLOWER
I20240503 01:59:32.887527 7594 consensus_queue.cc:260] T a5926dae08184952bc693342c2c61ebf P 4127249ba42644388d225d614c935a25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:32.887840 7594 ts_tablet_manager.cc:1398] T a5926dae08184952bc693342c2c61ebf P 4127249ba42644388d225d614c935a25: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:32.888038 7594 tablet_bootstrap.cc:492] T 21d27cca81e8455e9934e1cc604fc2b2 P 4127249ba42644388d225d614c935a25: Bootstrap starting.
I20240503 01:59:32.888710 7592 tablet_bootstrap.cc:492] T 21d27cca81e8455e9934e1cc604fc2b2 P 0c4821d9898d42799c5789d4d2da3f63: Bootstrap starting.
I20240503 01:59:32.888907 7594 tablet_bootstrap.cc:654] T 21d27cca81e8455e9934e1cc604fc2b2 P 4127249ba42644388d225d614c935a25: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:32.889686 7592 tablet_bootstrap.cc:654] T 21d27cca81e8455e9934e1cc604fc2b2 P 0c4821d9898d42799c5789d4d2da3f63: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:32.889915 7594 tablet_bootstrap.cc:492] T 21d27cca81e8455e9934e1cc604fc2b2 P 4127249ba42644388d225d614c935a25: No bootstrap required, opened a new log
I20240503 01:59:32.889994 7594 ts_tablet_manager.cc:1367] T 21d27cca81e8455e9934e1cc604fc2b2 P 4127249ba42644388d225d614c935a25: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:32.890372 7593 tablet_bootstrap.cc:492] T 21d27cca81e8455e9934e1cc604fc2b2 P 79163c69f6f04a9abd0208d74e1c61be: Bootstrap starting.
I20240503 01:59:32.890411 7594 raft_consensus.cc:348] T 21d27cca81e8455e9934e1cc604fc2b2 P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:32.890522 7594 raft_consensus.cc:374] T 21d27cca81e8455e9934e1cc604fc2b2 P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:32.890578 7594 raft_consensus.cc:729] T 21d27cca81e8455e9934e1cc604fc2b2 P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4127249ba42644388d225d614c935a25, State: Initialized, Role: FOLLOWER
I20240503 01:59:32.890702 7594 consensus_queue.cc:260] T 21d27cca81e8455e9934e1cc604fc2b2 P 4127249ba42644388d225d614c935a25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:32.890852 7592 tablet_bootstrap.cc:492] T 21d27cca81e8455e9934e1cc604fc2b2 P 0c4821d9898d42799c5789d4d2da3f63: No bootstrap required, opened a new log
I20240503 01:59:32.890951 7592 ts_tablet_manager.cc:1367] T 21d27cca81e8455e9934e1cc604fc2b2 P 0c4821d9898d42799c5789d4d2da3f63: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:32.890962 7594 ts_tablet_manager.cc:1398] T 21d27cca81e8455e9934e1cc604fc2b2 P 4127249ba42644388d225d614c935a25: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:32.891422 7592 raft_consensus.cc:348] T 21d27cca81e8455e9934e1cc604fc2b2 P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:32.891566 7592 raft_consensus.cc:374] T 21d27cca81e8455e9934e1cc604fc2b2 P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:32.891618 7592 raft_consensus.cc:729] T 21d27cca81e8455e9934e1cc604fc2b2 P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c4821d9898d42799c5789d4d2da3f63, State: Initialized, Role: FOLLOWER
I20240503 01:59:32.891749 7592 consensus_queue.cc:260] T 21d27cca81e8455e9934e1cc604fc2b2 P 0c4821d9898d42799c5789d4d2da3f63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:32.892033 7592 ts_tablet_manager.cc:1398] T 21d27cca81e8455e9934e1cc604fc2b2 P 0c4821d9898d42799c5789d4d2da3f63: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:32.892283 7592 tablet_bootstrap.cc:492] T 2c8c328220e64e168c74fea7619f0c3b P 0c4821d9898d42799c5789d4d2da3f63: Bootstrap starting.
I20240503 01:59:32.892671 7596 tablet_bootstrap.cc:492] T a5926dae08184952bc693342c2c61ebf P b6b1d5677b8d47788b7968f64b3dac3c: No bootstrap required, opened a new log
I20240503 01:59:32.892766 7596 ts_tablet_manager.cc:1367] T a5926dae08184952bc693342c2c61ebf P b6b1d5677b8d47788b7968f64b3dac3c: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 01:59:32.893121 7596 raft_consensus.cc:348] T a5926dae08184952bc693342c2c61ebf P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:32.893183 7592 tablet_bootstrap.cc:654] T 2c8c328220e64e168c74fea7619f0c3b P 0c4821d9898d42799c5789d4d2da3f63: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:32.893208 7596 raft_consensus.cc:374] T a5926dae08184952bc693342c2c61ebf P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:32.893246 7596 raft_consensus.cc:729] T a5926dae08184952bc693342c2c61ebf P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6b1d5677b8d47788b7968f64b3dac3c, State: Initialized, Role: FOLLOWER
I20240503 01:59:32.893364 7596 consensus_queue.cc:260] T a5926dae08184952bc693342c2c61ebf P b6b1d5677b8d47788b7968f64b3dac3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:32.893626 7596 ts_tablet_manager.cc:1398] T a5926dae08184952bc693342c2c61ebf P b6b1d5677b8d47788b7968f64b3dac3c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:32.893790 7596 tablet_bootstrap.cc:492] T 2c8c328220e64e168c74fea7619f0c3b P b6b1d5677b8d47788b7968f64b3dac3c: Bootstrap starting.
I20240503 01:59:32.894676 7596 tablet_bootstrap.cc:654] T 2c8c328220e64e168c74fea7619f0c3b P b6b1d5677b8d47788b7968f64b3dac3c: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:32.895387 7592 tablet_bootstrap.cc:492] T 2c8c328220e64e168c74fea7619f0c3b P 0c4821d9898d42799c5789d4d2da3f63: No bootstrap required, opened a new log
I20240503 01:59:32.895475 7592 ts_tablet_manager.cc:1367] T 2c8c328220e64e168c74fea7619f0c3b P 0c4821d9898d42799c5789d4d2da3f63: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240503 01:59:32.895710 7596 tablet_bootstrap.cc:492] T 2c8c328220e64e168c74fea7619f0c3b P b6b1d5677b8d47788b7968f64b3dac3c: No bootstrap required, opened a new log
I20240503 01:59:32.895792 7596 ts_tablet_manager.cc:1367] T 2c8c328220e64e168c74fea7619f0c3b P b6b1d5677b8d47788b7968f64b3dac3c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:32.895942 7592 raft_consensus.cc:348] T 2c8c328220e64e168c74fea7619f0c3b P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } }
I20240503 01:59:32.896061 7592 raft_consensus.cc:374] T 2c8c328220e64e168c74fea7619f0c3b P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:32.896145 7592 raft_consensus.cc:729] T 2c8c328220e64e168c74fea7619f0c3b P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c4821d9898d42799c5789d4d2da3f63, State: Initialized, Role: FOLLOWER
I20240503 01:59:32.896183 7596 raft_consensus.cc:348] T 2c8c328220e64e168c74fea7619f0c3b P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } }
I20240503 01:59:32.896299 7596 raft_consensus.cc:374] T 2c8c328220e64e168c74fea7619f0c3b P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:32.896340 7596 raft_consensus.cc:729] T 2c8c328220e64e168c74fea7619f0c3b P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6b1d5677b8d47788b7968f64b3dac3c, State: Initialized, Role: FOLLOWER
I20240503 01:59:32.896287 7592 consensus_queue.cc:260] T 2c8c328220e64e168c74fea7619f0c3b P 0c4821d9898d42799c5789d4d2da3f63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } }
I20240503 01:59:32.896492 7596 consensus_queue.cc:260] T 2c8c328220e64e168c74fea7619f0c3b P b6b1d5677b8d47788b7968f64b3dac3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } }
I20240503 01:59:32.896864 7596 ts_tablet_manager.cc:1398] T 2c8c328220e64e168c74fea7619f0c3b P b6b1d5677b8d47788b7968f64b3dac3c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:32.896907 7592 ts_tablet_manager.cc:1398] T 2c8c328220e64e168c74fea7619f0c3b P 0c4821d9898d42799c5789d4d2da3f63: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:32.897682 7593 tablet_bootstrap.cc:654] T 21d27cca81e8455e9934e1cc604fc2b2 P 79163c69f6f04a9abd0208d74e1c61be: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:32.899190 7593 tablet_bootstrap.cc:492] T 21d27cca81e8455e9934e1cc604fc2b2 P 79163c69f6f04a9abd0208d74e1c61be: No bootstrap required, opened a new log
I20240503 01:59:32.899274 7593 ts_tablet_manager.cc:1367] T 21d27cca81e8455e9934e1cc604fc2b2 P 79163c69f6f04a9abd0208d74e1c61be: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20240503 01:59:32.899623 7593 raft_consensus.cc:348] T 21d27cca81e8455e9934e1cc604fc2b2 P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:32.899711 7593 raft_consensus.cc:374] T 21d27cca81e8455e9934e1cc604fc2b2 P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:32.899746 7593 raft_consensus.cc:729] T 21d27cca81e8455e9934e1cc604fc2b2 P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79163c69f6f04a9abd0208d74e1c61be, State: Initialized, Role: FOLLOWER
I20240503 01:59:32.899875 7593 consensus_queue.cc:260] T 21d27cca81e8455e9934e1cc604fc2b2 P 79163c69f6f04a9abd0208d74e1c61be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:32.900082 7593 ts_tablet_manager.cc:1398] T 21d27cca81e8455e9934e1cc604fc2b2 P 79163c69f6f04a9abd0208d74e1c61be: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:32.900231 7593 tablet_bootstrap.cc:492] T 2c8c328220e64e168c74fea7619f0c3b P 79163c69f6f04a9abd0208d74e1c61be: Bootstrap starting.
I20240503 01:59:32.901093 7593 tablet_bootstrap.cc:654] T 2c8c328220e64e168c74fea7619f0c3b P 79163c69f6f04a9abd0208d74e1c61be: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:32.902189 7593 tablet_bootstrap.cc:492] T 2c8c328220e64e168c74fea7619f0c3b P 79163c69f6f04a9abd0208d74e1c61be: No bootstrap required, opened a new log
I20240503 01:59:32.902276 7593 ts_tablet_manager.cc:1367] T 2c8c328220e64e168c74fea7619f0c3b P 79163c69f6f04a9abd0208d74e1c61be: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:32.902707 7593 raft_consensus.cc:348] T 2c8c328220e64e168c74fea7619f0c3b P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } }
I20240503 01:59:32.902810 7593 raft_consensus.cc:374] T 2c8c328220e64e168c74fea7619f0c3b P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:32.902853 7593 raft_consensus.cc:729] T 2c8c328220e64e168c74fea7619f0c3b P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79163c69f6f04a9abd0208d74e1c61be, State: Initialized, Role: FOLLOWER
I20240503 01:59:32.902969 7593 consensus_queue.cc:260] T 2c8c328220e64e168c74fea7619f0c3b P 79163c69f6f04a9abd0208d74e1c61be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } }
I20240503 01:59:32.903226 7593 ts_tablet_manager.cc:1398] T 2c8c328220e64e168c74fea7619f0c3b P 79163c69f6f04a9abd0208d74e1c61be: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:32.903445 7593 tablet_bootstrap.cc:492] T a5926dae08184952bc693342c2c61ebf P 79163c69f6f04a9abd0208d74e1c61be: Bootstrap starting.
I20240503 01:59:32.904273 7593 tablet_bootstrap.cc:654] T a5926dae08184952bc693342c2c61ebf P 79163c69f6f04a9abd0208d74e1c61be: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:32.905285 7593 tablet_bootstrap.cc:492] T a5926dae08184952bc693342c2c61ebf P 79163c69f6f04a9abd0208d74e1c61be: No bootstrap required, opened a new log
I20240503 01:59:32.905397 7593 ts_tablet_manager.cc:1367] T a5926dae08184952bc693342c2c61ebf P 79163c69f6f04a9abd0208d74e1c61be: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 01:59:32.905814 7593 raft_consensus.cc:348] T a5926dae08184952bc693342c2c61ebf P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:32.905894 7593 raft_consensus.cc:374] T a5926dae08184952bc693342c2c61ebf P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:32.905926 7593 raft_consensus.cc:729] T a5926dae08184952bc693342c2c61ebf P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79163c69f6f04a9abd0208d74e1c61be, State: Initialized, Role: FOLLOWER
I20240503 01:59:32.906041 7593 consensus_queue.cc:260] T a5926dae08184952bc693342c2c61ebf P 79163c69f6f04a9abd0208d74e1c61be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:32.906257 7593 ts_tablet_manager.cc:1398] T a5926dae08184952bc693342c2c61ebf P 79163c69f6f04a9abd0208d74e1c61be: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:32.914285 7570 raft_consensus.cc:482] T a5926dae08184952bc693342c2c61ebf P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:32.914355 7570 raft_consensus.cc:504] T a5926dae08184952bc693342c2c61ebf P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:32.914629 7570 leader_election.cc:288] T a5926dae08184952bc693342c2c61ebf P 79163c69f6f04a9abd0208d74e1c61be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b6b1d5677b8d47788b7968f64b3dac3c (127.0.99.132:46067), 4127249ba42644388d225d614c935a25 (127.0.99.129:35613)
I20240503 01:59:32.914947 6757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5926dae08184952bc693342c2c61ebf" candidate_uuid: "79163c69f6f04a9abd0208d74e1c61be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" is_pre_election: true
I20240503 01:59:32.914980 6365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5926dae08184952bc693342c2c61ebf" candidate_uuid: "79163c69f6f04a9abd0208d74e1c61be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4127249ba42644388d225d614c935a25" is_pre_election: true
I20240503 01:59:32.915071 6757 raft_consensus.cc:2454] T a5926dae08184952bc693342c2c61ebf P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79163c69f6f04a9abd0208d74e1c61be in term 0.
I20240503 01:59:32.915081 6365 raft_consensus.cc:2454] T a5926dae08184952bc693342c2c61ebf P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79163c69f6f04a9abd0208d74e1c61be in term 0.
I20240503 01:59:32.915360 6562 leader_election.cc:302] T a5926dae08184952bc693342c2c61ebf P 79163c69f6f04a9abd0208d74e1c61be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79163c69f6f04a9abd0208d74e1c61be, b6b1d5677b8d47788b7968f64b3dac3c; no voters:
I20240503 01:59:32.915510 7570 raft_consensus.cc:2781] T a5926dae08184952bc693342c2c61ebf P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:32.915570 7570 raft_consensus.cc:482] T a5926dae08184952bc693342c2c61ebf P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:32.915604 7570 raft_consensus.cc:3037] T a5926dae08184952bc693342c2c61ebf P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:32.916625 7570 raft_consensus.cc:504] T a5926dae08184952bc693342c2c61ebf P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:32.916960 7570 leader_election.cc:288] T a5926dae08184952bc693342c2c61ebf P 79163c69f6f04a9abd0208d74e1c61be [CANDIDATE]: Term 1 election: Requested vote from peers b6b1d5677b8d47788b7968f64b3dac3c (127.0.99.132:46067), 4127249ba42644388d225d614c935a25 (127.0.99.129:35613)
I20240503 01:59:32.917256 6757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5926dae08184952bc693342c2c61ebf" candidate_uuid: "79163c69f6f04a9abd0208d74e1c61be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6b1d5677b8d47788b7968f64b3dac3c"
I20240503 01:59:32.917280 6365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5926dae08184952bc693342c2c61ebf" candidate_uuid: "79163c69f6f04a9abd0208d74e1c61be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4127249ba42644388d225d614c935a25"
I20240503 01:59:32.917349 6757 raft_consensus.cc:3037] T a5926dae08184952bc693342c2c61ebf P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:32.917423 6365 raft_consensus.cc:3037] T a5926dae08184952bc693342c2c61ebf P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:32.918022 6757 raft_consensus.cc:2454] T a5926dae08184952bc693342c2c61ebf P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79163c69f6f04a9abd0208d74e1c61be in term 1.
I20240503 01:59:32.918130 6365 raft_consensus.cc:2454] T a5926dae08184952bc693342c2c61ebf P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79163c69f6f04a9abd0208d74e1c61be in term 1.
I20240503 01:59:32.918277 6562 leader_election.cc:302] T a5926dae08184952bc693342c2c61ebf P 79163c69f6f04a9abd0208d74e1c61be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79163c69f6f04a9abd0208d74e1c61be, b6b1d5677b8d47788b7968f64b3dac3c; no voters:
I20240503 01:59:32.918495 7570 raft_consensus.cc:2781] T a5926dae08184952bc693342c2c61ebf P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:32.918576 7570 raft_consensus.cc:686] T a5926dae08184952bc693342c2c61ebf P 79163c69f6f04a9abd0208d74e1c61be [term 1 LEADER]: Becoming Leader. State: Replica: 79163c69f6f04a9abd0208d74e1c61be, State: Running, Role: LEADER
I20240503 01:59:32.918717 7570 consensus_queue.cc:237] T a5926dae08184952bc693342c2c61ebf P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:32.920105 6215 catalog_manager.cc:5484] T a5926dae08184952bc693342c2c61ebf P 79163c69f6f04a9abd0208d74e1c61be reported cstate change: term changed from 0 to 1, leader changed from <none> to 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131). New cstate: current_term: 1 leader_uuid: "79163c69f6f04a9abd0208d74e1c61be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:32.937758 7448 raft_consensus.cc:482] T 2c8c328220e64e168c74fea7619f0c3b P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:32.937868 7448 raft_consensus.cc:504] T 2c8c328220e64e168c74fea7619f0c3b P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } }
I20240503 01:59:32.938246 7448 leader_election.cc:288] T 2c8c328220e64e168c74fea7619f0c3b P b6b1d5677b8d47788b7968f64b3dac3c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131:44005), 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130:34189)
I20240503 01:59:32.938584 6627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c8c328220e64e168c74fea7619f0c3b" candidate_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79163c69f6f04a9abd0208d74e1c61be" is_pre_election: true
I20240503 01:59:32.938657 6494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c8c328220e64e168c74fea7619f0c3b" candidate_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c4821d9898d42799c5789d4d2da3f63" is_pre_election: true
I20240503 01:59:32.938705 6627 raft_consensus.cc:2454] T 2c8c328220e64e168c74fea7619f0c3b P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6b1d5677b8d47788b7968f64b3dac3c in term 0.
I20240503 01:59:32.938791 6494 raft_consensus.cc:2454] T 2c8c328220e64e168c74fea7619f0c3b P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6b1d5677b8d47788b7968f64b3dac3c in term 0.
I20240503 01:59:32.939014 6692 leader_election.cc:302] T 2c8c328220e64e168c74fea7619f0c3b P b6b1d5677b8d47788b7968f64b3dac3c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79163c69f6f04a9abd0208d74e1c61be, b6b1d5677b8d47788b7968f64b3dac3c; no voters:
I20240503 01:59:32.939177 7448 raft_consensus.cc:2781] T 2c8c328220e64e168c74fea7619f0c3b P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:32.939237 7448 raft_consensus.cc:482] T 2c8c328220e64e168c74fea7619f0c3b P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:32.939280 7448 raft_consensus.cc:3037] T 2c8c328220e64e168c74fea7619f0c3b P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:32.939988 7448 raft_consensus.cc:504] T 2c8c328220e64e168c74fea7619f0c3b P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } }
I20240503 01:59:32.940272 7448 leader_election.cc:288] T 2c8c328220e64e168c74fea7619f0c3b P b6b1d5677b8d47788b7968f64b3dac3c [CANDIDATE]: Term 1 election: Requested vote from peers 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131:44005), 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130:34189)
I20240503 01:59:32.940548 6627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c8c328220e64e168c74fea7619f0c3b" candidate_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79163c69f6f04a9abd0208d74e1c61be"
I20240503 01:59:32.940676 6627 raft_consensus.cc:3037] T 2c8c328220e64e168c74fea7619f0c3b P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:32.940603 6494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c8c328220e64e168c74fea7619f0c3b" candidate_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c4821d9898d42799c5789d4d2da3f63"
I20240503 01:59:32.940760 6494 raft_consensus.cc:3037] T 2c8c328220e64e168c74fea7619f0c3b P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:32.941356 6627 raft_consensus.cc:2454] T 2c8c328220e64e168c74fea7619f0c3b P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6b1d5677b8d47788b7968f64b3dac3c in term 1.
I20240503 01:59:32.941515 6494 raft_consensus.cc:2454] T 2c8c328220e64e168c74fea7619f0c3b P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6b1d5677b8d47788b7968f64b3dac3c in term 1.
I20240503 01:59:32.941615 6692 leader_election.cc:302] T 2c8c328220e64e168c74fea7619f0c3b P b6b1d5677b8d47788b7968f64b3dac3c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79163c69f6f04a9abd0208d74e1c61be, b6b1d5677b8d47788b7968f64b3dac3c; no voters:
I20240503 01:59:32.941797 7448 raft_consensus.cc:2781] T 2c8c328220e64e168c74fea7619f0c3b P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:32.941967 7448 raft_consensus.cc:686] T 2c8c328220e64e168c74fea7619f0c3b P b6b1d5677b8d47788b7968f64b3dac3c [term 1 LEADER]: Becoming Leader. State: Replica: b6b1d5677b8d47788b7968f64b3dac3c, State: Running, Role: LEADER
I20240503 01:59:32.942099 7448 consensus_queue.cc:237] T 2c8c328220e64e168c74fea7619f0c3b P b6b1d5677b8d47788b7968f64b3dac3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } }
I20240503 01:59:32.943444 6215 catalog_manager.cc:5484] T 2c8c328220e64e168c74fea7619f0c3b P b6b1d5677b8d47788b7968f64b3dac3c reported cstate change: term changed from 0 to 1, leader changed from <none> to b6b1d5677b8d47788b7968f64b3dac3c (127.0.99.132). New cstate: current_term: 1 leader_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } health_report { overall_health: HEALTHY } } }
I20240503 01:59:33.117471 7570 raft_consensus.cc:482] T 21d27cca81e8455e9934e1cc604fc2b2 P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:33.117630 7570 raft_consensus.cc:504] T 21d27cca81e8455e9934e1cc604fc2b2 P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:33.118013 7570 leader_election.cc:288] T 21d27cca81e8455e9934e1cc604fc2b2 P 79163c69f6f04a9abd0208d74e1c61be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130:34189), 4127249ba42644388d225d614c935a25 (127.0.99.129:35613)
I20240503 01:59:33.118376 6494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21d27cca81e8455e9934e1cc604fc2b2" candidate_uuid: "79163c69f6f04a9abd0208d74e1c61be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c4821d9898d42799c5789d4d2da3f63" is_pre_election: true
I20240503 01:59:33.118379 6365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21d27cca81e8455e9934e1cc604fc2b2" candidate_uuid: "79163c69f6f04a9abd0208d74e1c61be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4127249ba42644388d225d614c935a25" is_pre_election: true
I20240503 01:59:33.118516 6365 raft_consensus.cc:2454] T 21d27cca81e8455e9934e1cc604fc2b2 P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79163c69f6f04a9abd0208d74e1c61be in term 0.
I20240503 01:59:33.118516 6494 raft_consensus.cc:2454] T 21d27cca81e8455e9934e1cc604fc2b2 P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79163c69f6f04a9abd0208d74e1c61be in term 0.
I20240503 01:59:33.118826 6563 leader_election.cc:302] T 21d27cca81e8455e9934e1cc604fc2b2 P 79163c69f6f04a9abd0208d74e1c61be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c4821d9898d42799c5789d4d2da3f63, 79163c69f6f04a9abd0208d74e1c61be; no voters:
I20240503 01:59:33.118980 7570 raft_consensus.cc:2781] T 21d27cca81e8455e9934e1cc604fc2b2 P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:33.119037 7570 raft_consensus.cc:482] T 21d27cca81e8455e9934e1cc604fc2b2 P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:33.119076 7570 raft_consensus.cc:3037] T 21d27cca81e8455e9934e1cc604fc2b2 P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:33.119832 7570 raft_consensus.cc:504] T 21d27cca81e8455e9934e1cc604fc2b2 P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:33.120115 7570 leader_election.cc:288] T 21d27cca81e8455e9934e1cc604fc2b2 P 79163c69f6f04a9abd0208d74e1c61be [CANDIDATE]: Term 1 election: Requested vote from peers 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130:34189), 4127249ba42644388d225d614c935a25 (127.0.99.129:35613)
I20240503 01:59:33.120471 6494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21d27cca81e8455e9934e1cc604fc2b2" candidate_uuid: "79163c69f6f04a9abd0208d74e1c61be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c4821d9898d42799c5789d4d2da3f63"
I20240503 01:59:33.120491 6365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21d27cca81e8455e9934e1cc604fc2b2" candidate_uuid: "79163c69f6f04a9abd0208d74e1c61be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4127249ba42644388d225d614c935a25"
I20240503 01:59:33.120601 6365 raft_consensus.cc:3037] T 21d27cca81e8455e9934e1cc604fc2b2 P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:33.120622 6494 raft_consensus.cc:3037] T 21d27cca81e8455e9934e1cc604fc2b2 P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:33.121559 6494 raft_consensus.cc:2454] T 21d27cca81e8455e9934e1cc604fc2b2 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79163c69f6f04a9abd0208d74e1c61be in term 1.
I20240503 01:59:33.121613 6365 raft_consensus.cc:2454] T 21d27cca81e8455e9934e1cc604fc2b2 P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79163c69f6f04a9abd0208d74e1c61be in term 1.
I20240503 01:59:33.121897 6561 leader_election.cc:302] T 21d27cca81e8455e9934e1cc604fc2b2 P 79163c69f6f04a9abd0208d74e1c61be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4127249ba42644388d225d614c935a25, 79163c69f6f04a9abd0208d74e1c61be; no voters:
I20240503 01:59:33.122067 7570 raft_consensus.cc:2781] T 21d27cca81e8455e9934e1cc604fc2b2 P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:33.122145 7570 raft_consensus.cc:686] T 21d27cca81e8455e9934e1cc604fc2b2 P 79163c69f6f04a9abd0208d74e1c61be [term 1 LEADER]: Becoming Leader. State: Replica: 79163c69f6f04a9abd0208d74e1c61be, State: Running, Role: LEADER
I20240503 01:59:33.122282 7570 consensus_queue.cc:237] T 21d27cca81e8455e9934e1cc604fc2b2 P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:33.123795 6215 catalog_manager.cc:5484] T 21d27cca81e8455e9934e1cc604fc2b2 P 79163c69f6f04a9abd0208d74e1c61be reported cstate change: term changed from 0 to 1, leader changed from <none> to 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131). New cstate: current_term: 1 leader_uuid: "79163c69f6f04a9abd0208d74e1c61be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } health_report { overall_health: UNKNOWN } } }
W20240503 01:59:33.273866 7609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:33.274096 7609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:33.277108 7609 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240503 01:59:33.364357 6365 raft_consensus.cc:1261] T 21d27cca81e8455e9934e1cc604fc2b2 P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:33.364432 6494 raft_consensus.cc:1261] T 21d27cca81e8455e9934e1cc604fc2b2 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:33.364768 6627 raft_consensus.cc:1261] T 2c8c328220e64e168c74fea7619f0c3b P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Refusing update from remote peer b6b1d5677b8d47788b7968f64b3dac3c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:33.364797 7570 consensus_queue.cc:1035] T 21d27cca81e8455e9934e1cc604fc2b2 P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:33.365097 6495 raft_consensus.cc:1261] T 2c8c328220e64e168c74fea7619f0c3b P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Refusing update from remote peer b6b1d5677b8d47788b7968f64b3dac3c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:33.365470 7606 consensus_queue.cc:1035] T 2c8c328220e64e168c74fea7619f0c3b P b6b1d5677b8d47788b7968f64b3dac3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:33.365610 7448 consensus_queue.cc:1035] T 2c8c328220e64e168c74fea7619f0c3b P b6b1d5677b8d47788b7968f64b3dac3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:33.366175 7570 consensus_queue.cc:1035] T 21d27cca81e8455e9934e1cc604fc2b2 P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:33.368451 6364 raft_consensus.cc:1261] T a5926dae08184952bc693342c2c61ebf P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:33.368803 6757 raft_consensus.cc:1261] T a5926dae08184952bc693342c2c61ebf P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:33.368837 7570 consensus_queue.cc:1035] T a5926dae08184952bc693342c2c61ebf P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:33.371748 7489 consensus_queue.cc:1035] T a5926dae08184952bc693342c2c61ebf P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 116.299 ms
time per row: 0.0581493 ms
I20240503 01:59:33.434432 6220 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44084:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:59:33.439927 6605 tablet_service.cc:1467] Processing CreateTablet for tablet 5b599bd1fea6409fbe485c7a35c0891c (DEFAULT_TABLE table=rebalance_test_table_4 [id=d52e5f2a44ec4f12b7ef6d85bd14d4ac]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:33.439913 6476 tablet_service.cc:1467] Processing CreateTablet for tablet 5b599bd1fea6409fbe485c7a35c0891c (DEFAULT_TABLE table=rebalance_test_table_4 [id=d52e5f2a44ec4f12b7ef6d85bd14d4ac]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:33.440224 6476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b599bd1fea6409fbe485c7a35c0891c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:33.440224 6605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b599bd1fea6409fbe485c7a35c0891c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:33.440382 6343 tablet_service.cc:1467] Processing CreateTablet for tablet 2ac8004211e64c3481e75377092fa93b (DEFAULT_TABLE table=rebalance_test_table_4 [id=d52e5f2a44ec4f12b7ef6d85bd14d4ac]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:33.440717 6343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ac8004211e64c3481e75377092fa93b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:33.442653 6344 tablet_service.cc:1467] Processing CreateTablet for tablet 5b599bd1fea6409fbe485c7a35c0891c (DEFAULT_TABLE table=rebalance_test_table_4 [id=d52e5f2a44ec4f12b7ef6d85bd14d4ac]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:33.442781 7672 tablet_bootstrap.cc:492] T 5b599bd1fea6409fbe485c7a35c0891c P 79163c69f6f04a9abd0208d74e1c61be: Bootstrap starting.
I20240503 01:59:33.442876 6344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b599bd1fea6409fbe485c7a35c0891c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:33.442847 6605 tablet_service.cc:1467] Processing CreateTablet for tablet 679fc86ea2ad4bd6a9160fddf4c586d7 (DEFAULT_TABLE table=rebalance_test_table_4 [id=d52e5f2a44ec4f12b7ef6d85bd14d4ac]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:33.443095 6605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 679fc86ea2ad4bd6a9160fddf4c586d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:33.443655 7672 tablet_bootstrap.cc:654] T 5b599bd1fea6409fbe485c7a35c0891c P 79163c69f6f04a9abd0208d74e1c61be: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:33.443807 6343 tablet_service.cc:1467] Processing CreateTablet for tablet 679fc86ea2ad4bd6a9160fddf4c586d7 (DEFAULT_TABLE table=rebalance_test_table_4 [id=d52e5f2a44ec4f12b7ef6d85bd14d4ac]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:33.444053 6343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 679fc86ea2ad4bd6a9160fddf4c586d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:33.444476 6737 tablet_service.cc:1467] Processing CreateTablet for tablet 2ac8004211e64c3481e75377092fa93b (DEFAULT_TABLE table=rebalance_test_table_4 [id=d52e5f2a44ec4f12b7ef6d85bd14d4ac]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:33.444543 6738 tablet_service.cc:1467] Processing CreateTablet for tablet 679fc86ea2ad4bd6a9160fddf4c586d7 (DEFAULT_TABLE table=rebalance_test_table_4 [id=d52e5f2a44ec4f12b7ef6d85bd14d4ac]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:33.444765 6737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ac8004211e64c3481e75377092fa93b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:33.445166 6738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 679fc86ea2ad4bd6a9160fddf4c586d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:33.445788 6476 tablet_service.cc:1467] Processing CreateTablet for tablet 2ac8004211e64c3481e75377092fa93b (DEFAULT_TABLE table=rebalance_test_table_4 [id=d52e5f2a44ec4f12b7ef6d85bd14d4ac]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:33.446023 6476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ac8004211e64c3481e75377092fa93b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:33.447947 7674 tablet_bootstrap.cc:492] T 2ac8004211e64c3481e75377092fa93b P b6b1d5677b8d47788b7968f64b3dac3c: Bootstrap starting.
I20240503 01:59:33.448048 7671 tablet_bootstrap.cc:492] T 5b599bd1fea6409fbe485c7a35c0891c P 0c4821d9898d42799c5789d4d2da3f63: Bootstrap starting.
I20240503 01:59:33.448720 7674 tablet_bootstrap.cc:654] T 2ac8004211e64c3481e75377092fa93b P b6b1d5677b8d47788b7968f64b3dac3c: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:33.448848 7671 tablet_bootstrap.cc:654] T 5b599bd1fea6409fbe485c7a35c0891c P 0c4821d9898d42799c5789d4d2da3f63: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:33.449280 7672 tablet_bootstrap.cc:492] T 5b599bd1fea6409fbe485c7a35c0891c P 79163c69f6f04a9abd0208d74e1c61be: No bootstrap required, opened a new log
I20240503 01:59:33.449426 7672 ts_tablet_manager.cc:1367] T 5b599bd1fea6409fbe485c7a35c0891c P 79163c69f6f04a9abd0208d74e1c61be: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 01:59:33.449535 7670 tablet_bootstrap.cc:492] T 2ac8004211e64c3481e75377092fa93b P 4127249ba42644388d225d614c935a25: Bootstrap starting.
I20240503 01:59:33.449949 7674 tablet_bootstrap.cc:492] T 2ac8004211e64c3481e75377092fa93b P b6b1d5677b8d47788b7968f64b3dac3c: No bootstrap required, opened a new log
I20240503 01:59:33.449908 7672 raft_consensus.cc:348] T 5b599bd1fea6409fbe485c7a35c0891c P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:33.450013 7672 raft_consensus.cc:374] T 5b599bd1fea6409fbe485c7a35c0891c P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:33.450037 7674 ts_tablet_manager.cc:1367] T 2ac8004211e64c3481e75377092fa93b P b6b1d5677b8d47788b7968f64b3dac3c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 01:59:33.450083 7672 raft_consensus.cc:729] T 5b599bd1fea6409fbe485c7a35c0891c P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79163c69f6f04a9abd0208d74e1c61be, State: Initialized, Role: FOLLOWER
I20240503 01:59:33.450258 7672 consensus_queue.cc:260] T 5b599bd1fea6409fbe485c7a35c0891c P 79163c69f6f04a9abd0208d74e1c61be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:33.450419 7670 tablet_bootstrap.cc:654] T 2ac8004211e64c3481e75377092fa93b P 4127249ba42644388d225d614c935a25: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:33.450606 7672 ts_tablet_manager.cc:1398] T 5b599bd1fea6409fbe485c7a35c0891c P 79163c69f6f04a9abd0208d74e1c61be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:33.450835 7672 tablet_bootstrap.cc:492] T 679fc86ea2ad4bd6a9160fddf4c586d7 P 79163c69f6f04a9abd0208d74e1c61be: Bootstrap starting.
I20240503 01:59:33.450881 7674 raft_consensus.cc:348] T 2ac8004211e64c3481e75377092fa93b P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } }
I20240503 01:59:33.450992 7674 raft_consensus.cc:374] T 2ac8004211e64c3481e75377092fa93b P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:33.451028 7674 raft_consensus.cc:729] T 2ac8004211e64c3481e75377092fa93b P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6b1d5677b8d47788b7968f64b3dac3c, State: Initialized, Role: FOLLOWER
I20240503 01:59:33.451151 7674 consensus_queue.cc:260] T 2ac8004211e64c3481e75377092fa93b P b6b1d5677b8d47788b7968f64b3dac3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } }
I20240503 01:59:33.451440 7674 ts_tablet_manager.cc:1398] T 2ac8004211e64c3481e75377092fa93b P b6b1d5677b8d47788b7968f64b3dac3c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:33.451649 7674 tablet_bootstrap.cc:492] T 679fc86ea2ad4bd6a9160fddf4c586d7 P b6b1d5677b8d47788b7968f64b3dac3c: Bootstrap starting.
I20240503 01:59:33.451764 7672 tablet_bootstrap.cc:654] T 679fc86ea2ad4bd6a9160fddf4c586d7 P 79163c69f6f04a9abd0208d74e1c61be: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:33.452558 7674 tablet_bootstrap.cc:654] T 679fc86ea2ad4bd6a9160fddf4c586d7 P b6b1d5677b8d47788b7968f64b3dac3c: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:33.453524 7672 tablet_bootstrap.cc:492] T 679fc86ea2ad4bd6a9160fddf4c586d7 P 79163c69f6f04a9abd0208d74e1c61be: No bootstrap required, opened a new log
I20240503 01:59:33.453593 7672 ts_tablet_manager.cc:1367] T 679fc86ea2ad4bd6a9160fddf4c586d7 P 79163c69f6f04a9abd0208d74e1c61be: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:33.453681 7674 tablet_bootstrap.cc:492] T 679fc86ea2ad4bd6a9160fddf4c586d7 P b6b1d5677b8d47788b7968f64b3dac3c: No bootstrap required, opened a new log
I20240503 01:59:33.453763 7674 ts_tablet_manager.cc:1367] T 679fc86ea2ad4bd6a9160fddf4c586d7 P b6b1d5677b8d47788b7968f64b3dac3c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 01:59:33.453969 7672 raft_consensus.cc:348] T 679fc86ea2ad4bd6a9160fddf4c586d7 P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:33.454049 7672 raft_consensus.cc:374] T 679fc86ea2ad4bd6a9160fddf4c586d7 P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:33.454069 7672 raft_consensus.cc:729] T 679fc86ea2ad4bd6a9160fddf4c586d7 P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79163c69f6f04a9abd0208d74e1c61be, State: Initialized, Role: FOLLOWER
I20240503 01:59:33.454089 7674 raft_consensus.cc:348] T 679fc86ea2ad4bd6a9160fddf4c586d7 P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:33.454197 7674 raft_consensus.cc:374] T 679fc86ea2ad4bd6a9160fddf4c586d7 P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:33.454176 7672 consensus_queue.cc:260] T 679fc86ea2ad4bd6a9160fddf4c586d7 P 79163c69f6f04a9abd0208d74e1c61be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:33.454241 7674 raft_consensus.cc:729] T 679fc86ea2ad4bd6a9160fddf4c586d7 P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6b1d5677b8d47788b7968f64b3dac3c, State: Initialized, Role: FOLLOWER
I20240503 01:59:33.454362 7674 consensus_queue.cc:260] T 679fc86ea2ad4bd6a9160fddf4c586d7 P b6b1d5677b8d47788b7968f64b3dac3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:33.454424 7672 ts_tablet_manager.cc:1398] T 679fc86ea2ad4bd6a9160fddf4c586d7 P 79163c69f6f04a9abd0208d74e1c61be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:33.454617 7674 ts_tablet_manager.cc:1398] T 679fc86ea2ad4bd6a9160fddf4c586d7 P b6b1d5677b8d47788b7968f64b3dac3c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:33.455133 7671 tablet_bootstrap.cc:492] T 5b599bd1fea6409fbe485c7a35c0891c P 0c4821d9898d42799c5789d4d2da3f63: No bootstrap required, opened a new log
I20240503 01:59:33.455235 7671 ts_tablet_manager.cc:1367] T 5b599bd1fea6409fbe485c7a35c0891c P 0c4821d9898d42799c5789d4d2da3f63: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20240503 01:59:33.455698 7671 raft_consensus.cc:348] T 5b599bd1fea6409fbe485c7a35c0891c P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:33.455816 7671 raft_consensus.cc:374] T 5b599bd1fea6409fbe485c7a35c0891c P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:33.455852 7671 raft_consensus.cc:729] T 5b599bd1fea6409fbe485c7a35c0891c P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c4821d9898d42799c5789d4d2da3f63, State: Initialized, Role: FOLLOWER
I20240503 01:59:33.456393 7671 consensus_queue.cc:260] T 5b599bd1fea6409fbe485c7a35c0891c P 0c4821d9898d42799c5789d4d2da3f63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:33.456787 7671 ts_tablet_manager.cc:1398] T 5b599bd1fea6409fbe485c7a35c0891c P 0c4821d9898d42799c5789d4d2da3f63: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:33.456995 7671 tablet_bootstrap.cc:492] T 2ac8004211e64c3481e75377092fa93b P 0c4821d9898d42799c5789d4d2da3f63: Bootstrap starting.
I20240503 01:59:33.458025 7670 tablet_bootstrap.cc:492] T 2ac8004211e64c3481e75377092fa93b P 4127249ba42644388d225d614c935a25: No bootstrap required, opened a new log
I20240503 01:59:33.458124 7670 ts_tablet_manager.cc:1367] T 2ac8004211e64c3481e75377092fa93b P 4127249ba42644388d225d614c935a25: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20240503 01:59:33.458894 7671 tablet_bootstrap.cc:654] T 2ac8004211e64c3481e75377092fa93b P 0c4821d9898d42799c5789d4d2da3f63: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:33.459869 7670 raft_consensus.cc:348] T 2ac8004211e64c3481e75377092fa93b P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } }
I20240503 01:59:33.459961 7670 raft_consensus.cc:374] T 2ac8004211e64c3481e75377092fa93b P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:33.459983 7670 raft_consensus.cc:729] T 2ac8004211e64c3481e75377092fa93b P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4127249ba42644388d225d614c935a25, State: Initialized, Role: FOLLOWER
I20240503 01:59:33.460072 7670 consensus_queue.cc:260] T 2ac8004211e64c3481e75377092fa93b P 4127249ba42644388d225d614c935a25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } }
I20240503 01:59:33.460273 7670 ts_tablet_manager.cc:1398] T 2ac8004211e64c3481e75377092fa93b P 4127249ba42644388d225d614c935a25: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:33.460443 7670 tablet_bootstrap.cc:492] T 5b599bd1fea6409fbe485c7a35c0891c P 4127249ba42644388d225d614c935a25: Bootstrap starting.
I20240503 01:59:33.461042 7671 tablet_bootstrap.cc:492] T 2ac8004211e64c3481e75377092fa93b P 0c4821d9898d42799c5789d4d2da3f63: No bootstrap required, opened a new log
I20240503 01:59:33.461156 7671 ts_tablet_manager.cc:1367] T 2ac8004211e64c3481e75377092fa93b P 0c4821d9898d42799c5789d4d2da3f63: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 01:59:33.461323 7670 tablet_bootstrap.cc:654] T 5b599bd1fea6409fbe485c7a35c0891c P 4127249ba42644388d225d614c935a25: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:33.461632 7671 raft_consensus.cc:348] T 2ac8004211e64c3481e75377092fa93b P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } }
I20240503 01:59:33.461764 7671 raft_consensus.cc:374] T 2ac8004211e64c3481e75377092fa93b P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:33.461827 7671 raft_consensus.cc:729] T 2ac8004211e64c3481e75377092fa93b P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c4821d9898d42799c5789d4d2da3f63, State: Initialized, Role: FOLLOWER
I20240503 01:59:33.461952 7671 consensus_queue.cc:260] T 2ac8004211e64c3481e75377092fa93b P 0c4821d9898d42799c5789d4d2da3f63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } }
I20240503 01:59:33.462208 7671 ts_tablet_manager.cc:1398] T 2ac8004211e64c3481e75377092fa93b P 0c4821d9898d42799c5789d4d2da3f63: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:33.462810 7670 tablet_bootstrap.cc:492] T 5b599bd1fea6409fbe485c7a35c0891c P 4127249ba42644388d225d614c935a25: No bootstrap required, opened a new log
I20240503 01:59:33.462900 7670 ts_tablet_manager.cc:1367] T 5b599bd1fea6409fbe485c7a35c0891c P 4127249ba42644388d225d614c935a25: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 01:59:33.463382 7670 raft_consensus.cc:348] T 5b599bd1fea6409fbe485c7a35c0891c P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:33.463507 7670 raft_consensus.cc:374] T 5b599bd1fea6409fbe485c7a35c0891c P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:33.463577 7670 raft_consensus.cc:729] T 5b599bd1fea6409fbe485c7a35c0891c P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4127249ba42644388d225d614c935a25, State: Initialized, Role: FOLLOWER
I20240503 01:59:33.463732 7670 consensus_queue.cc:260] T 5b599bd1fea6409fbe485c7a35c0891c P 4127249ba42644388d225d614c935a25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:33.464012 7670 ts_tablet_manager.cc:1398] T 5b599bd1fea6409fbe485c7a35c0891c P 4127249ba42644388d225d614c935a25: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:33.464211 7670 tablet_bootstrap.cc:492] T 679fc86ea2ad4bd6a9160fddf4c586d7 P 4127249ba42644388d225d614c935a25: Bootstrap starting.
I20240503 01:59:33.465096 7670 tablet_bootstrap.cc:654] T 679fc86ea2ad4bd6a9160fddf4c586d7 P 4127249ba42644388d225d614c935a25: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:33.466130 7670 tablet_bootstrap.cc:492] T 679fc86ea2ad4bd6a9160fddf4c586d7 P 4127249ba42644388d225d614c935a25: No bootstrap required, opened a new log
I20240503 01:59:33.466235 7670 ts_tablet_manager.cc:1367] T 679fc86ea2ad4bd6a9160fddf4c586d7 P 4127249ba42644388d225d614c935a25: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:33.466625 7670 raft_consensus.cc:348] T 679fc86ea2ad4bd6a9160fddf4c586d7 P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:33.466710 7670 raft_consensus.cc:374] T 679fc86ea2ad4bd6a9160fddf4c586d7 P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:33.466737 7670 raft_consensus.cc:729] T 679fc86ea2ad4bd6a9160fddf4c586d7 P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4127249ba42644388d225d614c935a25, State: Initialized, Role: FOLLOWER
I20240503 01:59:33.466892 7670 consensus_queue.cc:260] T 679fc86ea2ad4bd6a9160fddf4c586d7 P 4127249ba42644388d225d614c935a25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:33.467162 7670 ts_tablet_manager.cc:1398] T 679fc86ea2ad4bd6a9160fddf4c586d7 P 4127249ba42644388d225d614c935a25: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:33.500274 7361 raft_consensus.cc:482] T 2ac8004211e64c3481e75377092fa93b P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:33.500416 7361 raft_consensus.cc:504] T 2ac8004211e64c3481e75377092fa93b P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } }
I20240503 01:59:33.500870 7361 leader_election.cc:288] T 2ac8004211e64c3481e75377092fa93b P 0c4821d9898d42799c5789d4d2da3f63 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b6b1d5677b8d47788b7968f64b3dac3c (127.0.99.132:46067), 4127249ba42644388d225d614c935a25 (127.0.99.129:35613)
I20240503 01:59:33.501279 6757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ac8004211e64c3481e75377092fa93b" candidate_uuid: "0c4821d9898d42799c5789d4d2da3f63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" is_pre_election: true
I20240503 01:59:33.501278 6364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ac8004211e64c3481e75377092fa93b" candidate_uuid: "0c4821d9898d42799c5789d4d2da3f63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4127249ba42644388d225d614c935a25" is_pre_election: true
I20240503 01:59:33.501437 6364 raft_consensus.cc:2454] T 2ac8004211e64c3481e75377092fa93b P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c4821d9898d42799c5789d4d2da3f63 in term 0.
I20240503 01:59:33.501435 6757 raft_consensus.cc:2454] T 2ac8004211e64c3481e75377092fa93b P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c4821d9898d42799c5789d4d2da3f63 in term 0.
I20240503 01:59:33.501714 6430 leader_election.cc:302] T 2ac8004211e64c3481e75377092fa93b P 0c4821d9898d42799c5789d4d2da3f63 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c4821d9898d42799c5789d4d2da3f63, 4127249ba42644388d225d614c935a25; no voters:
I20240503 01:59:33.501876 7361 raft_consensus.cc:2781] T 2ac8004211e64c3481e75377092fa93b P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:33.501942 7361 raft_consensus.cc:482] T 2ac8004211e64c3481e75377092fa93b P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:33.501974 7361 raft_consensus.cc:3037] T 2ac8004211e64c3481e75377092fa93b P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:33.502801 7361 raft_consensus.cc:504] T 2ac8004211e64c3481e75377092fa93b P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } }
I20240503 01:59:33.503109 7361 leader_election.cc:288] T 2ac8004211e64c3481e75377092fa93b P 0c4821d9898d42799c5789d4d2da3f63 [CANDIDATE]: Term 1 election: Requested vote from peers b6b1d5677b8d47788b7968f64b3dac3c (127.0.99.132:46067), 4127249ba42644388d225d614c935a25 (127.0.99.129:35613)
I20240503 01:59:33.503433 6757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ac8004211e64c3481e75377092fa93b" candidate_uuid: "0c4821d9898d42799c5789d4d2da3f63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6b1d5677b8d47788b7968f64b3dac3c"
I20240503 01:59:33.503538 6757 raft_consensus.cc:3037] T 2ac8004211e64c3481e75377092fa93b P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:33.503690 6364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ac8004211e64c3481e75377092fa93b" candidate_uuid: "0c4821d9898d42799c5789d4d2da3f63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4127249ba42644388d225d614c935a25"
I20240503 01:59:33.503800 6364 raft_consensus.cc:3037] T 2ac8004211e64c3481e75377092fa93b P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:33.504315 6757 raft_consensus.cc:2454] T 2ac8004211e64c3481e75377092fa93b P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c4821d9898d42799c5789d4d2da3f63 in term 1.
I20240503 01:59:33.504604 6431 leader_election.cc:302] T 2ac8004211e64c3481e75377092fa93b P 0c4821d9898d42799c5789d4d2da3f63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c4821d9898d42799c5789d4d2da3f63, b6b1d5677b8d47788b7968f64b3dac3c; no voters:
I20240503 01:59:33.504762 6364 raft_consensus.cc:2454] T 2ac8004211e64c3481e75377092fa93b P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c4821d9898d42799c5789d4d2da3f63 in term 1.
I20240503 01:59:33.504814 7361 raft_consensus.cc:2781] T 2ac8004211e64c3481e75377092fa93b P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:33.505025 7361 raft_consensus.cc:686] T 2ac8004211e64c3481e75377092fa93b P 0c4821d9898d42799c5789d4d2da3f63 [term 1 LEADER]: Becoming Leader. State: Replica: 0c4821d9898d42799c5789d4d2da3f63, State: Running, Role: LEADER
I20240503 01:59:33.505187 7361 consensus_queue.cc:237] T 2ac8004211e64c3481e75377092fa93b P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } }
I20240503 01:59:33.506759 6220 catalog_manager.cc:5484] T 2ac8004211e64c3481e75377092fa93b P 0c4821d9898d42799c5789d4d2da3f63 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "0c4821d9898d42799c5789d4d2da3f63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } health_report { overall_health: HEALTHY } } }
I20240503 01:59:33.566553 7646 raft_consensus.cc:482] T 679fc86ea2ad4bd6a9160fddf4c586d7 P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:33.566696 7646 raft_consensus.cc:504] T 679fc86ea2ad4bd6a9160fddf4c586d7 P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:33.567126 7646 leader_election.cc:288] T 679fc86ea2ad4bd6a9160fddf4c586d7 P 79163c69f6f04a9abd0208d74e1c61be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b6b1d5677b8d47788b7968f64b3dac3c (127.0.99.132:46067), 4127249ba42644388d225d614c935a25 (127.0.99.129:35613)
I20240503 01:59:33.567559 6757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "679fc86ea2ad4bd6a9160fddf4c586d7" candidate_uuid: "79163c69f6f04a9abd0208d74e1c61be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" is_pre_election: true
I20240503 01:59:33.567564 6364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "679fc86ea2ad4bd6a9160fddf4c586d7" candidate_uuid: "79163c69f6f04a9abd0208d74e1c61be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4127249ba42644388d225d614c935a25" is_pre_election: true
I20240503 01:59:33.567696 6364 raft_consensus.cc:2454] T 679fc86ea2ad4bd6a9160fddf4c586d7 P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79163c69f6f04a9abd0208d74e1c61be in term 0.
I20240503 01:59:33.567716 6757 raft_consensus.cc:2454] T 679fc86ea2ad4bd6a9160fddf4c586d7 P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79163c69f6f04a9abd0208d74e1c61be in term 0.
I20240503 01:59:33.568010 6561 leader_election.cc:302] T 679fc86ea2ad4bd6a9160fddf4c586d7 P 79163c69f6f04a9abd0208d74e1c61be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4127249ba42644388d225d614c935a25, 79163c69f6f04a9abd0208d74e1c61be; no voters:
I20240503 01:59:33.568179 7646 raft_consensus.cc:2781] T 679fc86ea2ad4bd6a9160fddf4c586d7 P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:33.568239 7646 raft_consensus.cc:482] T 679fc86ea2ad4bd6a9160fddf4c586d7 P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:33.568300 7646 raft_consensus.cc:3037] T 679fc86ea2ad4bd6a9160fddf4c586d7 P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:33.569245 7646 raft_consensus.cc:504] T 679fc86ea2ad4bd6a9160fddf4c586d7 P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:33.569571 7646 leader_election.cc:288] T 679fc86ea2ad4bd6a9160fddf4c586d7 P 79163c69f6f04a9abd0208d74e1c61be [CANDIDATE]: Term 1 election: Requested vote from peers b6b1d5677b8d47788b7968f64b3dac3c (127.0.99.132:46067), 4127249ba42644388d225d614c935a25 (127.0.99.129:35613)
I20240503 01:59:33.569898 6757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "679fc86ea2ad4bd6a9160fddf4c586d7" candidate_uuid: "79163c69f6f04a9abd0208d74e1c61be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6b1d5677b8d47788b7968f64b3dac3c"
I20240503 01:59:33.569885 6364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "679fc86ea2ad4bd6a9160fddf4c586d7" candidate_uuid: "79163c69f6f04a9abd0208d74e1c61be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4127249ba42644388d225d614c935a25"
I20240503 01:59:33.569995 6364 raft_consensus.cc:3037] T 679fc86ea2ad4bd6a9160fddf4c586d7 P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:33.570003 6757 raft_consensus.cc:3037] T 679fc86ea2ad4bd6a9160fddf4c586d7 P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:33.570720 6757 raft_consensus.cc:2454] T 679fc86ea2ad4bd6a9160fddf4c586d7 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79163c69f6f04a9abd0208d74e1c61be in term 1.
I20240503 01:59:33.570922 6364 raft_consensus.cc:2454] T 679fc86ea2ad4bd6a9160fddf4c586d7 P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79163c69f6f04a9abd0208d74e1c61be in term 1.
I20240503 01:59:33.570988 6562 leader_election.cc:302] T 679fc86ea2ad4bd6a9160fddf4c586d7 P 79163c69f6f04a9abd0208d74e1c61be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79163c69f6f04a9abd0208d74e1c61be, b6b1d5677b8d47788b7968f64b3dac3c; no voters:
I20240503 01:59:33.571141 7646 raft_consensus.cc:2781] T 679fc86ea2ad4bd6a9160fddf4c586d7 P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:33.571230 7646 raft_consensus.cc:686] T 679fc86ea2ad4bd6a9160fddf4c586d7 P 79163c69f6f04a9abd0208d74e1c61be [term 1 LEADER]: Becoming Leader. State: Replica: 79163c69f6f04a9abd0208d74e1c61be, State: Running, Role: LEADER
I20240503 01:59:33.571389 7646 consensus_queue.cc:237] T 679fc86ea2ad4bd6a9160fddf4c586d7 P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:33.573029 6220 catalog_manager.cc:5484] T 679fc86ea2ad4bd6a9160fddf4c586d7 P 79163c69f6f04a9abd0208d74e1c61be reported cstate change: term changed from 0 to 1, leader changed from <none> to 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131). New cstate: current_term: 1 leader_uuid: "79163c69f6f04a9abd0208d74e1c61be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:33.626739 7646 raft_consensus.cc:482] T 5b599bd1fea6409fbe485c7a35c0891c P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:33.626874 7646 raft_consensus.cc:504] T 5b599bd1fea6409fbe485c7a35c0891c P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:33.627239 7646 leader_election.cc:288] T 5b599bd1fea6409fbe485c7a35c0891c P 79163c69f6f04a9abd0208d74e1c61be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130:34189), 4127249ba42644388d225d614c935a25 (127.0.99.129:35613)
I20240503 01:59:33.627691 6495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b599bd1fea6409fbe485c7a35c0891c" candidate_uuid: "79163c69f6f04a9abd0208d74e1c61be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c4821d9898d42799c5789d4d2da3f63" is_pre_election: true
I20240503 01:59:33.627689 6364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b599bd1fea6409fbe485c7a35c0891c" candidate_uuid: "79163c69f6f04a9abd0208d74e1c61be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4127249ba42644388d225d614c935a25" is_pre_election: true
I20240503 01:59:33.627820 6364 raft_consensus.cc:2454] T 5b599bd1fea6409fbe485c7a35c0891c P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79163c69f6f04a9abd0208d74e1c61be in term 0.
I20240503 01:59:33.627833 6495 raft_consensus.cc:2454] T 5b599bd1fea6409fbe485c7a35c0891c P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79163c69f6f04a9abd0208d74e1c61be in term 0.
I20240503 01:59:33.628122 6561 leader_election.cc:302] T 5b599bd1fea6409fbe485c7a35c0891c P 79163c69f6f04a9abd0208d74e1c61be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4127249ba42644388d225d614c935a25, 79163c69f6f04a9abd0208d74e1c61be; no voters:
I20240503 01:59:33.628299 7646 raft_consensus.cc:2781] T 5b599bd1fea6409fbe485c7a35c0891c P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:33.628355 7646 raft_consensus.cc:482] T 5b599bd1fea6409fbe485c7a35c0891c P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:33.628401 7646 raft_consensus.cc:3037] T 5b599bd1fea6409fbe485c7a35c0891c P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:33.629267 7646 raft_consensus.cc:504] T 5b599bd1fea6409fbe485c7a35c0891c P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:33.629614 7646 leader_election.cc:288] T 5b599bd1fea6409fbe485c7a35c0891c P 79163c69f6f04a9abd0208d74e1c61be [CANDIDATE]: Term 1 election: Requested vote from peers 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130:34189), 4127249ba42644388d225d614c935a25 (127.0.99.129:35613)
I20240503 01:59:33.630010 6364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b599bd1fea6409fbe485c7a35c0891c" candidate_uuid: "79163c69f6f04a9abd0208d74e1c61be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4127249ba42644388d225d614c935a25"
I20240503 01:59:33.629999 6495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b599bd1fea6409fbe485c7a35c0891c" candidate_uuid: "79163c69f6f04a9abd0208d74e1c61be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c4821d9898d42799c5789d4d2da3f63"
I20240503 01:59:33.630123 6364 raft_consensus.cc:3037] T 5b599bd1fea6409fbe485c7a35c0891c P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:33.630124 6495 raft_consensus.cc:3037] T 5b599bd1fea6409fbe485c7a35c0891c P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:33.631049 6364 raft_consensus.cc:2454] T 5b599bd1fea6409fbe485c7a35c0891c P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79163c69f6f04a9abd0208d74e1c61be in term 1.
I20240503 01:59:33.631050 6495 raft_consensus.cc:2454] T 5b599bd1fea6409fbe485c7a35c0891c P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79163c69f6f04a9abd0208d74e1c61be in term 1.
I20240503 01:59:33.631341 6563 leader_election.cc:302] T 5b599bd1fea6409fbe485c7a35c0891c P 79163c69f6f04a9abd0208d74e1c61be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c4821d9898d42799c5789d4d2da3f63, 79163c69f6f04a9abd0208d74e1c61be; no voters:
I20240503 01:59:33.631538 7646 raft_consensus.cc:2781] T 5b599bd1fea6409fbe485c7a35c0891c P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:33.631623 7646 raft_consensus.cc:686] T 5b599bd1fea6409fbe485c7a35c0891c P 79163c69f6f04a9abd0208d74e1c61be [term 1 LEADER]: Becoming Leader. State: Replica: 79163c69f6f04a9abd0208d74e1c61be, State: Running, Role: LEADER
I20240503 01:59:33.631775 7646 consensus_queue.cc:237] T 5b599bd1fea6409fbe485c7a35c0891c P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:33.633440 6220 catalog_manager.cc:5484] T 5b599bd1fea6409fbe485c7a35c0891c P 79163c69f6f04a9abd0208d74e1c61be reported cstate change: term changed from 0 to 1, leader changed from <none> to 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131). New cstate: current_term: 1 leader_uuid: "79163c69f6f04a9abd0208d74e1c61be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } health_report { overall_health: UNKNOWN } } }
W20240503 01:59:33.781867 7687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:33.782200 7687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:33.785427 7687 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240503 01:59:33.881122 6757 raft_consensus.cc:1261] T 679fc86ea2ad4bd6a9160fddf4c586d7 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:33.881683 7646 consensus_queue.cc:1035] T 679fc86ea2ad4bd6a9160fddf4c586d7 P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:33.881911 6364 raft_consensus.cc:1261] T 679fc86ea2ad4bd6a9160fddf4c586d7 P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:33.882046 6365 raft_consensus.cc:1261] T 2ac8004211e64c3481e75377092fa93b P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Refusing update from remote peer 0c4821d9898d42799c5789d4d2da3f63: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:33.882431 7647 consensus_queue.cc:1035] T 679fc86ea2ad4bd6a9160fddf4c586d7 P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:33.882478 7361 consensus_queue.cc:1035] T 2ac8004211e64c3481e75377092fa93b P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:33.885054 6495 raft_consensus.cc:1261] T 5b599bd1fea6409fbe485c7a35c0891c P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:33.885916 7647 consensus_queue.cc:1035] T 5b599bd1fea6409fbe485c7a35c0891c P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:33.887599 6363 raft_consensus.cc:1261] T 5b599bd1fea6409fbe485c7a35c0891c P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:33.887760 6758 raft_consensus.cc:1261] T 2ac8004211e64c3481e75377092fa93b P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Refusing update from remote peer 0c4821d9898d42799c5789d4d2da3f63: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:33.888166 7683 consensus_queue.cc:1035] T 2ac8004211e64c3481e75377092fa93b P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:33.893981 7647 consensus_queue.cc:1035] T 5b599bd1fea6409fbe485c7a35c0891c P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 113.425 ms
time per row: 0.0567123 ms
I20240503 01:59:33.949381 6216 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44084:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:59:33.956091 6738 tablet_service.cc:1467] Processing CreateTablet for tablet dc7d272c4d324e10959c62362e0d81f2 (DEFAULT_TABLE table=rebalance_test_table_5 [id=ba15a57b1f0e43fd9bae24116dc815d0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:33.956422 6738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc7d272c4d324e10959c62362e0d81f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:33.957051 6343 tablet_service.cc:1467] Processing CreateTablet for tablet 2088c84e4eef4ccfbc70d7b668b8f2cf (DEFAULT_TABLE table=rebalance_test_table_5 [id=ba15a57b1f0e43fd9bae24116dc815d0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:33.957304 6343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2088c84e4eef4ccfbc70d7b668b8f2cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:33.957263 6605 tablet_service.cc:1467] Processing CreateTablet for tablet dc7d272c4d324e10959c62362e0d81f2 (DEFAULT_TABLE table=rebalance_test_table_5 [id=ba15a57b1f0e43fd9bae24116dc815d0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:33.957532 6605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc7d272c4d324e10959c62362e0d81f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:33.958959 6607 tablet_service.cc:1467] Processing CreateTablet for tablet 2088c84e4eef4ccfbc70d7b668b8f2cf (DEFAULT_TABLE table=rebalance_test_table_5 [id=ba15a57b1f0e43fd9bae24116dc815d0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:33.959169 6607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2088c84e4eef4ccfbc70d7b668b8f2cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:33.959291 7670 tablet_bootstrap.cc:492] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 4127249ba42644388d225d614c935a25: Bootstrap starting.
I20240503 01:59:33.959905 6606 tablet_service.cc:1467] Processing CreateTablet for tablet 1d392e08c15a4d4098a350287b3163ef (DEFAULT_TABLE table=rebalance_test_table_5 [id=ba15a57b1f0e43fd9bae24116dc815d0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:33.960098 7670 tablet_bootstrap.cc:654] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 4127249ba42644388d225d614c935a25: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:33.960111 6606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d392e08c15a4d4098a350287b3163ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:33.961102 7727 tablet_bootstrap.cc:492] T dc7d272c4d324e10959c62362e0d81f2 P 79163c69f6f04a9abd0208d74e1c61be: Bootstrap starting.
I20240503 01:59:33.961364 7726 tablet_bootstrap.cc:492] T dc7d272c4d324e10959c62362e0d81f2 P b6b1d5677b8d47788b7968f64b3dac3c: Bootstrap starting.
I20240503 01:59:33.961949 7727 tablet_bootstrap.cc:654] T dc7d272c4d324e10959c62362e0d81f2 P 79163c69f6f04a9abd0208d74e1c61be: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:33.962261 7726 tablet_bootstrap.cc:654] T dc7d272c4d324e10959c62362e0d81f2 P b6b1d5677b8d47788b7968f64b3dac3c: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:33.963425 7726 tablet_bootstrap.cc:492] T dc7d272c4d324e10959c62362e0d81f2 P b6b1d5677b8d47788b7968f64b3dac3c: No bootstrap required, opened a new log
I20240503 01:59:33.963404 6475 tablet_service.cc:1467] Processing CreateTablet for tablet 2088c84e4eef4ccfbc70d7b668b8f2cf (DEFAULT_TABLE table=rebalance_test_table_5 [id=ba15a57b1f0e43fd9bae24116dc815d0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:33.963519 7726 ts_tablet_manager.cc:1367] T dc7d272c4d324e10959c62362e0d81f2 P b6b1d5677b8d47788b7968f64b3dac3c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:33.963644 6475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2088c84e4eef4ccfbc70d7b668b8f2cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:33.963594 6474 tablet_service.cc:1467] Processing CreateTablet for tablet 1d392e08c15a4d4098a350287b3163ef (DEFAULT_TABLE table=rebalance_test_table_5 [id=ba15a57b1f0e43fd9bae24116dc815d0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:33.963821 6474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d392e08c15a4d4098a350287b3163ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:33.963404 6476 tablet_service.cc:1467] Processing CreateTablet for tablet dc7d272c4d324e10959c62362e0d81f2 (DEFAULT_TABLE table=rebalance_test_table_5 [id=ba15a57b1f0e43fd9bae24116dc815d0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:33.963953 7726 raft_consensus.cc:348] T dc7d272c4d324e10959c62362e0d81f2 P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } }
I20240503 01:59:33.964069 7726 raft_consensus.cc:374] T dc7d272c4d324e10959c62362e0d81f2 P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:33.964107 7726 raft_consensus.cc:729] T dc7d272c4d324e10959c62362e0d81f2 P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6b1d5677b8d47788b7968f64b3dac3c, State: Initialized, Role: FOLLOWER
I20240503 01:59:33.964247 7726 consensus_queue.cc:260] T dc7d272c4d324e10959c62362e0d81f2 P b6b1d5677b8d47788b7968f64b3dac3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } }
I20240503 01:59:33.964504 7726 ts_tablet_manager.cc:1398] T dc7d272c4d324e10959c62362e0d81f2 P b6b1d5677b8d47788b7968f64b3dac3c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:33.965881 7731 tablet_bootstrap.cc:492] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 0c4821d9898d42799c5789d4d2da3f63: Bootstrap starting.
I20240503 01:59:33.966166 7727 tablet_bootstrap.cc:492] T dc7d272c4d324e10959c62362e0d81f2 P 79163c69f6f04a9abd0208d74e1c61be: No bootstrap required, opened a new log
I20240503 01:59:33.966233 7727 ts_tablet_manager.cc:1367] T dc7d272c4d324e10959c62362e0d81f2 P 79163c69f6f04a9abd0208d74e1c61be: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240503 01:59:33.966624 7727 raft_consensus.cc:348] T dc7d272c4d324e10959c62362e0d81f2 P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } }
I20240503 01:59:33.966723 7727 raft_consensus.cc:374] T dc7d272c4d324e10959c62362e0d81f2 P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:33.966747 7731 tablet_bootstrap.cc:654] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 0c4821d9898d42799c5789d4d2da3f63: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:33.966755 7727 raft_consensus.cc:729] T dc7d272c4d324e10959c62362e0d81f2 P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79163c69f6f04a9abd0208d74e1c61be, State: Initialized, Role: FOLLOWER
I20240503 01:59:33.966866 7727 consensus_queue.cc:260] T dc7d272c4d324e10959c62362e0d81f2 P 79163c69f6f04a9abd0208d74e1c61be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } }
I20240503 01:59:33.967110 7727 ts_tablet_manager.cc:1398] T dc7d272c4d324e10959c62362e0d81f2 P 79163c69f6f04a9abd0208d74e1c61be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:33.967391 6476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc7d272c4d324e10959c62362e0d81f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:33.968310 6738 tablet_service.cc:1467] Processing CreateTablet for tablet 1d392e08c15a4d4098a350287b3163ef (DEFAULT_TABLE table=rebalance_test_table_5 [id=ba15a57b1f0e43fd9bae24116dc815d0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:33.968572 6738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d392e08c15a4d4098a350287b3163ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:33.969660 7727 tablet_bootstrap.cc:492] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 79163c69f6f04a9abd0208d74e1c61be: Bootstrap starting.
I20240503 01:59:33.970453 7727 tablet_bootstrap.cc:654] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 79163c69f6f04a9abd0208d74e1c61be: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:33.970603 7726 tablet_bootstrap.cc:492] T 1d392e08c15a4d4098a350287b3163ef P b6b1d5677b8d47788b7968f64b3dac3c: Bootstrap starting.
I20240503 01:59:33.971468 7726 tablet_bootstrap.cc:654] T 1d392e08c15a4d4098a350287b3163ef P b6b1d5677b8d47788b7968f64b3dac3c: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:33.972482 7726 tablet_bootstrap.cc:492] T 1d392e08c15a4d4098a350287b3163ef P b6b1d5677b8d47788b7968f64b3dac3c: No bootstrap required, opened a new log
I20240503 01:59:33.972553 7726 ts_tablet_manager.cc:1367] T 1d392e08c15a4d4098a350287b3163ef P b6b1d5677b8d47788b7968f64b3dac3c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:33.972796 7727 tablet_bootstrap.cc:492] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 79163c69f6f04a9abd0208d74e1c61be: No bootstrap required, opened a new log
I20240503 01:59:33.972875 7727 ts_tablet_manager.cc:1367] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 79163c69f6f04a9abd0208d74e1c61be: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:33.972963 7726 raft_consensus.cc:348] T 1d392e08c15a4d4098a350287b3163ef P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } }
I20240503 01:59:33.973052 7726 raft_consensus.cc:374] T 1d392e08c15a4d4098a350287b3163ef P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:33.973079 7726 raft_consensus.cc:729] T 1d392e08c15a4d4098a350287b3163ef P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6b1d5677b8d47788b7968f64b3dac3c, State: Initialized, Role: FOLLOWER
I20240503 01:59:33.973182 7726 consensus_queue.cc:260] T 1d392e08c15a4d4098a350287b3163ef P b6b1d5677b8d47788b7968f64b3dac3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } }
I20240503 01:59:33.973320 7727 raft_consensus.cc:348] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } }
I20240503 01:59:33.973390 7670 tablet_bootstrap.cc:492] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 4127249ba42644388d225d614c935a25: No bootstrap required, opened a new log
I20240503 01:59:33.973407 7727 raft_consensus.cc:374] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:33.973421 7726 ts_tablet_manager.cc:1398] T 1d392e08c15a4d4098a350287b3163ef P b6b1d5677b8d47788b7968f64b3dac3c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:33.973448 7727 raft_consensus.cc:729] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79163c69f6f04a9abd0208d74e1c61be, State: Initialized, Role: FOLLOWER
I20240503 01:59:33.973474 7670 ts_tablet_manager.cc:1367] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 4127249ba42644388d225d614c935a25: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.000s
I20240503 01:59:33.973567 7727 consensus_queue.cc:260] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 79163c69f6f04a9abd0208d74e1c61be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } }
I20240503 01:59:33.973814 7727 ts_tablet_manager.cc:1398] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 79163c69f6f04a9abd0208d74e1c61be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:33.973960 7727 tablet_bootstrap.cc:492] T 1d392e08c15a4d4098a350287b3163ef P 79163c69f6f04a9abd0208d74e1c61be: Bootstrap starting.
I20240503 01:59:33.973959 7670 raft_consensus.cc:348] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } }
I20240503 01:59:33.974059 7670 raft_consensus.cc:374] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:33.974090 7670 raft_consensus.cc:729] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4127249ba42644388d225d614c935a25, State: Initialized, Role: FOLLOWER
I20240503 01:59:33.974197 7670 consensus_queue.cc:260] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 4127249ba42644388d225d614c935a25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } }
I20240503 01:59:33.974498 7670 ts_tablet_manager.cc:1398] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 4127249ba42644388d225d614c935a25: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:33.974815 7727 tablet_bootstrap.cc:654] T 1d392e08c15a4d4098a350287b3163ef P 79163c69f6f04a9abd0208d74e1c61be: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:33.975996 7727 tablet_bootstrap.cc:492] T 1d392e08c15a4d4098a350287b3163ef P 79163c69f6f04a9abd0208d74e1c61be: No bootstrap required, opened a new log
I20240503 01:59:33.976125 7727 ts_tablet_manager.cc:1367] T 1d392e08c15a4d4098a350287b3163ef P 79163c69f6f04a9abd0208d74e1c61be: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:33.976563 7727 raft_consensus.cc:348] T 1d392e08c15a4d4098a350287b3163ef P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } }
I20240503 01:59:33.976812 7727 raft_consensus.cc:374] T 1d392e08c15a4d4098a350287b3163ef P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:33.976871 7727 raft_consensus.cc:729] T 1d392e08c15a4d4098a350287b3163ef P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79163c69f6f04a9abd0208d74e1c61be, State: Initialized, Role: FOLLOWER
I20240503 01:59:33.976999 7727 consensus_queue.cc:260] T 1d392e08c15a4d4098a350287b3163ef P 79163c69f6f04a9abd0208d74e1c61be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } }
I20240503 01:59:33.977315 7727 ts_tablet_manager.cc:1398] T 1d392e08c15a4d4098a350287b3163ef P 79163c69f6f04a9abd0208d74e1c61be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:33.978626 7731 tablet_bootstrap.cc:492] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 0c4821d9898d42799c5789d4d2da3f63: No bootstrap required, opened a new log
I20240503 01:59:33.979229 7731 ts_tablet_manager.cc:1367] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 0c4821d9898d42799c5789d4d2da3f63: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240503 01:59:33.979580 7731 raft_consensus.cc:348] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } }
I20240503 01:59:33.979676 7731 raft_consensus.cc:374] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:33.979705 7731 raft_consensus.cc:729] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c4821d9898d42799c5789d4d2da3f63, State: Initialized, Role: FOLLOWER
I20240503 01:59:33.979827 7731 consensus_queue.cc:260] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 0c4821d9898d42799c5789d4d2da3f63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } }
I20240503 01:59:33.980762 7731 ts_tablet_manager.cc:1398] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 0c4821d9898d42799c5789d4d2da3f63: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:33.980945 7731 tablet_bootstrap.cc:492] T dc7d272c4d324e10959c62362e0d81f2 P 0c4821d9898d42799c5789d4d2da3f63: Bootstrap starting.
I20240503 01:59:33.982410 7731 tablet_bootstrap.cc:654] T dc7d272c4d324e10959c62362e0d81f2 P 0c4821d9898d42799c5789d4d2da3f63: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:33.983500 7731 tablet_bootstrap.cc:492] T dc7d272c4d324e10959c62362e0d81f2 P 0c4821d9898d42799c5789d4d2da3f63: No bootstrap required, opened a new log
I20240503 01:59:33.983611 7731 ts_tablet_manager.cc:1367] T dc7d272c4d324e10959c62362e0d81f2 P 0c4821d9898d42799c5789d4d2da3f63: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:33.983973 7731 raft_consensus.cc:348] T dc7d272c4d324e10959c62362e0d81f2 P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } }
I20240503 01:59:33.984083 7731 raft_consensus.cc:374] T dc7d272c4d324e10959c62362e0d81f2 P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:33.984127 7731 raft_consensus.cc:729] T dc7d272c4d324e10959c62362e0d81f2 P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c4821d9898d42799c5789d4d2da3f63, State: Initialized, Role: FOLLOWER
I20240503 01:59:33.984269 7731 consensus_queue.cc:260] T dc7d272c4d324e10959c62362e0d81f2 P 0c4821d9898d42799c5789d4d2da3f63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } }
I20240503 01:59:33.984514 7731 ts_tablet_manager.cc:1398] T dc7d272c4d324e10959c62362e0d81f2 P 0c4821d9898d42799c5789d4d2da3f63: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:33.984742 7731 tablet_bootstrap.cc:492] T 1d392e08c15a4d4098a350287b3163ef P 0c4821d9898d42799c5789d4d2da3f63: Bootstrap starting.
I20240503 01:59:33.985415 7731 tablet_bootstrap.cc:654] T 1d392e08c15a4d4098a350287b3163ef P 0c4821d9898d42799c5789d4d2da3f63: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:33.986526 7731 tablet_bootstrap.cc:492] T 1d392e08c15a4d4098a350287b3163ef P 0c4821d9898d42799c5789d4d2da3f63: No bootstrap required, opened a new log
I20240503 01:59:33.986593 7731 ts_tablet_manager.cc:1367] T 1d392e08c15a4d4098a350287b3163ef P 0c4821d9898d42799c5789d4d2da3f63: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:33.986935 7731 raft_consensus.cc:348] T 1d392e08c15a4d4098a350287b3163ef P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } }
I20240503 01:59:33.987025 7731 raft_consensus.cc:374] T 1d392e08c15a4d4098a350287b3163ef P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:33.987074 7731 raft_consensus.cc:729] T 1d392e08c15a4d4098a350287b3163ef P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c4821d9898d42799c5789d4d2da3f63, State: Initialized, Role: FOLLOWER
I20240503 01:59:33.987191 7731 consensus_queue.cc:260] T 1d392e08c15a4d4098a350287b3163ef P 0c4821d9898d42799c5789d4d2da3f63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } }
I20240503 01:59:33.987454 7731 ts_tablet_manager.cc:1398] T 1d392e08c15a4d4098a350287b3163ef P 0c4821d9898d42799c5789d4d2da3f63: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:34.037987 7361 raft_consensus.cc:482] T dc7d272c4d324e10959c62362e0d81f2 P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:34.038125 7361 raft_consensus.cc:504] T dc7d272c4d324e10959c62362e0d81f2 P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } }
I20240503 01:59:34.038465 7361 leader_election.cc:288] T dc7d272c4d324e10959c62362e0d81f2 P 0c4821d9898d42799c5789d4d2da3f63 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b6b1d5677b8d47788b7968f64b3dac3c (127.0.99.132:46067), 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131:44005)
I20240503 01:59:34.038849 6627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc7d272c4d324e10959c62362e0d81f2" candidate_uuid: "0c4821d9898d42799c5789d4d2da3f63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79163c69f6f04a9abd0208d74e1c61be" is_pre_election: true
I20240503 01:59:34.038900 6758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc7d272c4d324e10959c62362e0d81f2" candidate_uuid: "0c4821d9898d42799c5789d4d2da3f63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" is_pre_election: true
I20240503 01:59:34.038978 6627 raft_consensus.cc:2454] T dc7d272c4d324e10959c62362e0d81f2 P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c4821d9898d42799c5789d4d2da3f63 in term 0.
I20240503 01:59:34.039048 6758 raft_consensus.cc:2454] T dc7d272c4d324e10959c62362e0d81f2 P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c4821d9898d42799c5789d4d2da3f63 in term 0.
I20240503 01:59:34.039320 6430 leader_election.cc:302] T dc7d272c4d324e10959c62362e0d81f2 P 0c4821d9898d42799c5789d4d2da3f63 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c4821d9898d42799c5789d4d2da3f63, 79163c69f6f04a9abd0208d74e1c61be; no voters:
I20240503 01:59:34.039498 7361 raft_consensus.cc:2781] T dc7d272c4d324e10959c62362e0d81f2 P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:34.039553 7361 raft_consensus.cc:482] T dc7d272c4d324e10959c62362e0d81f2 P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:34.039602 7361 raft_consensus.cc:3037] T dc7d272c4d324e10959c62362e0d81f2 P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:34.040428 7361 raft_consensus.cc:504] T dc7d272c4d324e10959c62362e0d81f2 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } }
I20240503 01:59:34.040763 7361 leader_election.cc:288] T dc7d272c4d324e10959c62362e0d81f2 P 0c4821d9898d42799c5789d4d2da3f63 [CANDIDATE]: Term 1 election: Requested vote from peers b6b1d5677b8d47788b7968f64b3dac3c (127.0.99.132:46067), 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131:44005)
I20240503 01:59:34.041110 6627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc7d272c4d324e10959c62362e0d81f2" candidate_uuid: "0c4821d9898d42799c5789d4d2da3f63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79163c69f6f04a9abd0208d74e1c61be"
I20240503 01:59:34.041110 6758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc7d272c4d324e10959c62362e0d81f2" candidate_uuid: "0c4821d9898d42799c5789d4d2da3f63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6b1d5677b8d47788b7968f64b3dac3c"
I20240503 01:59:34.041231 6627 raft_consensus.cc:3037] T dc7d272c4d324e10959c62362e0d81f2 P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:34.041231 6758 raft_consensus.cc:3037] T dc7d272c4d324e10959c62362e0d81f2 P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:34.042181 6758 raft_consensus.cc:2454] T dc7d272c4d324e10959c62362e0d81f2 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c4821d9898d42799c5789d4d2da3f63 in term 1.
I20240503 01:59:34.042181 6627 raft_consensus.cc:2454] T dc7d272c4d324e10959c62362e0d81f2 P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c4821d9898d42799c5789d4d2da3f63 in term 1.
I20240503 01:59:34.042450 6430 leader_election.cc:302] T dc7d272c4d324e10959c62362e0d81f2 P 0c4821d9898d42799c5789d4d2da3f63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c4821d9898d42799c5789d4d2da3f63, 79163c69f6f04a9abd0208d74e1c61be; no voters:
I20240503 01:59:34.042645 7361 raft_consensus.cc:2781] T dc7d272c4d324e10959c62362e0d81f2 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:34.042739 7361 raft_consensus.cc:686] T dc7d272c4d324e10959c62362e0d81f2 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 LEADER]: Becoming Leader. State: Replica: 0c4821d9898d42799c5789d4d2da3f63, State: Running, Role: LEADER
I20240503 01:59:34.042889 7361 consensus_queue.cc:237] T dc7d272c4d324e10959c62362e0d81f2 P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } }
I20240503 01:59:34.044776 6216 catalog_manager.cc:5484] T dc7d272c4d324e10959c62362e0d81f2 P 0c4821d9898d42799c5789d4d2da3f63 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "0c4821d9898d42799c5789d4d2da3f63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:34.066586 7361 raft_consensus.cc:482] T 1d392e08c15a4d4098a350287b3163ef P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:34.066720 7361 raft_consensus.cc:504] T 1d392e08c15a4d4098a350287b3163ef P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } }
I20240503 01:59:34.067112 7361 leader_election.cc:288] T 1d392e08c15a4d4098a350287b3163ef P 0c4821d9898d42799c5789d4d2da3f63 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131:44005), b6b1d5677b8d47788b7968f64b3dac3c (127.0.99.132:46067)
I20240503 01:59:34.067474 6627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d392e08c15a4d4098a350287b3163ef" candidate_uuid: "0c4821d9898d42799c5789d4d2da3f63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79163c69f6f04a9abd0208d74e1c61be" is_pre_election: true
I20240503 01:59:34.067528 6758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d392e08c15a4d4098a350287b3163ef" candidate_uuid: "0c4821d9898d42799c5789d4d2da3f63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" is_pre_election: true
I20240503 01:59:34.067624 6627 raft_consensus.cc:2454] T 1d392e08c15a4d4098a350287b3163ef P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c4821d9898d42799c5789d4d2da3f63 in term 0.
I20240503 01:59:34.067670 6758 raft_consensus.cc:2454] T 1d392e08c15a4d4098a350287b3163ef P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c4821d9898d42799c5789d4d2da3f63 in term 0.
I20240503 01:59:34.067924 6431 leader_election.cc:302] T 1d392e08c15a4d4098a350287b3163ef P 0c4821d9898d42799c5789d4d2da3f63 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c4821d9898d42799c5789d4d2da3f63, b6b1d5677b8d47788b7968f64b3dac3c; no voters:
I20240503 01:59:34.068089 7361 raft_consensus.cc:2781] T 1d392e08c15a4d4098a350287b3163ef P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:34.068145 7361 raft_consensus.cc:482] T 1d392e08c15a4d4098a350287b3163ef P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:34.068185 7361 raft_consensus.cc:3037] T 1d392e08c15a4d4098a350287b3163ef P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:34.089442 7361 raft_consensus.cc:504] T 1d392e08c15a4d4098a350287b3163ef P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } }
I20240503 01:59:34.089841 7361 leader_election.cc:288] T 1d392e08c15a4d4098a350287b3163ef P 0c4821d9898d42799c5789d4d2da3f63 [CANDIDATE]: Term 1 election: Requested vote from peers 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131:44005), b6b1d5677b8d47788b7968f64b3dac3c (127.0.99.132:46067)
I20240503 01:59:34.090236 6627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d392e08c15a4d4098a350287b3163ef" candidate_uuid: "0c4821d9898d42799c5789d4d2da3f63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79163c69f6f04a9abd0208d74e1c61be"
I20240503 01:59:34.090237 6758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d392e08c15a4d4098a350287b3163ef" candidate_uuid: "0c4821d9898d42799c5789d4d2da3f63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6b1d5677b8d47788b7968f64b3dac3c"
I20240503 01:59:34.090356 6758 raft_consensus.cc:3037] T 1d392e08c15a4d4098a350287b3163ef P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:34.090355 6627 raft_consensus.cc:3037] T 1d392e08c15a4d4098a350287b3163ef P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:34.091074 6627 raft_consensus.cc:2454] T 1d392e08c15a4d4098a350287b3163ef P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c4821d9898d42799c5789d4d2da3f63 in term 1.
I20240503 01:59:34.091142 6758 raft_consensus.cc:2454] T 1d392e08c15a4d4098a350287b3163ef P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c4821d9898d42799c5789d4d2da3f63 in term 1.
I20240503 01:59:34.091361 6430 leader_election.cc:302] T 1d392e08c15a4d4098a350287b3163ef P 0c4821d9898d42799c5789d4d2da3f63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c4821d9898d42799c5789d4d2da3f63, 79163c69f6f04a9abd0208d74e1c61be; no voters:
I20240503 01:59:34.091536 7361 raft_consensus.cc:2781] T 1d392e08c15a4d4098a350287b3163ef P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:34.091647 7361 raft_consensus.cc:686] T 1d392e08c15a4d4098a350287b3163ef P 0c4821d9898d42799c5789d4d2da3f63 [term 1 LEADER]: Becoming Leader. State: Replica: 0c4821d9898d42799c5789d4d2da3f63, State: Running, Role: LEADER
I20240503 01:59:34.091866 7361 consensus_queue.cc:237] T 1d392e08c15a4d4098a350287b3163ef P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } }
I20240503 01:59:34.093664 6216 catalog_manager.cc:5484] T 1d392e08c15a4d4098a350287b3163ef P 0c4821d9898d42799c5789d4d2da3f63 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "0c4821d9898d42799c5789d4d2da3f63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } health_report { overall_health: HEALTHY } } }
I20240503 01:59:34.142756 7723 raft_consensus.cc:482] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:34.142889 7723 raft_consensus.cc:504] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } }
I20240503 01:59:34.143375 7723 leader_election.cc:288] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 4127249ba42644388d225d614c935a25 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130:34189), 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131:44005)
I20240503 01:59:34.143780 6495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2088c84e4eef4ccfbc70d7b668b8f2cf" candidate_uuid: "4127249ba42644388d225d614c935a25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c4821d9898d42799c5789d4d2da3f63" is_pre_election: true
I20240503 01:59:34.143925 6495 raft_consensus.cc:2454] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4127249ba42644388d225d614c935a25 in term 0.
I20240503 01:59:34.144320 6301 leader_election.cc:302] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 4127249ba42644388d225d614c935a25 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c4821d9898d42799c5789d4d2da3f63, 4127249ba42644388d225d614c935a25; no voters:
I20240503 01:59:34.144506 7723 raft_consensus.cc:2781] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:34.144551 7723 raft_consensus.cc:482] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:34.144580 7723 raft_consensus.cc:3037] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:34.145594 7723 raft_consensus.cc:504] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } }
I20240503 01:59:34.145923 7723 leader_election.cc:288] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 4127249ba42644388d225d614c935a25 [CANDIDATE]: Term 1 election: Requested vote from peers 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130:34189), 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131:44005)
I20240503 01:59:34.146214 6495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2088c84e4eef4ccfbc70d7b668b8f2cf" candidate_uuid: "4127249ba42644388d225d614c935a25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c4821d9898d42799c5789d4d2da3f63"
I20240503 01:59:34.146318 6495 raft_consensus.cc:3037] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:34.147186 6495 raft_consensus.cc:2454] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4127249ba42644388d225d614c935a25 in term 1.
I20240503 01:59:34.147325 6627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2088c84e4eef4ccfbc70d7b668b8f2cf" candidate_uuid: "4127249ba42644388d225d614c935a25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79163c69f6f04a9abd0208d74e1c61be" is_pre_election: true
I20240503 01:59:34.147387 6626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2088c84e4eef4ccfbc70d7b668b8f2cf" candidate_uuid: "4127249ba42644388d225d614c935a25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79163c69f6f04a9abd0208d74e1c61be"
I20240503 01:59:34.147449 6627 raft_consensus.cc:2454] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4127249ba42644388d225d614c935a25 in term 0.
I20240503 01:59:34.147514 6301 leader_election.cc:302] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 4127249ba42644388d225d614c935a25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c4821d9898d42799c5789d4d2da3f63, 4127249ba42644388d225d614c935a25; no voters:
I20240503 01:59:34.147743 7723 raft_consensus.cc:2781] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:34.147823 7723 raft_consensus.cc:686] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 4127249ba42644388d225d614c935a25 [term 1 LEADER]: Becoming Leader. State: Replica: 4127249ba42644388d225d614c935a25, State: Running, Role: LEADER
I20240503 01:59:34.147951 7723 consensus_queue.cc:237] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 4127249ba42644388d225d614c935a25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } }
I20240503 01:59:34.149683 6216 catalog_manager.cc:5484] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 4127249ba42644388d225d614c935a25 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4127249ba42644388d225d614c935a25 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "4127249ba42644388d225d614c935a25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } health_report { overall_health: UNKNOWN } } }
W20240503 01:59:34.261231 7742 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:34.261423 7742 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:34.264120 7742 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_5'
I20240503 01:59:34.351233 6627 raft_consensus.cc:1261] T dc7d272c4d324e10959c62362e0d81f2 P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Refusing update from remote peer 0c4821d9898d42799c5789d4d2da3f63: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:34.351230 6758 raft_consensus.cc:1261] T dc7d272c4d324e10959c62362e0d81f2 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Refusing update from remote peer 0c4821d9898d42799c5789d4d2da3f63: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:34.351637 7683 consensus_queue.cc:1035] T dc7d272c4d324e10959c62362e0d81f2 P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:34.351779 7361 consensus_queue.cc:1035] T dc7d272c4d324e10959c62362e0d81f2 P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:34.352540 6757 raft_consensus.cc:1261] T 1d392e08c15a4d4098a350287b3163ef P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Refusing update from remote peer 0c4821d9898d42799c5789d4d2da3f63: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:34.355974 7361 consensus_queue.cc:1035] T 1d392e08c15a4d4098a350287b3163ef P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:34.356983 6627 raft_consensus.cc:1261] T 1d392e08c15a4d4098a350287b3163ef P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Refusing update from remote peer 0c4821d9898d42799c5789d4d2da3f63: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:34.357643 7361 consensus_queue.cc:1035] T 1d392e08c15a4d4098a350287b3163ef P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:34.367091 6627 raft_consensus.cc:3037] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:34.367630 6495 raft_consensus.cc:1261] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Refusing update from remote peer 4127249ba42644388d225d614c935a25: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:34.368098 7499 consensus_queue.cc:1035] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 4127249ba42644388d225d614c935a25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:34.380509 7721 mvcc.cc:205] Tried to move back new op lower bound from 7023417648538132480 to 7023417647280685056. Current Snapshot: MvccSnapshot[applied={T|T < 7023417648538132480}]
I20240503 01:59:34.388042 6627 raft_consensus.cc:1261] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Refusing update from remote peer 4127249ba42644388d225d614c935a25: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:34.389966 7499 consensus_queue.cc:1035] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 4127249ba42644388d225d614c935a25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:34.397821 7650 mvcc.cc:205] Tried to move back new op lower bound from 7023417648587235328 to 7023417647481372672. Current Snapshot: MvccSnapshot[applied={T|T < 7023417648587235328}]
INSERT report
rows total: 2000
time total: 120.933 ms
time per row: 0.0604664 ms
I20240503 01:59:34.428231 6222 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44084:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:59:34.433445 6343 tablet_service.cc:1467] Processing CreateTablet for tablet 82cfe99cd6fe4a82a7e35298ec634b8b (DEFAULT_TABLE table=rebalance_test_table_6 [id=80cb1486f22e4bee930434fdfc3463d2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:34.433815 6343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82cfe99cd6fe4a82a7e35298ec634b8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:34.433856 6606 tablet_service.cc:1467] Processing CreateTablet for tablet 0a369bc048e74fc8ba666d3318d66ae8 (DEFAULT_TABLE table=rebalance_test_table_6 [id=80cb1486f22e4bee930434fdfc3463d2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:34.433820 6344 tablet_service.cc:1467] Processing CreateTablet for tablet 0a369bc048e74fc8ba666d3318d66ae8 (DEFAULT_TABLE table=rebalance_test_table_6 [id=80cb1486f22e4bee930434fdfc3463d2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:34.434090 6606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a369bc048e74fc8ba666d3318d66ae8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:34.434103 6344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a369bc048e74fc8ba666d3318d66ae8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:34.434486 6738 tablet_service.cc:1467] Processing CreateTablet for tablet 82cfe99cd6fe4a82a7e35298ec634b8b (DEFAULT_TABLE table=rebalance_test_table_6 [id=80cb1486f22e4bee930434fdfc3463d2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:34.434732 6738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82cfe99cd6fe4a82a7e35298ec634b8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:34.436334 7670 tablet_bootstrap.cc:492] T 82cfe99cd6fe4a82a7e35298ec634b8b P 4127249ba42644388d225d614c935a25: Bootstrap starting.
I20240503 01:59:34.436465 6474 tablet_service.cc:1467] Processing CreateTablet for tablet 82cfe99cd6fe4a82a7e35298ec634b8b (DEFAULT_TABLE table=rebalance_test_table_6 [id=80cb1486f22e4bee930434fdfc3463d2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:34.436766 6474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82cfe99cd6fe4a82a7e35298ec634b8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:34.436782 7726 tablet_bootstrap.cc:492] T 82cfe99cd6fe4a82a7e35298ec634b8b P b6b1d5677b8d47788b7968f64b3dac3c: Bootstrap starting.
I20240503 01:59:34.436868 7727 tablet_bootstrap.cc:492] T 0a369bc048e74fc8ba666d3318d66ae8 P 79163c69f6f04a9abd0208d74e1c61be: Bootstrap starting.
I20240503 01:59:34.437127 7670 tablet_bootstrap.cc:654] T 82cfe99cd6fe4a82a7e35298ec634b8b P 4127249ba42644388d225d614c935a25: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:34.437590 7727 tablet_bootstrap.cc:654] T 0a369bc048e74fc8ba666d3318d66ae8 P 79163c69f6f04a9abd0208d74e1c61be: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:34.437644 7726 tablet_bootstrap.cc:654] T 82cfe99cd6fe4a82a7e35298ec634b8b P b6b1d5677b8d47788b7968f64b3dac3c: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:34.438545 6344 tablet_service.cc:1467] Processing CreateTablet for tablet 120d5dd4f26c42399702141094ce7982 (DEFAULT_TABLE table=rebalance_test_table_6 [id=80cb1486f22e4bee930434fdfc3463d2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:34.438984 7731 tablet_bootstrap.cc:492] T 82cfe99cd6fe4a82a7e35298ec634b8b P 0c4821d9898d42799c5789d4d2da3f63: Bootstrap starting.
I20240503 01:59:34.438951 6476 tablet_service.cc:1467] Processing CreateTablet for tablet 120d5dd4f26c42399702141094ce7982 (DEFAULT_TABLE table=rebalance_test_table_6 [id=80cb1486f22e4bee930434fdfc3463d2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:34.439189 6476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 120d5dd4f26c42399702141094ce7982. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:34.439149 6737 tablet_service.cc:1467] Processing CreateTablet for tablet 120d5dd4f26c42399702141094ce7982 (DEFAULT_TABLE table=rebalance_test_table_6 [id=80cb1486f22e4bee930434fdfc3463d2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:34.439356 6737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 120d5dd4f26c42399702141094ce7982. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:34.439606 7670 tablet_bootstrap.cc:492] T 82cfe99cd6fe4a82a7e35298ec634b8b P 4127249ba42644388d225d614c935a25: No bootstrap required, opened a new log
I20240503 01:59:34.439677 7670 ts_tablet_manager.cc:1367] T 82cfe99cd6fe4a82a7e35298ec634b8b P 4127249ba42644388d225d614c935a25: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 01:59:34.439872 7731 tablet_bootstrap.cc:654] T 82cfe99cd6fe4a82a7e35298ec634b8b P 0c4821d9898d42799c5789d4d2da3f63: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:34.439958 7727 tablet_bootstrap.cc:492] T 0a369bc048e74fc8ba666d3318d66ae8 P 79163c69f6f04a9abd0208d74e1c61be: No bootstrap required, opened a new log
I20240503 01:59:34.440026 7727 ts_tablet_manager.cc:1367] T 0a369bc048e74fc8ba666d3318d66ae8 P 79163c69f6f04a9abd0208d74e1c61be: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 01:59:34.440124 7670 raft_consensus.cc:348] T 82cfe99cd6fe4a82a7e35298ec634b8b P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } }
I20240503 01:59:34.440227 7670 raft_consensus.cc:374] T 82cfe99cd6fe4a82a7e35298ec634b8b P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:34.440264 7670 raft_consensus.cc:729] T 82cfe99cd6fe4a82a7e35298ec634b8b P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4127249ba42644388d225d614c935a25, State: Initialized, Role: FOLLOWER
I20240503 01:59:34.440398 7670 consensus_queue.cc:260] T 82cfe99cd6fe4a82a7e35298ec634b8b P 4127249ba42644388d225d614c935a25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } }
I20240503 01:59:34.440428 7727 raft_consensus.cc:348] T 0a369bc048e74fc8ba666d3318d66ae8 P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } }
I20240503 01:59:34.440558 7727 raft_consensus.cc:374] T 0a369bc048e74fc8ba666d3318d66ae8 P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:34.440586 7727 raft_consensus.cc:729] T 0a369bc048e74fc8ba666d3318d66ae8 P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79163c69f6f04a9abd0208d74e1c61be, State: Initialized, Role: FOLLOWER
I20240503 01:59:34.440712 7670 ts_tablet_manager.cc:1398] T 82cfe99cd6fe4a82a7e35298ec634b8b P 4127249ba42644388d225d614c935a25: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:34.440712 7727 consensus_queue.cc:260] T 0a369bc048e74fc8ba666d3318d66ae8 P 79163c69f6f04a9abd0208d74e1c61be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } }
I20240503 01:59:34.440874 7670 tablet_bootstrap.cc:492] T 0a369bc048e74fc8ba666d3318d66ae8 P 4127249ba42644388d225d614c935a25: Bootstrap starting.
I20240503 01:59:34.440975 7727 ts_tablet_manager.cc:1398] T 0a369bc048e74fc8ba666d3318d66ae8 P 79163c69f6f04a9abd0208d74e1c61be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:34.441639 7670 tablet_bootstrap.cc:654] T 0a369bc048e74fc8ba666d3318d66ae8 P 4127249ba42644388d225d614c935a25: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:34.441680 6738 tablet_service.cc:1467] Processing CreateTablet for tablet 0a369bc048e74fc8ba666d3318d66ae8 (DEFAULT_TABLE table=rebalance_test_table_6 [id=80cb1486f22e4bee930434fdfc3463d2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:34.441877 6738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a369bc048e74fc8ba666d3318d66ae8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:34.441946 6344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 120d5dd4f26c42399702141094ce7982. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:34.442631 7670 tablet_bootstrap.cc:492] T 0a369bc048e74fc8ba666d3318d66ae8 P 4127249ba42644388d225d614c935a25: No bootstrap required, opened a new log
I20240503 01:59:34.442726 7670 ts_tablet_manager.cc:1367] T 0a369bc048e74fc8ba666d3318d66ae8 P 4127249ba42644388d225d614c935a25: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:34.443717 7731 tablet_bootstrap.cc:492] T 82cfe99cd6fe4a82a7e35298ec634b8b P 0c4821d9898d42799c5789d4d2da3f63: No bootstrap required, opened a new log
I20240503 01:59:34.443805 7731 ts_tablet_manager.cc:1367] T 82cfe99cd6fe4a82a7e35298ec634b8b P 0c4821d9898d42799c5789d4d2da3f63: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 01:59:34.443226 7670 raft_consensus.cc:348] T 0a369bc048e74fc8ba666d3318d66ae8 P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } }
I20240503 01:59:34.444180 7670 raft_consensus.cc:374] T 0a369bc048e74fc8ba666d3318d66ae8 P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:34.444269 7670 raft_consensus.cc:729] T 0a369bc048e74fc8ba666d3318d66ae8 P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4127249ba42644388d225d614c935a25, State: Initialized, Role: FOLLOWER
I20240503 01:59:34.444239 7731 raft_consensus.cc:348] T 82cfe99cd6fe4a82a7e35298ec634b8b P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } }
I20240503 01:59:34.444334 7731 raft_consensus.cc:374] T 82cfe99cd6fe4a82a7e35298ec634b8b P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:34.444387 7731 raft_consensus.cc:729] T 82cfe99cd6fe4a82a7e35298ec634b8b P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c4821d9898d42799c5789d4d2da3f63, State: Initialized, Role: FOLLOWER
I20240503 01:59:34.444510 7670 consensus_queue.cc:260] T 0a369bc048e74fc8ba666d3318d66ae8 P 4127249ba42644388d225d614c935a25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } }
I20240503 01:59:34.444510 7731 consensus_queue.cc:260] T 82cfe99cd6fe4a82a7e35298ec634b8b P 0c4821d9898d42799c5789d4d2da3f63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } }
I20240503 01:59:34.444806 7731 ts_tablet_manager.cc:1398] T 82cfe99cd6fe4a82a7e35298ec634b8b P 0c4821d9898d42799c5789d4d2da3f63: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:34.444869 7670 ts_tablet_manager.cc:1398] T 0a369bc048e74fc8ba666d3318d66ae8 P 4127249ba42644388d225d614c935a25: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:34.444967 7731 tablet_bootstrap.cc:492] T 120d5dd4f26c42399702141094ce7982 P 0c4821d9898d42799c5789d4d2da3f63: Bootstrap starting.
I20240503 01:59:34.445685 7731 tablet_bootstrap.cc:654] T 120d5dd4f26c42399702141094ce7982 P 0c4821d9898d42799c5789d4d2da3f63: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:34.447594 7731 tablet_bootstrap.cc:492] T 120d5dd4f26c42399702141094ce7982 P 0c4821d9898d42799c5789d4d2da3f63: No bootstrap required, opened a new log
I20240503 01:59:34.447714 7731 ts_tablet_manager.cc:1367] T 120d5dd4f26c42399702141094ce7982 P 0c4821d9898d42799c5789d4d2da3f63: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:34.448282 7731 raft_consensus.cc:348] T 120d5dd4f26c42399702141094ce7982 P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:34.448493 7731 raft_consensus.cc:374] T 120d5dd4f26c42399702141094ce7982 P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:34.448541 7731 raft_consensus.cc:729] T 120d5dd4f26c42399702141094ce7982 P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c4821d9898d42799c5789d4d2da3f63, State: Initialized, Role: FOLLOWER
I20240503 01:59:34.448712 7731 consensus_queue.cc:260] T 120d5dd4f26c42399702141094ce7982 P 0c4821d9898d42799c5789d4d2da3f63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:34.449978 7670 tablet_bootstrap.cc:492] T 120d5dd4f26c42399702141094ce7982 P 4127249ba42644388d225d614c935a25: Bootstrap starting.
I20240503 01:59:34.450819 7670 tablet_bootstrap.cc:654] T 120d5dd4f26c42399702141094ce7982 P 4127249ba42644388d225d614c935a25: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:34.451984 7731 ts_tablet_manager.cc:1398] T 120d5dd4f26c42399702141094ce7982 P 0c4821d9898d42799c5789d4d2da3f63: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 01:59:34.452009 7670 tablet_bootstrap.cc:492] T 120d5dd4f26c42399702141094ce7982 P 4127249ba42644388d225d614c935a25: No bootstrap required, opened a new log
I20240503 01:59:34.452087 7670 ts_tablet_manager.cc:1367] T 120d5dd4f26c42399702141094ce7982 P 4127249ba42644388d225d614c935a25: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:34.452606 7670 raft_consensus.cc:348] T 120d5dd4f26c42399702141094ce7982 P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:34.452733 7670 raft_consensus.cc:374] T 120d5dd4f26c42399702141094ce7982 P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:34.452769 7670 raft_consensus.cc:729] T 120d5dd4f26c42399702141094ce7982 P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4127249ba42644388d225d614c935a25, State: Initialized, Role: FOLLOWER
I20240503 01:59:34.452872 7670 consensus_queue.cc:260] T 120d5dd4f26c42399702141094ce7982 P 4127249ba42644388d225d614c935a25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:34.453874 7670 ts_tablet_manager.cc:1398] T 120d5dd4f26c42399702141094ce7982 P 4127249ba42644388d225d614c935a25: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:34.457008 7726 tablet_bootstrap.cc:492] T 82cfe99cd6fe4a82a7e35298ec634b8b P b6b1d5677b8d47788b7968f64b3dac3c: No bootstrap required, opened a new log
I20240503 01:59:34.457108 7726 ts_tablet_manager.cc:1367] T 82cfe99cd6fe4a82a7e35298ec634b8b P b6b1d5677b8d47788b7968f64b3dac3c: Time spent bootstrapping tablet: real 0.020s user 0.000s sys 0.002s
I20240503 01:59:34.457489 7726 raft_consensus.cc:348] T 82cfe99cd6fe4a82a7e35298ec634b8b P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } }
I20240503 01:59:34.457583 7726 raft_consensus.cc:374] T 82cfe99cd6fe4a82a7e35298ec634b8b P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:34.457618 7726 raft_consensus.cc:729] T 82cfe99cd6fe4a82a7e35298ec634b8b P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6b1d5677b8d47788b7968f64b3dac3c, State: Initialized, Role: FOLLOWER
I20240503 01:59:34.457756 7726 consensus_queue.cc:260] T 82cfe99cd6fe4a82a7e35298ec634b8b P b6b1d5677b8d47788b7968f64b3dac3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } }
I20240503 01:59:34.457965 7726 ts_tablet_manager.cc:1398] T 82cfe99cd6fe4a82a7e35298ec634b8b P b6b1d5677b8d47788b7968f64b3dac3c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:34.458505 7726 tablet_bootstrap.cc:492] T 120d5dd4f26c42399702141094ce7982 P b6b1d5677b8d47788b7968f64b3dac3c: Bootstrap starting.
I20240503 01:59:34.459355 7726 tablet_bootstrap.cc:654] T 120d5dd4f26c42399702141094ce7982 P b6b1d5677b8d47788b7968f64b3dac3c: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:34.460557 7726 tablet_bootstrap.cc:492] T 120d5dd4f26c42399702141094ce7982 P b6b1d5677b8d47788b7968f64b3dac3c: No bootstrap required, opened a new log
I20240503 01:59:34.460675 7726 ts_tablet_manager.cc:1367] T 120d5dd4f26c42399702141094ce7982 P b6b1d5677b8d47788b7968f64b3dac3c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240503 01:59:34.461144 7726 raft_consensus.cc:348] T 120d5dd4f26c42399702141094ce7982 P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:34.461234 7726 raft_consensus.cc:374] T 120d5dd4f26c42399702141094ce7982 P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:34.461269 7726 raft_consensus.cc:729] T 120d5dd4f26c42399702141094ce7982 P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6b1d5677b8d47788b7968f64b3dac3c, State: Initialized, Role: FOLLOWER
I20240503 01:59:34.461380 7726 consensus_queue.cc:260] T 120d5dd4f26c42399702141094ce7982 P b6b1d5677b8d47788b7968f64b3dac3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:34.461864 7726 ts_tablet_manager.cc:1398] T 120d5dd4f26c42399702141094ce7982 P b6b1d5677b8d47788b7968f64b3dac3c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:34.462030 7726 tablet_bootstrap.cc:492] T 0a369bc048e74fc8ba666d3318d66ae8 P b6b1d5677b8d47788b7968f64b3dac3c: Bootstrap starting.
I20240503 01:59:34.463057 7726 tablet_bootstrap.cc:654] T 0a369bc048e74fc8ba666d3318d66ae8 P b6b1d5677b8d47788b7968f64b3dac3c: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:34.464478 7726 tablet_bootstrap.cc:492] T 0a369bc048e74fc8ba666d3318d66ae8 P b6b1d5677b8d47788b7968f64b3dac3c: No bootstrap required, opened a new log
I20240503 01:59:34.464562 7726 ts_tablet_manager.cc:1367] T 0a369bc048e74fc8ba666d3318d66ae8 P b6b1d5677b8d47788b7968f64b3dac3c: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240503 01:59:34.464912 7726 raft_consensus.cc:348] T 0a369bc048e74fc8ba666d3318d66ae8 P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } }
I20240503 01:59:34.464995 7726 raft_consensus.cc:374] T 0a369bc048e74fc8ba666d3318d66ae8 P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:34.465025 7726 raft_consensus.cc:729] T 0a369bc048e74fc8ba666d3318d66ae8 P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6b1d5677b8d47788b7968f64b3dac3c, State: Initialized, Role: FOLLOWER
I20240503 01:59:34.465134 7726 consensus_queue.cc:260] T 0a369bc048e74fc8ba666d3318d66ae8 P b6b1d5677b8d47788b7968f64b3dac3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } }
I20240503 01:59:34.465405 7726 ts_tablet_manager.cc:1398] T 0a369bc048e74fc8ba666d3318d66ae8 P b6b1d5677b8d47788b7968f64b3dac3c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:34.479996 7499 raft_consensus.cc:482] T 0a369bc048e74fc8ba666d3318d66ae8 P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:34.480085 7499 raft_consensus.cc:504] T 0a369bc048e74fc8ba666d3318d66ae8 P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } }
I20240503 01:59:34.480429 7499 leader_election.cc:288] T 0a369bc048e74fc8ba666d3318d66ae8 P 4127249ba42644388d225d614c935a25 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131:44005), b6b1d5677b8d47788b7968f64b3dac3c (127.0.99.132:46067)
I20240503 01:59:34.480792 6758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a369bc048e74fc8ba666d3318d66ae8" candidate_uuid: "4127249ba42644388d225d614c935a25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" is_pre_election: true
I20240503 01:59:34.480823 6625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a369bc048e74fc8ba666d3318d66ae8" candidate_uuid: "4127249ba42644388d225d614c935a25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79163c69f6f04a9abd0208d74e1c61be" is_pre_election: true
I20240503 01:59:34.480896 6758 raft_consensus.cc:2454] T 0a369bc048e74fc8ba666d3318d66ae8 P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4127249ba42644388d225d614c935a25 in term 0.
I20240503 01:59:34.480955 6625 raft_consensus.cc:2454] T 0a369bc048e74fc8ba666d3318d66ae8 P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4127249ba42644388d225d614c935a25 in term 0.
I20240503 01:59:34.481139 6300 leader_election.cc:302] T 0a369bc048e74fc8ba666d3318d66ae8 P 4127249ba42644388d225d614c935a25 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4127249ba42644388d225d614c935a25, b6b1d5677b8d47788b7968f64b3dac3c; no voters:
I20240503 01:59:34.481314 7499 raft_consensus.cc:2781] T 0a369bc048e74fc8ba666d3318d66ae8 P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:34.481367 7499 raft_consensus.cc:482] T 0a369bc048e74fc8ba666d3318d66ae8 P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:34.481393 7499 raft_consensus.cc:3037] T 0a369bc048e74fc8ba666d3318d66ae8 P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:34.482411 7499 raft_consensus.cc:504] T 0a369bc048e74fc8ba666d3318d66ae8 P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } }
I20240503 01:59:34.482746 7499 leader_election.cc:288] T 0a369bc048e74fc8ba666d3318d66ae8 P 4127249ba42644388d225d614c935a25 [CANDIDATE]: Term 1 election: Requested vote from peers 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131:44005), b6b1d5677b8d47788b7968f64b3dac3c (127.0.99.132:46067)
I20240503 01:59:34.483053 6625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a369bc048e74fc8ba666d3318d66ae8" candidate_uuid: "4127249ba42644388d225d614c935a25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79163c69f6f04a9abd0208d74e1c61be"
I20240503 01:59:34.483170 6625 raft_consensus.cc:3037] T 0a369bc048e74fc8ba666d3318d66ae8 P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:34.483112 6758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a369bc048e74fc8ba666d3318d66ae8" candidate_uuid: "4127249ba42644388d225d614c935a25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6b1d5677b8d47788b7968f64b3dac3c"
I20240503 01:59:34.483310 6758 raft_consensus.cc:3037] T 0a369bc048e74fc8ba666d3318d66ae8 P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:34.483963 6625 raft_consensus.cc:2454] T 0a369bc048e74fc8ba666d3318d66ae8 P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4127249ba42644388d225d614c935a25 in term 1.
I20240503 01:59:34.484259 6758 raft_consensus.cc:2454] T 0a369bc048e74fc8ba666d3318d66ae8 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4127249ba42644388d225d614c935a25 in term 1.
I20240503 01:59:34.484270 6299 leader_election.cc:302] T 0a369bc048e74fc8ba666d3318d66ae8 P 4127249ba42644388d225d614c935a25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4127249ba42644388d225d614c935a25, 79163c69f6f04a9abd0208d74e1c61be; no voters:
I20240503 01:59:34.484467 7499 raft_consensus.cc:2781] T 0a369bc048e74fc8ba666d3318d66ae8 P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:34.484607 7499 raft_consensus.cc:686] T 0a369bc048e74fc8ba666d3318d66ae8 P 4127249ba42644388d225d614c935a25 [term 1 LEADER]: Becoming Leader. State: Replica: 4127249ba42644388d225d614c935a25, State: Running, Role: LEADER
I20240503 01:59:34.484766 7499 consensus_queue.cc:237] T 0a369bc048e74fc8ba666d3318d66ae8 P 4127249ba42644388d225d614c935a25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } }
I20240503 01:59:34.486187 6223 catalog_manager.cc:5484] T 0a369bc048e74fc8ba666d3318d66ae8 P 4127249ba42644388d225d614c935a25 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4127249ba42644388d225d614c935a25 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "4127249ba42644388d225d614c935a25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:34.505774 7361 raft_consensus.cc:482] T 120d5dd4f26c42399702141094ce7982 P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:34.505903 7361 raft_consensus.cc:504] T 120d5dd4f26c42399702141094ce7982 P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:34.506489 7361 leader_election.cc:288] T 120d5dd4f26c42399702141094ce7982 P 0c4821d9898d42799c5789d4d2da3f63 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b6b1d5677b8d47788b7968f64b3dac3c (127.0.99.132:46067), 4127249ba42644388d225d614c935a25 (127.0.99.129:35613)
I20240503 01:59:34.506799 6758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "120d5dd4f26c42399702141094ce7982" candidate_uuid: "0c4821d9898d42799c5789d4d2da3f63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" is_pre_election: true
I20240503 01:59:34.506932 6758 raft_consensus.cc:2454] T 120d5dd4f26c42399702141094ce7982 P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c4821d9898d42799c5789d4d2da3f63 in term 0.
I20240503 01:59:34.506930 6365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "120d5dd4f26c42399702141094ce7982" candidate_uuid: "0c4821d9898d42799c5789d4d2da3f63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4127249ba42644388d225d614c935a25" is_pre_election: true
I20240503 01:59:34.507069 6365 raft_consensus.cc:2454] T 120d5dd4f26c42399702141094ce7982 P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c4821d9898d42799c5789d4d2da3f63 in term 0.
I20240503 01:59:34.507266 6431 leader_election.cc:302] T 120d5dd4f26c42399702141094ce7982 P 0c4821d9898d42799c5789d4d2da3f63 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c4821d9898d42799c5789d4d2da3f63, b6b1d5677b8d47788b7968f64b3dac3c; no voters:
I20240503 01:59:34.507444 7361 raft_consensus.cc:2781] T 120d5dd4f26c42399702141094ce7982 P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:34.507503 7361 raft_consensus.cc:482] T 120d5dd4f26c42399702141094ce7982 P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:34.507555 7361 raft_consensus.cc:3037] T 120d5dd4f26c42399702141094ce7982 P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:34.508260 7361 raft_consensus.cc:504] T 120d5dd4f26c42399702141094ce7982 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:34.508534 7361 leader_election.cc:288] T 120d5dd4f26c42399702141094ce7982 P 0c4821d9898d42799c5789d4d2da3f63 [CANDIDATE]: Term 1 election: Requested vote from peers b6b1d5677b8d47788b7968f64b3dac3c (127.0.99.132:46067), 4127249ba42644388d225d614c935a25 (127.0.99.129:35613)
I20240503 01:59:34.508822 6758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "120d5dd4f26c42399702141094ce7982" candidate_uuid: "0c4821d9898d42799c5789d4d2da3f63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6b1d5677b8d47788b7968f64b3dac3c"
I20240503 01:59:34.508920 6758 raft_consensus.cc:3037] T 120d5dd4f26c42399702141094ce7982 P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:34.508883 6365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "120d5dd4f26c42399702141094ce7982" candidate_uuid: "0c4821d9898d42799c5789d4d2da3f63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4127249ba42644388d225d614c935a25"
I20240503 01:59:34.508986 6365 raft_consensus.cc:3037] T 120d5dd4f26c42399702141094ce7982 P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:34.509563 6758 raft_consensus.cc:2454] T 120d5dd4f26c42399702141094ce7982 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c4821d9898d42799c5789d4d2da3f63 in term 1.
I20240503 01:59:34.509657 6365 raft_consensus.cc:2454] T 120d5dd4f26c42399702141094ce7982 P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c4821d9898d42799c5789d4d2da3f63 in term 1.
I20240503 01:59:34.509837 6431 leader_election.cc:302] T 120d5dd4f26c42399702141094ce7982 P 0c4821d9898d42799c5789d4d2da3f63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c4821d9898d42799c5789d4d2da3f63, b6b1d5677b8d47788b7968f64b3dac3c; no voters:
I20240503 01:59:34.510007 7361 raft_consensus.cc:2781] T 120d5dd4f26c42399702141094ce7982 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:34.510107 7361 raft_consensus.cc:686] T 120d5dd4f26c42399702141094ce7982 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 LEADER]: Becoming Leader. State: Replica: 0c4821d9898d42799c5789d4d2da3f63, State: Running, Role: LEADER
I20240503 01:59:34.510257 7361 consensus_queue.cc:237] T 120d5dd4f26c42399702141094ce7982 P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:34.511835 6223 catalog_manager.cc:5484] T 120d5dd4f26c42399702141094ce7982 P 0c4821d9898d42799c5789d4d2da3f63 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "0c4821d9898d42799c5789d4d2da3f63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:34.519387 7499 raft_consensus.cc:482] T 82cfe99cd6fe4a82a7e35298ec634b8b P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:34.519466 7499 raft_consensus.cc:504] T 82cfe99cd6fe4a82a7e35298ec634b8b P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } }
I20240503 01:59:34.519804 7499 leader_election.cc:288] T 82cfe99cd6fe4a82a7e35298ec634b8b P 4127249ba42644388d225d614c935a25 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b6b1d5677b8d47788b7968f64b3dac3c (127.0.99.132:46067), 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130:34189)
I20240503 01:59:34.520109 6495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82cfe99cd6fe4a82a7e35298ec634b8b" candidate_uuid: "4127249ba42644388d225d614c935a25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c4821d9898d42799c5789d4d2da3f63" is_pre_election: true
I20240503 01:59:34.520124 6758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82cfe99cd6fe4a82a7e35298ec634b8b" candidate_uuid: "4127249ba42644388d225d614c935a25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" is_pre_election: true
I20240503 01:59:34.520227 6758 raft_consensus.cc:2454] T 82cfe99cd6fe4a82a7e35298ec634b8b P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4127249ba42644388d225d614c935a25 in term 0.
I20240503 01:59:34.520236 6495 raft_consensus.cc:2454] T 82cfe99cd6fe4a82a7e35298ec634b8b P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4127249ba42644388d225d614c935a25 in term 0.
I20240503 01:59:34.520478 6300 leader_election.cc:302] T 82cfe99cd6fe4a82a7e35298ec634b8b P 4127249ba42644388d225d614c935a25 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4127249ba42644388d225d614c935a25, b6b1d5677b8d47788b7968f64b3dac3c; no voters:
I20240503 01:59:34.520617 7499 raft_consensus.cc:2781] T 82cfe99cd6fe4a82a7e35298ec634b8b P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:34.520691 7499 raft_consensus.cc:482] T 82cfe99cd6fe4a82a7e35298ec634b8b P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:34.520740 7499 raft_consensus.cc:3037] T 82cfe99cd6fe4a82a7e35298ec634b8b P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:34.521385 7499 raft_consensus.cc:504] T 82cfe99cd6fe4a82a7e35298ec634b8b P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } }
I20240503 01:59:34.521674 7499 leader_election.cc:288] T 82cfe99cd6fe4a82a7e35298ec634b8b P 4127249ba42644388d225d614c935a25 [CANDIDATE]: Term 1 election: Requested vote from peers b6b1d5677b8d47788b7968f64b3dac3c (127.0.99.132:46067), 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130:34189)
I20240503 01:59:34.521975 6758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82cfe99cd6fe4a82a7e35298ec634b8b" candidate_uuid: "4127249ba42644388d225d614c935a25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6b1d5677b8d47788b7968f64b3dac3c"
I20240503 01:59:34.522014 6495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82cfe99cd6fe4a82a7e35298ec634b8b" candidate_uuid: "4127249ba42644388d225d614c935a25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c4821d9898d42799c5789d4d2da3f63"
I20240503 01:59:34.522082 6758 raft_consensus.cc:3037] T 82cfe99cd6fe4a82a7e35298ec634b8b P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:34.522102 6495 raft_consensus.cc:3037] T 82cfe99cd6fe4a82a7e35298ec634b8b P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:34.522954 6758 raft_consensus.cc:2454] T 82cfe99cd6fe4a82a7e35298ec634b8b P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4127249ba42644388d225d614c935a25 in term 1.
I20240503 01:59:34.523010 6495 raft_consensus.cc:2454] T 82cfe99cd6fe4a82a7e35298ec634b8b P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4127249ba42644388d225d614c935a25 in term 1.
I20240503 01:59:34.523265 6301 leader_election.cc:302] T 82cfe99cd6fe4a82a7e35298ec634b8b P 4127249ba42644388d225d614c935a25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c4821d9898d42799c5789d4d2da3f63, 4127249ba42644388d225d614c935a25; no voters:
I20240503 01:59:34.523474 7499 raft_consensus.cc:2781] T 82cfe99cd6fe4a82a7e35298ec634b8b P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:34.523545 7499 raft_consensus.cc:686] T 82cfe99cd6fe4a82a7e35298ec634b8b P 4127249ba42644388d225d614c935a25 [term 1 LEADER]: Becoming Leader. State: Replica: 4127249ba42644388d225d614c935a25, State: Running, Role: LEADER
I20240503 01:59:34.523674 7499 consensus_queue.cc:237] T 82cfe99cd6fe4a82a7e35298ec634b8b P 4127249ba42644388d225d614c935a25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } }
I20240503 01:59:34.525225 6223 catalog_manager.cc:5484] T 82cfe99cd6fe4a82a7e35298ec634b8b P 4127249ba42644388d225d614c935a25 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4127249ba42644388d225d614c935a25 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "4127249ba42644388d225d614c935a25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } health_report { overall_health: UNKNOWN } } }
W20240503 01:59:34.648392 7791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:34.648677 7791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:34.651583 7791 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_6'
I20240503 01:59:34.725814 6758 raft_consensus.cc:1261] T 0a369bc048e74fc8ba666d3318d66ae8 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Refusing update from remote peer 4127249ba42644388d225d614c935a25: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:34.726322 6495 raft_consensus.cc:1261] T 82cfe99cd6fe4a82a7e35298ec634b8b P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Refusing update from remote peer 4127249ba42644388d225d614c935a25: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:34.726480 6625 raft_consensus.cc:1261] T 0a369bc048e74fc8ba666d3318d66ae8 P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Refusing update from remote peer 4127249ba42644388d225d614c935a25: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:34.726833 7723 consensus_queue.cc:1035] T 82cfe99cd6fe4a82a7e35298ec634b8b P 4127249ba42644388d225d614c935a25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:34.726902 7499 consensus_queue.cc:1035] T 0a369bc048e74fc8ba666d3318d66ae8 P 4127249ba42644388d225d614c935a25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:34.729133 6758 raft_consensus.cc:1261] T 120d5dd4f26c42399702141094ce7982 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Refusing update from remote peer 0c4821d9898d42799c5789d4d2da3f63: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:34.729496 7770 consensus_queue.cc:1035] T 120d5dd4f26c42399702141094ce7982 P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:34.733083 7499 consensus_queue.cc:1035] T 0a369bc048e74fc8ba666d3318d66ae8 P 4127249ba42644388d225d614c935a25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:34.738330 6365 raft_consensus.cc:1261] T 120d5dd4f26c42399702141094ce7982 P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Refusing update from remote peer 0c4821d9898d42799c5789d4d2da3f63: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:34.738752 7770 consensus_queue.cc:1035] T 120d5dd4f26c42399702141094ce7982 P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:34.739342 6756 raft_consensus.cc:1261] T 82cfe99cd6fe4a82a7e35298ec634b8b P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Refusing update from remote peer 4127249ba42644388d225d614c935a25: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:34.739641 7643 mvcc.cc:205] Tried to move back new op lower bound from 7023417650073055232 to 7023417649090945024. Current Snapshot: MvccSnapshot[applied={T|T < 7023417650073055232}]
I20240503 01:59:34.739861 7723 consensus_queue.cc:1035] T 82cfe99cd6fe4a82a7e35298ec634b8b P 4127249ba42644388d225d614c935a25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:34.741062 7640 mvcc.cc:205] Tried to move back new op lower bound from 7023417650073055232 to 7023417649090945024. Current Snapshot: MvccSnapshot[applied={T|T < 7023417650073055232}]
INSERT report
rows total: 2000
time total: 100.202 ms
time per row: 0.0501009 ms
I20240503 01:59:34.791074 6222 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44084:
name: "rebalance_test_table_7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:59:34.796612 6606 tablet_service.cc:1467] Processing CreateTablet for tablet 4e7ceb1453804246a41bd1dda0e39d02 (DEFAULT_TABLE table=rebalance_test_table_7 [id=dbd6f15c28634e3dac01249aa7cc2e25]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:34.796959 6606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e7ceb1453804246a41bd1dda0e39d02. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:34.797300 6476 tablet_service.cc:1467] Processing CreateTablet for tablet 4e7ceb1453804246a41bd1dda0e39d02 (DEFAULT_TABLE table=rebalance_test_table_7 [id=dbd6f15c28634e3dac01249aa7cc2e25]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:34.797331 6474 tablet_service.cc:1467] Processing CreateTablet for tablet c3337c95b354418faf37684e7440b1c7 (DEFAULT_TABLE table=rebalance_test_table_7 [id=dbd6f15c28634e3dac01249aa7cc2e25]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:34.797528 6476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e7ceb1453804246a41bd1dda0e39d02. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:34.797648 6474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3337c95b354418faf37684e7440b1c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:34.799171 7727 tablet_bootstrap.cc:492] T 4e7ceb1453804246a41bd1dda0e39d02 P 79163c69f6f04a9abd0208d74e1c61be: Bootstrap starting.
I20240503 01:59:34.799655 7731 tablet_bootstrap.cc:492] T 4e7ceb1453804246a41bd1dda0e39d02 P 0c4821d9898d42799c5789d4d2da3f63: Bootstrap starting.
I20240503 01:59:34.800119 7727 tablet_bootstrap.cc:654] T 4e7ceb1453804246a41bd1dda0e39d02 P 79163c69f6f04a9abd0208d74e1c61be: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:34.800541 7731 tablet_bootstrap.cc:654] T 4e7ceb1453804246a41bd1dda0e39d02 P 0c4821d9898d42799c5789d4d2da3f63: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:34.801111 6738 tablet_service.cc:1467] Processing CreateTablet for tablet 23b3b3d386544e5d9bfae8041cdf01a0 (DEFAULT_TABLE table=rebalance_test_table_7 [id=dbd6f15c28634e3dac01249aa7cc2e25]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:34.801229 6737 tablet_service.cc:1467] Processing CreateTablet for tablet c3337c95b354418faf37684e7440b1c7 (DEFAULT_TABLE table=rebalance_test_table_7 [id=dbd6f15c28634e3dac01249aa7cc2e25]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:34.801368 6738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23b3b3d386544e5d9bfae8041cdf01a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:34.801492 6737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3337c95b354418faf37684e7440b1c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:34.803432 7726 tablet_bootstrap.cc:492] T 23b3b3d386544e5d9bfae8041cdf01a0 P b6b1d5677b8d47788b7968f64b3dac3c: Bootstrap starting.
I20240503 01:59:34.804289 7726 tablet_bootstrap.cc:654] T 23b3b3d386544e5d9bfae8041cdf01a0 P b6b1d5677b8d47788b7968f64b3dac3c: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:34.805594 6343 tablet_service.cc:1467] Processing CreateTablet for tablet 23b3b3d386544e5d9bfae8041cdf01a0 (DEFAULT_TABLE table=rebalance_test_table_7 [id=dbd6f15c28634e3dac01249aa7cc2e25]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:34.805617 6344 tablet_service.cc:1467] Processing CreateTablet for tablet 4e7ceb1453804246a41bd1dda0e39d02 (DEFAULT_TABLE table=rebalance_test_table_7 [id=dbd6f15c28634e3dac01249aa7cc2e25]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:34.805754 7726 tablet_bootstrap.cc:492] T 23b3b3d386544e5d9bfae8041cdf01a0 P b6b1d5677b8d47788b7968f64b3dac3c: No bootstrap required, opened a new log
I20240503 01:59:34.805840 7726 ts_tablet_manager.cc:1367] T 23b3b3d386544e5d9bfae8041cdf01a0 P b6b1d5677b8d47788b7968f64b3dac3c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:34.805860 6343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23b3b3d386544e5d9bfae8041cdf01a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:34.805969 6607 tablet_service.cc:1467] Processing CreateTablet for tablet c3337c95b354418faf37684e7440b1c7 (DEFAULT_TABLE table=rebalance_test_table_7 [id=dbd6f15c28634e3dac01249aa7cc2e25]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:34.806213 6607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3337c95b354418faf37684e7440b1c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:34.806286 7726 raft_consensus.cc:348] T 23b3b3d386544e5d9bfae8041cdf01a0 P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:34.806387 7726 raft_consensus.cc:374] T 23b3b3d386544e5d9bfae8041cdf01a0 P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:34.806427 7726 raft_consensus.cc:729] T 23b3b3d386544e5d9bfae8041cdf01a0 P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6b1d5677b8d47788b7968f64b3dac3c, State: Initialized, Role: FOLLOWER
I20240503 01:59:34.806563 7726 consensus_queue.cc:260] T 23b3b3d386544e5d9bfae8041cdf01a0 P b6b1d5677b8d47788b7968f64b3dac3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:34.806844 7726 ts_tablet_manager.cc:1398] T 23b3b3d386544e5d9bfae8041cdf01a0 P b6b1d5677b8d47788b7968f64b3dac3c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:34.807241 6606 tablet_service.cc:1467] Processing CreateTablet for tablet 23b3b3d386544e5d9bfae8041cdf01a0 (DEFAULT_TABLE table=rebalance_test_table_7 [id=dbd6f15c28634e3dac01249aa7cc2e25]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:34.807446 6606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23b3b3d386544e5d9bfae8041cdf01a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:34.807899 6344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e7ceb1453804246a41bd1dda0e39d02. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:34.807974 7670 tablet_bootstrap.cc:492] T 23b3b3d386544e5d9bfae8041cdf01a0 P 4127249ba42644388d225d614c935a25: Bootstrap starting.
I20240503 01:59:34.808755 7670 tablet_bootstrap.cc:654] T 23b3b3d386544e5d9bfae8041cdf01a0 P 4127249ba42644388d225d614c935a25: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:34.809011 7727 tablet_bootstrap.cc:492] T 4e7ceb1453804246a41bd1dda0e39d02 P 79163c69f6f04a9abd0208d74e1c61be: No bootstrap required, opened a new log
I20240503 01:59:34.809070 7727 ts_tablet_manager.cc:1367] T 4e7ceb1453804246a41bd1dda0e39d02 P 79163c69f6f04a9abd0208d74e1c61be: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.002s
I20240503 01:59:34.809655 7727 raft_consensus.cc:348] T 4e7ceb1453804246a41bd1dda0e39d02 P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:34.809736 7727 raft_consensus.cc:374] T 4e7ceb1453804246a41bd1dda0e39d02 P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:34.809757 7727 raft_consensus.cc:729] T 4e7ceb1453804246a41bd1dda0e39d02 P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79163c69f6f04a9abd0208d74e1c61be, State: Initialized, Role: FOLLOWER
I20240503 01:59:34.809890 7670 tablet_bootstrap.cc:492] T 23b3b3d386544e5d9bfae8041cdf01a0 P 4127249ba42644388d225d614c935a25: No bootstrap required, opened a new log
I20240503 01:59:34.809877 7727 consensus_queue.cc:260] T 4e7ceb1453804246a41bd1dda0e39d02 P 79163c69f6f04a9abd0208d74e1c61be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:34.809952 7670 ts_tablet_manager.cc:1367] T 23b3b3d386544e5d9bfae8041cdf01a0 P 4127249ba42644388d225d614c935a25: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 01:59:34.810182 7727 ts_tablet_manager.cc:1398] T 4e7ceb1453804246a41bd1dda0e39d02 P 79163c69f6f04a9abd0208d74e1c61be: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:34.810336 7727 tablet_bootstrap.cc:492] T c3337c95b354418faf37684e7440b1c7 P 79163c69f6f04a9abd0208d74e1c61be: Bootstrap starting.
I20240503 01:59:34.810321 7670 raft_consensus.cc:348] T 23b3b3d386544e5d9bfae8041cdf01a0 P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:34.810420 7670 raft_consensus.cc:374] T 23b3b3d386544e5d9bfae8041cdf01a0 P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:34.810449 7670 raft_consensus.cc:729] T 23b3b3d386544e5d9bfae8041cdf01a0 P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4127249ba42644388d225d614c935a25, State: Initialized, Role: FOLLOWER
I20240503 01:59:34.810564 7670 consensus_queue.cc:260] T 23b3b3d386544e5d9bfae8041cdf01a0 P 4127249ba42644388d225d614c935a25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:34.810783 7670 ts_tablet_manager.cc:1398] T 23b3b3d386544e5d9bfae8041cdf01a0 P 4127249ba42644388d225d614c935a25: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:34.810920 7670 tablet_bootstrap.cc:492] T 4e7ceb1453804246a41bd1dda0e39d02 P 4127249ba42644388d225d614c935a25: Bootstrap starting.
I20240503 01:59:34.811105 7727 tablet_bootstrap.cc:654] T c3337c95b354418faf37684e7440b1c7 P 79163c69f6f04a9abd0208d74e1c61be: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:34.811679 7670 tablet_bootstrap.cc:654] T 4e7ceb1453804246a41bd1dda0e39d02 P 4127249ba42644388d225d614c935a25: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:34.812229 7726 tablet_bootstrap.cc:492] T c3337c95b354418faf37684e7440b1c7 P b6b1d5677b8d47788b7968f64b3dac3c: Bootstrap starting.
I20240503 01:59:34.813141 7726 tablet_bootstrap.cc:654] T c3337c95b354418faf37684e7440b1c7 P b6b1d5677b8d47788b7968f64b3dac3c: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:34.817514 7726 tablet_bootstrap.cc:492] T c3337c95b354418faf37684e7440b1c7 P b6b1d5677b8d47788b7968f64b3dac3c: No bootstrap required, opened a new log
I20240503 01:59:34.817590 7726 ts_tablet_manager.cc:1367] T c3337c95b354418faf37684e7440b1c7 P b6b1d5677b8d47788b7968f64b3dac3c: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20240503 01:59:34.818114 7726 raft_consensus.cc:348] T c3337c95b354418faf37684e7440b1c7 P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } }
I20240503 01:59:34.818217 7726 raft_consensus.cc:374] T c3337c95b354418faf37684e7440b1c7 P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:34.818246 7726 raft_consensus.cc:729] T c3337c95b354418faf37684e7440b1c7 P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6b1d5677b8d47788b7968f64b3dac3c, State: Initialized, Role: FOLLOWER
I20240503 01:59:34.818359 7726 consensus_queue.cc:260] T c3337c95b354418faf37684e7440b1c7 P b6b1d5677b8d47788b7968f64b3dac3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } }
I20240503 01:59:34.818631 7726 ts_tablet_manager.cc:1398] T c3337c95b354418faf37684e7440b1c7 P b6b1d5677b8d47788b7968f64b3dac3c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:34.819262 7727 tablet_bootstrap.cc:492] T c3337c95b354418faf37684e7440b1c7 P 79163c69f6f04a9abd0208d74e1c61be: No bootstrap required, opened a new log
I20240503 01:59:34.819343 7727 ts_tablet_manager.cc:1367] T c3337c95b354418faf37684e7440b1c7 P 79163c69f6f04a9abd0208d74e1c61be: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.001s
I20240503 01:59:34.819479 7731 tablet_bootstrap.cc:492] T 4e7ceb1453804246a41bd1dda0e39d02 P 0c4821d9898d42799c5789d4d2da3f63: No bootstrap required, opened a new log
I20240503 01:59:34.819571 7731 ts_tablet_manager.cc:1367] T 4e7ceb1453804246a41bd1dda0e39d02 P 0c4821d9898d42799c5789d4d2da3f63: Time spent bootstrapping tablet: real 0.020s user 0.000s sys 0.002s
I20240503 01:59:34.819769 7727 raft_consensus.cc:348] T c3337c95b354418faf37684e7440b1c7 P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } }
I20240503 01:59:34.819870 7727 raft_consensus.cc:374] T c3337c95b354418faf37684e7440b1c7 P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:34.819908 7727 raft_consensus.cc:729] T c3337c95b354418faf37684e7440b1c7 P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79163c69f6f04a9abd0208d74e1c61be, State: Initialized, Role: FOLLOWER
I20240503 01:59:34.820008 7731 raft_consensus.cc:348] T 4e7ceb1453804246a41bd1dda0e39d02 P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:34.820022 7727 consensus_queue.cc:260] T c3337c95b354418faf37684e7440b1c7 P 79163c69f6f04a9abd0208d74e1c61be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } }
I20240503 01:59:34.820106 7731 raft_consensus.cc:374] T 4e7ceb1453804246a41bd1dda0e39d02 P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:34.820137 7731 raft_consensus.cc:729] T 4e7ceb1453804246a41bd1dda0e39d02 P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c4821d9898d42799c5789d4d2da3f63, State: Initialized, Role: FOLLOWER
I20240503 01:59:34.820243 7731 consensus_queue.cc:260] T 4e7ceb1453804246a41bd1dda0e39d02 P 0c4821d9898d42799c5789d4d2da3f63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:34.820284 7727 ts_tablet_manager.cc:1398] T c3337c95b354418faf37684e7440b1c7 P 79163c69f6f04a9abd0208d74e1c61be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:34.820425 7727 tablet_bootstrap.cc:492] T 23b3b3d386544e5d9bfae8041cdf01a0 P 79163c69f6f04a9abd0208d74e1c61be: Bootstrap starting.
I20240503 01:59:34.820515 7731 ts_tablet_manager.cc:1398] T 4e7ceb1453804246a41bd1dda0e39d02 P 0c4821d9898d42799c5789d4d2da3f63: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:34.821024 7731 tablet_bootstrap.cc:492] T c3337c95b354418faf37684e7440b1c7 P 0c4821d9898d42799c5789d4d2da3f63: Bootstrap starting.
I20240503 01:59:34.821285 7670 tablet_bootstrap.cc:492] T 4e7ceb1453804246a41bd1dda0e39d02 P 4127249ba42644388d225d614c935a25: No bootstrap required, opened a new log
I20240503 01:59:34.821359 7670 ts_tablet_manager.cc:1367] T 4e7ceb1453804246a41bd1dda0e39d02 P 4127249ba42644388d225d614c935a25: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.001s
I20240503 01:59:34.821780 7670 raft_consensus.cc:348] T 4e7ceb1453804246a41bd1dda0e39d02 P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:34.821868 7670 raft_consensus.cc:374] T 4e7ceb1453804246a41bd1dda0e39d02 P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:34.821903 7670 raft_consensus.cc:729] T 4e7ceb1453804246a41bd1dda0e39d02 P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4127249ba42644388d225d614c935a25, State: Initialized, Role: FOLLOWER
I20240503 01:59:34.821918 7731 tablet_bootstrap.cc:654] T c3337c95b354418faf37684e7440b1c7 P 0c4821d9898d42799c5789d4d2da3f63: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:34.822012 7670 consensus_queue.cc:260] T 4e7ceb1453804246a41bd1dda0e39d02 P 4127249ba42644388d225d614c935a25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:34.822247 7670 ts_tablet_manager.cc:1398] T 4e7ceb1453804246a41bd1dda0e39d02 P 4127249ba42644388d225d614c935a25: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:34.823632 7731 tablet_bootstrap.cc:492] T c3337c95b354418faf37684e7440b1c7 P 0c4821d9898d42799c5789d4d2da3f63: No bootstrap required, opened a new log
I20240503 01:59:34.823714 7731 ts_tablet_manager.cc:1367] T c3337c95b354418faf37684e7440b1c7 P 0c4821d9898d42799c5789d4d2da3f63: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 01:59:34.824132 7731 raft_consensus.cc:348] T c3337c95b354418faf37684e7440b1c7 P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } }
I20240503 01:59:34.824244 7731 raft_consensus.cc:374] T c3337c95b354418faf37684e7440b1c7 P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:34.824288 7731 raft_consensus.cc:729] T c3337c95b354418faf37684e7440b1c7 P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c4821d9898d42799c5789d4d2da3f63, State: Initialized, Role: FOLLOWER
I20240503 01:59:34.824417 7731 consensus_queue.cc:260] T c3337c95b354418faf37684e7440b1c7 P 0c4821d9898d42799c5789d4d2da3f63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } }
I20240503 01:59:34.824720 7731 ts_tablet_manager.cc:1398] T c3337c95b354418faf37684e7440b1c7 P 0c4821d9898d42799c5789d4d2da3f63: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:34.825644 7727 tablet_bootstrap.cc:654] T 23b3b3d386544e5d9bfae8041cdf01a0 P 79163c69f6f04a9abd0208d74e1c61be: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:34.826906 7727 tablet_bootstrap.cc:492] T 23b3b3d386544e5d9bfae8041cdf01a0 P 79163c69f6f04a9abd0208d74e1c61be: No bootstrap required, opened a new log
I20240503 01:59:34.826992 7727 ts_tablet_manager.cc:1367] T 23b3b3d386544e5d9bfae8041cdf01a0 P 79163c69f6f04a9abd0208d74e1c61be: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 01:59:34.827450 7727 raft_consensus.cc:348] T 23b3b3d386544e5d9bfae8041cdf01a0 P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:34.827561 7727 raft_consensus.cc:374] T 23b3b3d386544e5d9bfae8041cdf01a0 P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:34.827589 7727 raft_consensus.cc:729] T 23b3b3d386544e5d9bfae8041cdf01a0 P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79163c69f6f04a9abd0208d74e1c61be, State: Initialized, Role: FOLLOWER
I20240503 01:59:34.827704 7727 consensus_queue.cc:260] T 23b3b3d386544e5d9bfae8041cdf01a0 P 79163c69f6f04a9abd0208d74e1c61be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:34.827947 7727 ts_tablet_manager.cc:1398] T 23b3b3d386544e5d9bfae8041cdf01a0 P 79163c69f6f04a9abd0208d74e1c61be: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:34.841980 7647 raft_consensus.cc:482] T c3337c95b354418faf37684e7440b1c7 P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:34.842077 7647 raft_consensus.cc:504] T c3337c95b354418faf37684e7440b1c7 P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } }
I20240503 01:59:34.842443 7647 leader_election.cc:288] T c3337c95b354418faf37684e7440b1c7 P 79163c69f6f04a9abd0208d74e1c61be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b6b1d5677b8d47788b7968f64b3dac3c (127.0.99.132:46067), 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130:34189)
I20240503 01:59:34.842764 6495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3337c95b354418faf37684e7440b1c7" candidate_uuid: "79163c69f6f04a9abd0208d74e1c61be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c4821d9898d42799c5789d4d2da3f63" is_pre_election: true
I20240503 01:59:34.842871 6495 raft_consensus.cc:2454] T c3337c95b354418faf37684e7440b1c7 P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79163c69f6f04a9abd0208d74e1c61be in term 0.
I20240503 01:59:34.843101 6563 leader_election.cc:302] T c3337c95b354418faf37684e7440b1c7 P 79163c69f6f04a9abd0208d74e1c61be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c4821d9898d42799c5789d4d2da3f63, 79163c69f6f04a9abd0208d74e1c61be; no voters:
I20240503 01:59:34.842753 6756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3337c95b354418faf37684e7440b1c7" candidate_uuid: "79163c69f6f04a9abd0208d74e1c61be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" is_pre_election: true
I20240503 01:59:34.843312 7647 raft_consensus.cc:2781] T c3337c95b354418faf37684e7440b1c7 P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:34.843310 6756 raft_consensus.cc:2454] T c3337c95b354418faf37684e7440b1c7 P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79163c69f6f04a9abd0208d74e1c61be in term 0.
I20240503 01:59:34.843371 7647 raft_consensus.cc:482] T c3337c95b354418faf37684e7440b1c7 P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:34.843410 7647 raft_consensus.cc:3037] T c3337c95b354418faf37684e7440b1c7 P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:34.844205 7647 raft_consensus.cc:504] T c3337c95b354418faf37684e7440b1c7 P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } }
I20240503 01:59:34.844488 7647 leader_election.cc:288] T c3337c95b354418faf37684e7440b1c7 P 79163c69f6f04a9abd0208d74e1c61be [CANDIDATE]: Term 1 election: Requested vote from peers b6b1d5677b8d47788b7968f64b3dac3c (127.0.99.132:46067), 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130:34189)
I20240503 01:59:34.844798 6756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3337c95b354418faf37684e7440b1c7" candidate_uuid: "79163c69f6f04a9abd0208d74e1c61be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6b1d5677b8d47788b7968f64b3dac3c"
I20240503 01:59:34.844811 6495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3337c95b354418faf37684e7440b1c7" candidate_uuid: "79163c69f6f04a9abd0208d74e1c61be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c4821d9898d42799c5789d4d2da3f63"
I20240503 01:59:34.844902 6756 raft_consensus.cc:3037] T c3337c95b354418faf37684e7440b1c7 P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:34.844905 6495 raft_consensus.cc:3037] T c3337c95b354418faf37684e7440b1c7 P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:34.845784 6495 raft_consensus.cc:2454] T c3337c95b354418faf37684e7440b1c7 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79163c69f6f04a9abd0208d74e1c61be in term 1.
I20240503 01:59:34.845836 6756 raft_consensus.cc:2454] T c3337c95b354418faf37684e7440b1c7 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79163c69f6f04a9abd0208d74e1c61be in term 1.
I20240503 01:59:34.846053 6563 leader_election.cc:302] T c3337c95b354418faf37684e7440b1c7 P 79163c69f6f04a9abd0208d74e1c61be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c4821d9898d42799c5789d4d2da3f63, 79163c69f6f04a9abd0208d74e1c61be; no voters:
I20240503 01:59:34.846241 7647 raft_consensus.cc:2781] T c3337c95b354418faf37684e7440b1c7 P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:34.846349 7647 raft_consensus.cc:686] T c3337c95b354418faf37684e7440b1c7 P 79163c69f6f04a9abd0208d74e1c61be [term 1 LEADER]: Becoming Leader. State: Replica: 79163c69f6f04a9abd0208d74e1c61be, State: Running, Role: LEADER
I20240503 01:59:34.846469 7647 consensus_queue.cc:237] T c3337c95b354418faf37684e7440b1c7 P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } }
I20240503 01:59:34.847966 6222 catalog_manager.cc:5484] T c3337c95b354418faf37684e7440b1c7 P 79163c69f6f04a9abd0208d74e1c61be reported cstate change: term changed from 0 to 1, leader changed from <none> to 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131). New cstate: current_term: 1 leader_uuid: "79163c69f6f04a9abd0208d74e1c61be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:35.035126 7826 raft_consensus.cc:482] T 23b3b3d386544e5d9bfae8041cdf01a0 P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:35.035275 7826 raft_consensus.cc:504] T 23b3b3d386544e5d9bfae8041cdf01a0 P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:35.035698 7826 leader_election.cc:288] T 23b3b3d386544e5d9bfae8041cdf01a0 P 4127249ba42644388d225d614c935a25 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131:44005), b6b1d5677b8d47788b7968f64b3dac3c (127.0.99.132:46067)
I20240503 01:59:35.036134 6756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b3b3d386544e5d9bfae8041cdf01a0" candidate_uuid: "4127249ba42644388d225d614c935a25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" is_pre_election: true
I20240503 01:59:35.036152 6625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b3b3d386544e5d9bfae8041cdf01a0" candidate_uuid: "4127249ba42644388d225d614c935a25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79163c69f6f04a9abd0208d74e1c61be" is_pre_election: true
I20240503 01:59:35.036295 6756 raft_consensus.cc:2454] T 23b3b3d386544e5d9bfae8041cdf01a0 P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4127249ba42644388d225d614c935a25 in term 0.
I20240503 01:59:35.036362 6625 raft_consensus.cc:2454] T 23b3b3d386544e5d9bfae8041cdf01a0 P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4127249ba42644388d225d614c935a25 in term 0.
I20240503 01:59:35.036625 6300 leader_election.cc:302] T 23b3b3d386544e5d9bfae8041cdf01a0 P 4127249ba42644388d225d614c935a25 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4127249ba42644388d225d614c935a25, b6b1d5677b8d47788b7968f64b3dac3c; no voters:
I20240503 01:59:35.036841 7825 raft_consensus.cc:2781] T 23b3b3d386544e5d9bfae8041cdf01a0 P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:35.036898 7825 raft_consensus.cc:482] T 23b3b3d386544e5d9bfae8041cdf01a0 P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:35.036931 7825 raft_consensus.cc:3037] T 23b3b3d386544e5d9bfae8041cdf01a0 P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:35.037799 7825 raft_consensus.cc:504] T 23b3b3d386544e5d9bfae8041cdf01a0 P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:35.038122 7825 leader_election.cc:288] T 23b3b3d386544e5d9bfae8041cdf01a0 P 4127249ba42644388d225d614c935a25 [CANDIDATE]: Term 1 election: Requested vote from peers 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131:44005), b6b1d5677b8d47788b7968f64b3dac3c (127.0.99.132:46067)
I20240503 01:59:35.038424 6625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b3b3d386544e5d9bfae8041cdf01a0" candidate_uuid: "4127249ba42644388d225d614c935a25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79163c69f6f04a9abd0208d74e1c61be"
I20240503 01:59:35.038437 6756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b3b3d386544e5d9bfae8041cdf01a0" candidate_uuid: "4127249ba42644388d225d614c935a25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6b1d5677b8d47788b7968f64b3dac3c"
I20240503 01:59:35.038528 6756 raft_consensus.cc:3037] T 23b3b3d386544e5d9bfae8041cdf01a0 P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:35.038517 6625 raft_consensus.cc:3037] T 23b3b3d386544e5d9bfae8041cdf01a0 P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:35.039261 6625 raft_consensus.cc:2454] T 23b3b3d386544e5d9bfae8041cdf01a0 P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4127249ba42644388d225d614c935a25 in term 1.
I20240503 01:59:35.039316 6756 raft_consensus.cc:2454] T 23b3b3d386544e5d9bfae8041cdf01a0 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4127249ba42644388d225d614c935a25 in term 1.
I20240503 01:59:35.039556 6299 leader_election.cc:302] T 23b3b3d386544e5d9bfae8041cdf01a0 P 4127249ba42644388d225d614c935a25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4127249ba42644388d225d614c935a25, 79163c69f6f04a9abd0208d74e1c61be; no voters:
I20240503 01:59:35.039721 7825 raft_consensus.cc:2781] T 23b3b3d386544e5d9bfae8041cdf01a0 P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:35.039819 7825 raft_consensus.cc:686] T 23b3b3d386544e5d9bfae8041cdf01a0 P 4127249ba42644388d225d614c935a25 [term 1 LEADER]: Becoming Leader. State: Replica: 4127249ba42644388d225d614c935a25, State: Running, Role: LEADER
I20240503 01:59:35.039963 7825 consensus_queue.cc:237] T 23b3b3d386544e5d9bfae8041cdf01a0 P 4127249ba42644388d225d614c935a25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:35.041679 6222 catalog_manager.cc:5484] T 23b3b3d386544e5d9bfae8041cdf01a0 P 4127249ba42644388d225d614c935a25 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4127249ba42644388d225d614c935a25 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "4127249ba42644388d225d614c935a25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } health_report { overall_health: HEALTHY } } }
I20240503 01:59:35.080840 7825 raft_consensus.cc:482] T 4e7ceb1453804246a41bd1dda0e39d02 P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:35.080960 7825 raft_consensus.cc:504] T 4e7ceb1453804246a41bd1dda0e39d02 P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:35.081326 7825 leader_election.cc:288] T 4e7ceb1453804246a41bd1dda0e39d02 P 4127249ba42644388d225d614c935a25 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131:44005), 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130:34189)
I20240503 01:59:35.081812 6495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e7ceb1453804246a41bd1dda0e39d02" candidate_uuid: "4127249ba42644388d225d614c935a25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c4821d9898d42799c5789d4d2da3f63" is_pre_election: true
I20240503 01:59:35.081809 6625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e7ceb1453804246a41bd1dda0e39d02" candidate_uuid: "4127249ba42644388d225d614c935a25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79163c69f6f04a9abd0208d74e1c61be" is_pre_election: true
I20240503 01:59:35.081983 6495 raft_consensus.cc:2454] T 4e7ceb1453804246a41bd1dda0e39d02 P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4127249ba42644388d225d614c935a25 in term 0.
I20240503 01:59:35.082002 6625 raft_consensus.cc:2454] T 4e7ceb1453804246a41bd1dda0e39d02 P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4127249ba42644388d225d614c935a25 in term 0.
I20240503 01:59:35.082347 6301 leader_election.cc:302] T 4e7ceb1453804246a41bd1dda0e39d02 P 4127249ba42644388d225d614c935a25 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c4821d9898d42799c5789d4d2da3f63, 4127249ba42644388d225d614c935a25; no voters:
I20240503 01:59:35.082520 7825 raft_consensus.cc:2781] T 4e7ceb1453804246a41bd1dda0e39d02 P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:35.082580 7825 raft_consensus.cc:482] T 4e7ceb1453804246a41bd1dda0e39d02 P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:35.082623 7825 raft_consensus.cc:3037] T 4e7ceb1453804246a41bd1dda0e39d02 P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:35.083516 7825 raft_consensus.cc:504] T 4e7ceb1453804246a41bd1dda0e39d02 P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:35.083863 7825 leader_election.cc:288] T 4e7ceb1453804246a41bd1dda0e39d02 P 4127249ba42644388d225d614c935a25 [CANDIDATE]: Term 1 election: Requested vote from peers 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131:44005), 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130:34189)
I20240503 01:59:35.084234 6625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e7ceb1453804246a41bd1dda0e39d02" candidate_uuid: "4127249ba42644388d225d614c935a25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79163c69f6f04a9abd0208d74e1c61be"
I20240503 01:59:35.084268 6495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e7ceb1453804246a41bd1dda0e39d02" candidate_uuid: "4127249ba42644388d225d614c935a25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c4821d9898d42799c5789d4d2da3f63"
I20240503 01:59:35.084344 6625 raft_consensus.cc:3037] T 4e7ceb1453804246a41bd1dda0e39d02 P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:35.084386 6495 raft_consensus.cc:3037] T 4e7ceb1453804246a41bd1dda0e39d02 P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:35.085148 6625 raft_consensus.cc:2454] T 4e7ceb1453804246a41bd1dda0e39d02 P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4127249ba42644388d225d614c935a25 in term 1.
I20240503 01:59:35.085218 6495 raft_consensus.cc:2454] T 4e7ceb1453804246a41bd1dda0e39d02 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4127249ba42644388d225d614c935a25 in term 1.
I20240503 01:59:35.085494 6299 leader_election.cc:302] T 4e7ceb1453804246a41bd1dda0e39d02 P 4127249ba42644388d225d614c935a25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4127249ba42644388d225d614c935a25, 79163c69f6f04a9abd0208d74e1c61be; no voters:
I20240503 01:59:35.085645 7825 raft_consensus.cc:2781] T 4e7ceb1453804246a41bd1dda0e39d02 P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:35.085724 7825 raft_consensus.cc:686] T 4e7ceb1453804246a41bd1dda0e39d02 P 4127249ba42644388d225d614c935a25 [term 1 LEADER]: Becoming Leader. State: Replica: 4127249ba42644388d225d614c935a25, State: Running, Role: LEADER
I20240503 01:59:35.085887 7825 consensus_queue.cc:237] T 4e7ceb1453804246a41bd1dda0e39d02 P 4127249ba42644388d225d614c935a25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:35.087637 6222 catalog_manager.cc:5484] T 4e7ceb1453804246a41bd1dda0e39d02 P 4127249ba42644388d225d614c935a25 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4127249ba42644388d225d614c935a25 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "4127249ba42644388d225d614c935a25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } health_report { overall_health: HEALTHY } } }
W20240503 01:59:35.195410 7841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:35.195600 7841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:35.198467 7841 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_7'
I20240503 01:59:35.270612 6625 raft_consensus.cc:1261] T 23b3b3d386544e5d9bfae8041cdf01a0 P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Refusing update from remote peer 4127249ba42644388d225d614c935a25: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:35.270735 6756 raft_consensus.cc:1261] T 23b3b3d386544e5d9bfae8041cdf01a0 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Refusing update from remote peer 4127249ba42644388d225d614c935a25: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:35.271157 7825 consensus_queue.cc:1035] T 23b3b3d386544e5d9bfae8041cdf01a0 P 4127249ba42644388d225d614c935a25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:35.271284 7826 consensus_queue.cc:1035] T 23b3b3d386544e5d9bfae8041cdf01a0 P 4127249ba42644388d225d614c935a25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:35.276377 6495 raft_consensus.cc:1261] T c3337c95b354418faf37684e7440b1c7 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:35.276932 6495 raft_consensus.cc:1261] T 4e7ceb1453804246a41bd1dda0e39d02 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Refusing update from remote peer 4127249ba42644388d225d614c935a25: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 01:59:35.276999 7647 consensus_queue.cc:1035] T c3337c95b354418faf37684e7440b1c7 P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:35.277518 6758 raft_consensus.cc:1261] T c3337c95b354418faf37684e7440b1c7 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:35.277647 7877 consensus_queue.cc:1035] T 4e7ceb1453804246a41bd1dda0e39d02 P 4127249ba42644388d225d614c935a25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:35.277724 6627 raft_consensus.cc:1261] T 4e7ceb1453804246a41bd1dda0e39d02 P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Refusing update from remote peer 4127249ba42644388d225d614c935a25: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 01:59:35.278172 7646 consensus_queue.cc:1035] T c3337c95b354418faf37684e7440b1c7 P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:35.279058 7877 consensus_queue.cc:1035] T 4e7ceb1453804246a41bd1dda0e39d02 P 4127249ba42644388d225d614c935a25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 96.7487 ms
time per row: 0.0483744 ms
I20240503 01:59:35.336339 6223 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44084:
name: "rebalance_test_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:59:35.341585 6344 tablet_service.cc:1467] Processing CreateTablet for tablet ccb34a4b52244d90850ab04e21299f55 (DEFAULT_TABLE table=rebalance_test_table_8 [id=85186d17f9464a408a3790b0978eb502]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:35.341984 6344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccb34a4b52244d90850ab04e21299f55. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:35.342082 6737 tablet_service.cc:1467] Processing CreateTablet for tablet a4983426b2d843e7a9ed335c79178678 (DEFAULT_TABLE table=rebalance_test_table_8 [id=85186d17f9464a408a3790b0978eb502]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:35.342339 6737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4983426b2d843e7a9ed335c79178678. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:35.343780 7904 tablet_bootstrap.cc:492] T ccb34a4b52244d90850ab04e21299f55 P 4127249ba42644388d225d614c935a25: Bootstrap starting.
I20240503 01:59:35.343943 6738 tablet_service.cc:1467] Processing CreateTablet for tablet 0bdcea4e59ec4e2da63b9316f9d718be (DEFAULT_TABLE table=rebalance_test_table_8 [id=85186d17f9464a408a3790b0978eb502]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:35.344177 6738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bdcea4e59ec4e2da63b9316f9d718be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:35.344607 7904 tablet_bootstrap.cc:654] T ccb34a4b52244d90850ab04e21299f55 P 4127249ba42644388d225d614c935a25: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:35.345548 6607 tablet_service.cc:1467] Processing CreateTablet for tablet 0bdcea4e59ec4e2da63b9316f9d718be (DEFAULT_TABLE table=rebalance_test_table_8 [id=85186d17f9464a408a3790b0978eb502]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:35.345548 6606 tablet_service.cc:1467] Processing CreateTablet for tablet ccb34a4b52244d90850ab04e21299f55 (DEFAULT_TABLE table=rebalance_test_table_8 [id=85186d17f9464a408a3790b0978eb502]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:35.345835 6607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bdcea4e59ec4e2da63b9316f9d718be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:35.345978 6606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccb34a4b52244d90850ab04e21299f55. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:35.344630 6474 tablet_service.cc:1467] Processing CreateTablet for tablet ccb34a4b52244d90850ab04e21299f55 (DEFAULT_TABLE table=rebalance_test_table_8 [id=85186d17f9464a408a3790b0978eb502]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:35.346259 6474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccb34a4b52244d90850ab04e21299f55. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:35.346786 6343 tablet_service.cc:1467] Processing CreateTablet for tablet 0bdcea4e59ec4e2da63b9316f9d718be (DEFAULT_TABLE table=rebalance_test_table_8 [id=85186d17f9464a408a3790b0978eb502]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:35.347019 6343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bdcea4e59ec4e2da63b9316f9d718be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:35.344879 6476 tablet_service.cc:1467] Processing CreateTablet for tablet a4983426b2d843e7a9ed335c79178678 (DEFAULT_TABLE table=rebalance_test_table_8 [id=85186d17f9464a408a3790b0978eb502]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:35.348440 7907 tablet_bootstrap.cc:492] T ccb34a4b52244d90850ab04e21299f55 P 79163c69f6f04a9abd0208d74e1c61be: Bootstrap starting.
I20240503 01:59:35.348547 6476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4983426b2d843e7a9ed335c79178678. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:35.349278 7907 tablet_bootstrap.cc:654] T ccb34a4b52244d90850ab04e21299f55 P 79163c69f6f04a9abd0208d74e1c61be: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:35.350273 7907 tablet_bootstrap.cc:492] T ccb34a4b52244d90850ab04e21299f55 P 79163c69f6f04a9abd0208d74e1c61be: No bootstrap required, opened a new log
I20240503 01:59:35.350371 7907 ts_tablet_manager.cc:1367] T ccb34a4b52244d90850ab04e21299f55 P 79163c69f6f04a9abd0208d74e1c61be: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:35.350430 6344 tablet_service.cc:1467] Processing CreateTablet for tablet a4983426b2d843e7a9ed335c79178678 (DEFAULT_TABLE table=rebalance_test_table_8 [id=85186d17f9464a408a3790b0978eb502]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:35.350639 6344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4983426b2d843e7a9ed335c79178678. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:35.350759 7907 raft_consensus.cc:348] T ccb34a4b52244d90850ab04e21299f55 P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } }
I20240503 01:59:35.350867 7907 raft_consensus.cc:374] T ccb34a4b52244d90850ab04e21299f55 P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:35.350903 7907 raft_consensus.cc:729] T ccb34a4b52244d90850ab04e21299f55 P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79163c69f6f04a9abd0208d74e1c61be, State: Initialized, Role: FOLLOWER
I20240503 01:59:35.351013 7907 consensus_queue.cc:260] T ccb34a4b52244d90850ab04e21299f55 P 79163c69f6f04a9abd0208d74e1c61be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } }
I20240503 01:59:35.351261 7907 ts_tablet_manager.cc:1398] T ccb34a4b52244d90850ab04e21299f55 P 79163c69f6f04a9abd0208d74e1c61be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:35.352329 7907 tablet_bootstrap.cc:492] T 0bdcea4e59ec4e2da63b9316f9d718be P 79163c69f6f04a9abd0208d74e1c61be: Bootstrap starting.
I20240503 01:59:35.352414 7908 tablet_bootstrap.cc:492] T ccb34a4b52244d90850ab04e21299f55 P 0c4821d9898d42799c5789d4d2da3f63: Bootstrap starting.
I20240503 01:59:35.353143 7907 tablet_bootstrap.cc:654] T 0bdcea4e59ec4e2da63b9316f9d718be P 79163c69f6f04a9abd0208d74e1c61be: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:35.354133 7908 tablet_bootstrap.cc:654] T ccb34a4b52244d90850ab04e21299f55 P 0c4821d9898d42799c5789d4d2da3f63: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:35.354872 7904 tablet_bootstrap.cc:492] T ccb34a4b52244d90850ab04e21299f55 P 4127249ba42644388d225d614c935a25: No bootstrap required, opened a new log
I20240503 01:59:35.354959 7904 ts_tablet_manager.cc:1367] T ccb34a4b52244d90850ab04e21299f55 P 4127249ba42644388d225d614c935a25: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.002s
I20240503 01:59:35.355365 7904 raft_consensus.cc:348] T ccb34a4b52244d90850ab04e21299f55 P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } }
I20240503 01:59:35.355463 7904 raft_consensus.cc:374] T ccb34a4b52244d90850ab04e21299f55 P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:35.355494 7904 raft_consensus.cc:729] T ccb34a4b52244d90850ab04e21299f55 P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4127249ba42644388d225d614c935a25, State: Initialized, Role: FOLLOWER
I20240503 01:59:35.355628 7904 consensus_queue.cc:260] T ccb34a4b52244d90850ab04e21299f55 P 4127249ba42644388d225d614c935a25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } }
I20240503 01:59:35.355937 7904 ts_tablet_manager.cc:1398] T ccb34a4b52244d90850ab04e21299f55 P 4127249ba42644388d225d614c935a25: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:35.356112 7904 tablet_bootstrap.cc:492] T 0bdcea4e59ec4e2da63b9316f9d718be P 4127249ba42644388d225d614c935a25: Bootstrap starting.
I20240503 01:59:35.357225 7904 tablet_bootstrap.cc:654] T 0bdcea4e59ec4e2da63b9316f9d718be P 4127249ba42644388d225d614c935a25: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:35.357395 7907 tablet_bootstrap.cc:492] T 0bdcea4e59ec4e2da63b9316f9d718be P 79163c69f6f04a9abd0208d74e1c61be: No bootstrap required, opened a new log
I20240503 01:59:35.357468 7907 ts_tablet_manager.cc:1367] T 0bdcea4e59ec4e2da63b9316f9d718be P 79163c69f6f04a9abd0208d74e1c61be: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240503 01:59:35.354310 7906 tablet_bootstrap.cc:492] T 0bdcea4e59ec4e2da63b9316f9d718be P b6b1d5677b8d47788b7968f64b3dac3c: Bootstrap starting.
I20240503 01:59:35.357887 7907 raft_consensus.cc:348] T 0bdcea4e59ec4e2da63b9316f9d718be P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } }
I20240503 01:59:35.357970 7907 raft_consensus.cc:374] T 0bdcea4e59ec4e2da63b9316f9d718be P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:35.358006 7907 raft_consensus.cc:729] T 0bdcea4e59ec4e2da63b9316f9d718be P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79163c69f6f04a9abd0208d74e1c61be, State: Initialized, Role: FOLLOWER
I20240503 01:59:35.358122 7907 consensus_queue.cc:260] T 0bdcea4e59ec4e2da63b9316f9d718be P 79163c69f6f04a9abd0208d74e1c61be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } }
I20240503 01:59:35.358215 7906 tablet_bootstrap.cc:654] T 0bdcea4e59ec4e2da63b9316f9d718be P b6b1d5677b8d47788b7968f64b3dac3c: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:35.358387 7907 ts_tablet_manager.cc:1398] T 0bdcea4e59ec4e2da63b9316f9d718be P 79163c69f6f04a9abd0208d74e1c61be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:35.359452 7906 tablet_bootstrap.cc:492] T 0bdcea4e59ec4e2da63b9316f9d718be P b6b1d5677b8d47788b7968f64b3dac3c: No bootstrap required, opened a new log
I20240503 01:59:35.359580 7906 ts_tablet_manager.cc:1367] T 0bdcea4e59ec4e2da63b9316f9d718be P b6b1d5677b8d47788b7968f64b3dac3c: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 01:59:35.359926 7904 tablet_bootstrap.cc:492] T 0bdcea4e59ec4e2da63b9316f9d718be P 4127249ba42644388d225d614c935a25: No bootstrap required, opened a new log
I20240503 01:59:35.360015 7904 ts_tablet_manager.cc:1367] T 0bdcea4e59ec4e2da63b9316f9d718be P 4127249ba42644388d225d614c935a25: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240503 01:59:35.360427 7904 raft_consensus.cc:348] T 0bdcea4e59ec4e2da63b9316f9d718be P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } }
I20240503 01:59:35.360533 7904 raft_consensus.cc:374] T 0bdcea4e59ec4e2da63b9316f9d718be P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:35.360566 7904 raft_consensus.cc:729] T 0bdcea4e59ec4e2da63b9316f9d718be P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4127249ba42644388d225d614c935a25, State: Initialized, Role: FOLLOWER
I20240503 01:59:35.360714 7904 consensus_queue.cc:260] T 0bdcea4e59ec4e2da63b9316f9d718be P 4127249ba42644388d225d614c935a25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } }
I20240503 01:59:35.361001 7904 ts_tablet_manager.cc:1398] T 0bdcea4e59ec4e2da63b9316f9d718be P 4127249ba42644388d225d614c935a25: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:35.361203 7904 tablet_bootstrap.cc:492] T a4983426b2d843e7a9ed335c79178678 P 4127249ba42644388d225d614c935a25: Bootstrap starting.
I20240503 01:59:35.361485 7906 raft_consensus.cc:348] T 0bdcea4e59ec4e2da63b9316f9d718be P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } }
I20240503 01:59:35.362063 7904 tablet_bootstrap.cc:654] T a4983426b2d843e7a9ed335c79178678 P 4127249ba42644388d225d614c935a25: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:35.361663 7906 raft_consensus.cc:374] T 0bdcea4e59ec4e2da63b9316f9d718be P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:35.362447 7906 raft_consensus.cc:729] T 0bdcea4e59ec4e2da63b9316f9d718be P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6b1d5677b8d47788b7968f64b3dac3c, State: Initialized, Role: FOLLOWER
I20240503 01:59:35.362654 7906 consensus_queue.cc:260] T 0bdcea4e59ec4e2da63b9316f9d718be P b6b1d5677b8d47788b7968f64b3dac3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } }
I20240503 01:59:35.363118 7906 ts_tablet_manager.cc:1398] T 0bdcea4e59ec4e2da63b9316f9d718be P b6b1d5677b8d47788b7968f64b3dac3c: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 01:59:35.363337 7906 tablet_bootstrap.cc:492] T a4983426b2d843e7a9ed335c79178678 P b6b1d5677b8d47788b7968f64b3dac3c: Bootstrap starting.
I20240503 01:59:35.363346 7908 tablet_bootstrap.cc:492] T ccb34a4b52244d90850ab04e21299f55 P 0c4821d9898d42799c5789d4d2da3f63: No bootstrap required, opened a new log
I20240503 01:59:35.363420 7908 ts_tablet_manager.cc:1367] T ccb34a4b52244d90850ab04e21299f55 P 0c4821d9898d42799c5789d4d2da3f63: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240503 01:59:35.363866 7908 raft_consensus.cc:348] T ccb34a4b52244d90850ab04e21299f55 P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } }
I20240503 01:59:35.363986 7908 raft_consensus.cc:374] T ccb34a4b52244d90850ab04e21299f55 P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:35.364024 7908 raft_consensus.cc:729] T ccb34a4b52244d90850ab04e21299f55 P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c4821d9898d42799c5789d4d2da3f63, State: Initialized, Role: FOLLOWER
I20240503 01:59:35.364151 7908 consensus_queue.cc:260] T ccb34a4b52244d90850ab04e21299f55 P 0c4821d9898d42799c5789d4d2da3f63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } }
I20240503 01:59:35.364416 7908 ts_tablet_manager.cc:1398] T ccb34a4b52244d90850ab04e21299f55 P 0c4821d9898d42799c5789d4d2da3f63: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:35.364603 7906 tablet_bootstrap.cc:654] T a4983426b2d843e7a9ed335c79178678 P b6b1d5677b8d47788b7968f64b3dac3c: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:35.365015 7908 tablet_bootstrap.cc:492] T a4983426b2d843e7a9ed335c79178678 P 0c4821d9898d42799c5789d4d2da3f63: Bootstrap starting.
I20240503 01:59:35.365657 7908 tablet_bootstrap.cc:654] T a4983426b2d843e7a9ed335c79178678 P 0c4821d9898d42799c5789d4d2da3f63: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:35.367254 7904 tablet_bootstrap.cc:492] T a4983426b2d843e7a9ed335c79178678 P 4127249ba42644388d225d614c935a25: No bootstrap required, opened a new log
I20240503 01:59:35.367347 7904 ts_tablet_manager.cc:1367] T a4983426b2d843e7a9ed335c79178678 P 4127249ba42644388d225d614c935a25: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20240503 01:59:35.367797 7904 raft_consensus.cc:348] T a4983426b2d843e7a9ed335c79178678 P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:35.367895 7904 raft_consensus.cc:374] T a4983426b2d843e7a9ed335c79178678 P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:35.367937 7904 raft_consensus.cc:729] T a4983426b2d843e7a9ed335c79178678 P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4127249ba42644388d225d614c935a25, State: Initialized, Role: FOLLOWER
I20240503 01:59:35.368067 7904 consensus_queue.cc:260] T a4983426b2d843e7a9ed335c79178678 P 4127249ba42644388d225d614c935a25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:35.368312 7904 ts_tablet_manager.cc:1398] T a4983426b2d843e7a9ed335c79178678 P 4127249ba42644388d225d614c935a25: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:35.369310 7908 tablet_bootstrap.cc:492] T a4983426b2d843e7a9ed335c79178678 P 0c4821d9898d42799c5789d4d2da3f63: No bootstrap required, opened a new log
I20240503 01:59:35.369390 7908 ts_tablet_manager.cc:1367] T a4983426b2d843e7a9ed335c79178678 P 0c4821d9898d42799c5789d4d2da3f63: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 01:59:35.369879 7908 raft_consensus.cc:348] T a4983426b2d843e7a9ed335c79178678 P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:35.369983 7908 raft_consensus.cc:374] T a4983426b2d843e7a9ed335c79178678 P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:35.370018 7908 raft_consensus.cc:729] T a4983426b2d843e7a9ed335c79178678 P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c4821d9898d42799c5789d4d2da3f63, State: Initialized, Role: FOLLOWER
I20240503 01:59:35.370114 7908 consensus_queue.cc:260] T a4983426b2d843e7a9ed335c79178678 P 0c4821d9898d42799c5789d4d2da3f63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:35.370396 7908 ts_tablet_manager.cc:1398] T a4983426b2d843e7a9ed335c79178678 P 0c4821d9898d42799c5789d4d2da3f63: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:35.371045 7906 tablet_bootstrap.cc:492] T a4983426b2d843e7a9ed335c79178678 P b6b1d5677b8d47788b7968f64b3dac3c: No bootstrap required, opened a new log
I20240503 01:59:35.371318 7906 ts_tablet_manager.cc:1367] T a4983426b2d843e7a9ed335c79178678 P b6b1d5677b8d47788b7968f64b3dac3c: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 01:59:35.372411 7906 raft_consensus.cc:348] T a4983426b2d843e7a9ed335c79178678 P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:35.372495 7906 raft_consensus.cc:374] T a4983426b2d843e7a9ed335c79178678 P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:35.372529 7906 raft_consensus.cc:729] T a4983426b2d843e7a9ed335c79178678 P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6b1d5677b8d47788b7968f64b3dac3c, State: Initialized, Role: FOLLOWER
I20240503 01:59:35.372692 7906 consensus_queue.cc:260] T a4983426b2d843e7a9ed335c79178678 P b6b1d5677b8d47788b7968f64b3dac3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:35.372970 7906 ts_tablet_manager.cc:1398] T a4983426b2d843e7a9ed335c79178678 P b6b1d5677b8d47788b7968f64b3dac3c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:35.374341 7361 raft_consensus.cc:482] T ccb34a4b52244d90850ab04e21299f55 P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:35.374425 7361 raft_consensus.cc:504] T ccb34a4b52244d90850ab04e21299f55 P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } }
I20240503 01:59:35.374838 7361 leader_election.cc:288] T ccb34a4b52244d90850ab04e21299f55 P 0c4821d9898d42799c5789d4d2da3f63 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4127249ba42644388d225d614c935a25 (127.0.99.129:35613), 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131:44005)
I20240503 01:59:35.375109 6365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccb34a4b52244d90850ab04e21299f55" candidate_uuid: "0c4821d9898d42799c5789d4d2da3f63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4127249ba42644388d225d614c935a25" is_pre_election: true
I20240503 01:59:35.375224 6365 raft_consensus.cc:2454] T ccb34a4b52244d90850ab04e21299f55 P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c4821d9898d42799c5789d4d2da3f63 in term 0.
I20240503 01:59:35.375406 6625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccb34a4b52244d90850ab04e21299f55" candidate_uuid: "0c4821d9898d42799c5789d4d2da3f63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79163c69f6f04a9abd0208d74e1c61be" is_pre_election: true
I20240503 01:59:35.375511 6625 raft_consensus.cc:2454] T ccb34a4b52244d90850ab04e21299f55 P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c4821d9898d42799c5789d4d2da3f63 in term 0.
I20240503 01:59:35.375768 6430 leader_election.cc:302] T ccb34a4b52244d90850ab04e21299f55 P 0c4821d9898d42799c5789d4d2da3f63 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c4821d9898d42799c5789d4d2da3f63, 4127249ba42644388d225d614c935a25; no voters:
I20240503 01:59:35.375923 7361 raft_consensus.cc:2781] T ccb34a4b52244d90850ab04e21299f55 P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:35.375974 7361 raft_consensus.cc:482] T ccb34a4b52244d90850ab04e21299f55 P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:35.376013 7361 raft_consensus.cc:3037] T ccb34a4b52244d90850ab04e21299f55 P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:35.376730 7361 raft_consensus.cc:504] T ccb34a4b52244d90850ab04e21299f55 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } }
I20240503 01:59:35.377020 7361 leader_election.cc:288] T ccb34a4b52244d90850ab04e21299f55 P 0c4821d9898d42799c5789d4d2da3f63 [CANDIDATE]: Term 1 election: Requested vote from peers 4127249ba42644388d225d614c935a25 (127.0.99.129:35613), 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131:44005)
I20240503 01:59:35.377283 6365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccb34a4b52244d90850ab04e21299f55" candidate_uuid: "0c4821d9898d42799c5789d4d2da3f63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4127249ba42644388d225d614c935a25"
I20240503 01:59:35.377355 6625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccb34a4b52244d90850ab04e21299f55" candidate_uuid: "0c4821d9898d42799c5789d4d2da3f63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79163c69f6f04a9abd0208d74e1c61be"
I20240503 01:59:35.377393 6365 raft_consensus.cc:3037] T ccb34a4b52244d90850ab04e21299f55 P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:35.377445 6625 raft_consensus.cc:3037] T ccb34a4b52244d90850ab04e21299f55 P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:35.378284 6625 raft_consensus.cc:2454] T ccb34a4b52244d90850ab04e21299f55 P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c4821d9898d42799c5789d4d2da3f63 in term 1.
I20240503 01:59:35.378284 6365 raft_consensus.cc:2454] T ccb34a4b52244d90850ab04e21299f55 P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c4821d9898d42799c5789d4d2da3f63 in term 1.
I20240503 01:59:35.378525 6430 leader_election.cc:302] T ccb34a4b52244d90850ab04e21299f55 P 0c4821d9898d42799c5789d4d2da3f63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c4821d9898d42799c5789d4d2da3f63, 79163c69f6f04a9abd0208d74e1c61be; no voters:
I20240503 01:59:35.378907 7361 raft_consensus.cc:2781] T ccb34a4b52244d90850ab04e21299f55 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:35.379089 7361 raft_consensus.cc:686] T ccb34a4b52244d90850ab04e21299f55 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 LEADER]: Becoming Leader. State: Replica: 0c4821d9898d42799c5789d4d2da3f63, State: Running, Role: LEADER
I20240503 01:59:35.379235 7361 consensus_queue.cc:237] T ccb34a4b52244d90850ab04e21299f55 P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } }
I20240503 01:59:35.380636 6222 catalog_manager.cc:5484] T ccb34a4b52244d90850ab04e21299f55 P 0c4821d9898d42799c5789d4d2da3f63 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "0c4821d9898d42799c5789d4d2da3f63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:35.443380 7877 raft_consensus.cc:482] T a4983426b2d843e7a9ed335c79178678 P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:35.443511 7877 raft_consensus.cc:504] T a4983426b2d843e7a9ed335c79178678 P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:35.443986 7877 leader_election.cc:288] T a4983426b2d843e7a9ed335c79178678 P 4127249ba42644388d225d614c935a25 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130:34189), b6b1d5677b8d47788b7968f64b3dac3c (127.0.99.132:46067)
I20240503 01:59:35.444422 6756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4983426b2d843e7a9ed335c79178678" candidate_uuid: "4127249ba42644388d225d614c935a25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" is_pre_election: true
I20240503 01:59:35.444424 6494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4983426b2d843e7a9ed335c79178678" candidate_uuid: "4127249ba42644388d225d614c935a25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c4821d9898d42799c5789d4d2da3f63" is_pre_election: true
I20240503 01:59:35.444561 6494 raft_consensus.cc:2454] T a4983426b2d843e7a9ed335c79178678 P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4127249ba42644388d225d614c935a25 in term 0.
I20240503 01:59:35.444561 6756 raft_consensus.cc:2454] T a4983426b2d843e7a9ed335c79178678 P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4127249ba42644388d225d614c935a25 in term 0.
I20240503 01:59:35.444847 6300 leader_election.cc:302] T a4983426b2d843e7a9ed335c79178678 P 4127249ba42644388d225d614c935a25 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4127249ba42644388d225d614c935a25, b6b1d5677b8d47788b7968f64b3dac3c; no voters:
I20240503 01:59:35.445077 7877 raft_consensus.cc:2781] T a4983426b2d843e7a9ed335c79178678 P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:35.445134 7877 raft_consensus.cc:482] T a4983426b2d843e7a9ed335c79178678 P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:35.445183 7877 raft_consensus.cc:3037] T a4983426b2d843e7a9ed335c79178678 P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:35.446014 7877 raft_consensus.cc:504] T a4983426b2d843e7a9ed335c79178678 P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:35.446345 7877 leader_election.cc:288] T a4983426b2d843e7a9ed335c79178678 P 4127249ba42644388d225d614c935a25 [CANDIDATE]: Term 1 election: Requested vote from peers 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130:34189), b6b1d5677b8d47788b7968f64b3dac3c (127.0.99.132:46067)
I20240503 01:59:35.446636 6756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4983426b2d843e7a9ed335c79178678" candidate_uuid: "4127249ba42644388d225d614c935a25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6b1d5677b8d47788b7968f64b3dac3c"
I20240503 01:59:35.446636 6494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4983426b2d843e7a9ed335c79178678" candidate_uuid: "4127249ba42644388d225d614c935a25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c4821d9898d42799c5789d4d2da3f63"
I20240503 01:59:35.446745 6494 raft_consensus.cc:3037] T a4983426b2d843e7a9ed335c79178678 P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:35.446744 6756 raft_consensus.cc:3037] T a4983426b2d843e7a9ed335c79178678 P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:35.447640 6756 raft_consensus.cc:2454] T a4983426b2d843e7a9ed335c79178678 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4127249ba42644388d225d614c935a25 in term 1.
I20240503 01:59:35.447640 6494 raft_consensus.cc:2454] T a4983426b2d843e7a9ed335c79178678 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4127249ba42644388d225d614c935a25 in term 1.
I20240503 01:59:35.447940 6301 leader_election.cc:302] T a4983426b2d843e7a9ed335c79178678 P 4127249ba42644388d225d614c935a25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c4821d9898d42799c5789d4d2da3f63, 4127249ba42644388d225d614c935a25; no voters:
I20240503 01:59:35.448114 7877 raft_consensus.cc:2781] T a4983426b2d843e7a9ed335c79178678 P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:35.448220 7877 raft_consensus.cc:686] T a4983426b2d843e7a9ed335c79178678 P 4127249ba42644388d225d614c935a25 [term 1 LEADER]: Becoming Leader. State: Replica: 4127249ba42644388d225d614c935a25, State: Running, Role: LEADER
I20240503 01:59:35.448338 7877 consensus_queue.cc:237] T a4983426b2d843e7a9ed335c79178678 P 4127249ba42644388d225d614c935a25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } }
I20240503 01:59:35.449976 6222 catalog_manager.cc:5484] T a4983426b2d843e7a9ed335c79178678 P 4127249ba42644388d225d614c935a25 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4127249ba42644388d225d614c935a25 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "4127249ba42644388d225d614c935a25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } health_report { overall_health: HEALTHY } } }
I20240503 01:59:35.521137 7894 raft_consensus.cc:482] T 0bdcea4e59ec4e2da63b9316f9d718be P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:35.521286 7894 raft_consensus.cc:504] T 0bdcea4e59ec4e2da63b9316f9d718be P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } }
I20240503 01:59:35.521855 7894 leader_election.cc:288] T 0bdcea4e59ec4e2da63b9316f9d718be P 79163c69f6f04a9abd0208d74e1c61be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4127249ba42644388d225d614c935a25 (127.0.99.129:35613), b6b1d5677b8d47788b7968f64b3dac3c (127.0.99.132:46067)
I20240503 01:59:35.522071 6365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bdcea4e59ec4e2da63b9316f9d718be" candidate_uuid: "79163c69f6f04a9abd0208d74e1c61be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4127249ba42644388d225d614c935a25" is_pre_election: true
I20240503 01:59:35.522213 6365 raft_consensus.cc:2454] T 0bdcea4e59ec4e2da63b9316f9d718be P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79163c69f6f04a9abd0208d74e1c61be in term 0.
I20240503 01:59:35.522229 6756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bdcea4e59ec4e2da63b9316f9d718be" candidate_uuid: "79163c69f6f04a9abd0208d74e1c61be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" is_pre_election: true
I20240503 01:59:35.522372 6756 raft_consensus.cc:2454] T 0bdcea4e59ec4e2da63b9316f9d718be P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79163c69f6f04a9abd0208d74e1c61be in term 0.
I20240503 01:59:35.522500 6561 leader_election.cc:302] T 0bdcea4e59ec4e2da63b9316f9d718be P 79163c69f6f04a9abd0208d74e1c61be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4127249ba42644388d225d614c935a25, 79163c69f6f04a9abd0208d74e1c61be; no voters:
I20240503 01:59:35.522665 7894 raft_consensus.cc:2781] T 0bdcea4e59ec4e2da63b9316f9d718be P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:35.522732 7894 raft_consensus.cc:482] T 0bdcea4e59ec4e2da63b9316f9d718be P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:35.522783 7894 raft_consensus.cc:3037] T 0bdcea4e59ec4e2da63b9316f9d718be P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:35.523594 7894 raft_consensus.cc:504] T 0bdcea4e59ec4e2da63b9316f9d718be P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } }
I20240503 01:59:35.523871 7894 leader_election.cc:288] T 0bdcea4e59ec4e2da63b9316f9d718be P 79163c69f6f04a9abd0208d74e1c61be [CANDIDATE]: Term 1 election: Requested vote from peers 4127249ba42644388d225d614c935a25 (127.0.99.129:35613), b6b1d5677b8d47788b7968f64b3dac3c (127.0.99.132:46067)
I20240503 01:59:35.524200 6365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bdcea4e59ec4e2da63b9316f9d718be" candidate_uuid: "79163c69f6f04a9abd0208d74e1c61be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4127249ba42644388d225d614c935a25"
I20240503 01:59:35.524200 6756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bdcea4e59ec4e2da63b9316f9d718be" candidate_uuid: "79163c69f6f04a9abd0208d74e1c61be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6b1d5677b8d47788b7968f64b3dac3c"
I20240503 01:59:35.524318 6756 raft_consensus.cc:3037] T 0bdcea4e59ec4e2da63b9316f9d718be P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:35.524318 6365 raft_consensus.cc:3037] T 0bdcea4e59ec4e2da63b9316f9d718be P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:35.525254 6365 raft_consensus.cc:2454] T 0bdcea4e59ec4e2da63b9316f9d718be P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79163c69f6f04a9abd0208d74e1c61be in term 1.
I20240503 01:59:35.525254 6756 raft_consensus.cc:2454] T 0bdcea4e59ec4e2da63b9316f9d718be P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79163c69f6f04a9abd0208d74e1c61be in term 1.
I20240503 01:59:35.525591 6562 leader_election.cc:302] T 0bdcea4e59ec4e2da63b9316f9d718be P 79163c69f6f04a9abd0208d74e1c61be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79163c69f6f04a9abd0208d74e1c61be, b6b1d5677b8d47788b7968f64b3dac3c; no voters:
I20240503 01:59:35.525765 7894 raft_consensus.cc:2781] T 0bdcea4e59ec4e2da63b9316f9d718be P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:35.525861 7894 raft_consensus.cc:686] T 0bdcea4e59ec4e2da63b9316f9d718be P 79163c69f6f04a9abd0208d74e1c61be [term 1 LEADER]: Becoming Leader. State: Replica: 79163c69f6f04a9abd0208d74e1c61be, State: Running, Role: LEADER
I20240503 01:59:35.526018 7894 consensus_queue.cc:237] T 0bdcea4e59ec4e2da63b9316f9d718be P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } }
I20240503 01:59:35.527828 6222 catalog_manager.cc:5484] T 0bdcea4e59ec4e2da63b9316f9d718be P 79163c69f6f04a9abd0208d74e1c61be reported cstate change: term changed from 0 to 1, leader changed from <none> to 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131). New cstate: current_term: 1 leader_uuid: "79163c69f6f04a9abd0208d74e1c61be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } health_report { overall_health: UNKNOWN } } }
W20240503 01:59:35.647792 7922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:35.647994 7922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:35.650945 7922 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_8'
I20240503 01:59:35.738201 6625 raft_consensus.cc:1261] T ccb34a4b52244d90850ab04e21299f55 P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Refusing update from remote peer 0c4821d9898d42799c5789d4d2da3f63: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:35.738576 6756 raft_consensus.cc:1261] T 0bdcea4e59ec4e2da63b9316f9d718be P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:35.739096 7361 consensus_queue.cc:1035] T ccb34a4b52244d90850ab04e21299f55 P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:35.739259 6365 raft_consensus.cc:1261] T ccb34a4b52244d90850ab04e21299f55 P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Refusing update from remote peer 0c4821d9898d42799c5789d4d2da3f63: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:35.739704 7361 consensus_queue.cc:1035] T ccb34a4b52244d90850ab04e21299f55 P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:35.742977 6365 raft_consensus.cc:1261] T 0bdcea4e59ec4e2da63b9316f9d718be P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:35.742981 7894 consensus_queue.cc:1035] T 0bdcea4e59ec4e2da63b9316f9d718be P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:35.744235 7894 consensus_queue.cc:1035] T 0bdcea4e59ec4e2da63b9316f9d718be P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:35.744727 6757 raft_consensus.cc:1261] T a4983426b2d843e7a9ed335c79178678 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Refusing update from remote peer 4127249ba42644388d225d614c935a25: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:35.745263 6494 raft_consensus.cc:1261] T a4983426b2d843e7a9ed335c79178678 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Refusing update from remote peer 4127249ba42644388d225d614c935a25: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:35.749503 7877 consensus_queue.cc:1035] T a4983426b2d843e7a9ed335c79178678 P 4127249ba42644388d225d614c935a25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:35.753659 7825 consensus_queue.cc:1035] T a4983426b2d843e7a9ed335c79178678 P 4127249ba42644388d225d614c935a25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 119.948 ms
time per row: 0.0599739 ms
I20240503 01:59:35.812757 6223 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44084:
name: "rebalance_test_table_9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:59:35.818840 6737 tablet_service.cc:1467] Processing CreateTablet for tablet 897e88ac93484a11829189e027de8bae (DEFAULT_TABLE table=rebalance_test_table_9 [id=27f301d87e884061bddb2605833c29a6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:35.818979 6607 tablet_service.cc:1467] Processing CreateTablet for tablet 897e88ac93484a11829189e027de8bae (DEFAULT_TABLE table=rebalance_test_table_9 [id=27f301d87e884061bddb2605833c29a6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:35.819135 6737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 897e88ac93484a11829189e027de8bae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:35.819267 6607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 897e88ac93484a11829189e027de8bae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:35.821717 7907 tablet_bootstrap.cc:492] T 897e88ac93484a11829189e027de8bae P 79163c69f6f04a9abd0208d74e1c61be: Bootstrap starting.
I20240503 01:59:35.822093 6476 tablet_service.cc:1467] Processing CreateTablet for tablet 897e88ac93484a11829189e027de8bae (DEFAULT_TABLE table=rebalance_test_table_9 [id=27f301d87e884061bddb2605833c29a6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:35.822386 6476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 897e88ac93484a11829189e027de8bae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:35.822520 7907 tablet_bootstrap.cc:654] T 897e88ac93484a11829189e027de8bae P 79163c69f6f04a9abd0208d74e1c61be: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:35.823496 7907 tablet_bootstrap.cc:492] T 897e88ac93484a11829189e027de8bae P 79163c69f6f04a9abd0208d74e1c61be: No bootstrap required, opened a new log
I20240503 01:59:35.823556 7907 ts_tablet_manager.cc:1367] T 897e88ac93484a11829189e027de8bae P 79163c69f6f04a9abd0208d74e1c61be: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:35.824012 7907 raft_consensus.cc:348] T 897e88ac93484a11829189e027de8bae P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } }
I20240503 01:59:35.824136 7907 raft_consensus.cc:374] T 897e88ac93484a11829189e027de8bae P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:35.824173 7907 raft_consensus.cc:729] T 897e88ac93484a11829189e027de8bae P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79163c69f6f04a9abd0208d74e1c61be, State: Initialized, Role: FOLLOWER
I20240503 01:59:35.824312 7907 consensus_queue.cc:260] T 897e88ac93484a11829189e027de8bae P 79163c69f6f04a9abd0208d74e1c61be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } }
I20240503 01:59:35.824457 7908 tablet_bootstrap.cc:492] T 897e88ac93484a11829189e027de8bae P 0c4821d9898d42799c5789d4d2da3f63: Bootstrap starting.
I20240503 01:59:35.824678 7907 ts_tablet_manager.cc:1398] T 897e88ac93484a11829189e027de8bae P 79163c69f6f04a9abd0208d74e1c61be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:35.825322 7908 tablet_bootstrap.cc:654] T 897e88ac93484a11829189e027de8bae P 0c4821d9898d42799c5789d4d2da3f63: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:35.827610 7908 tablet_bootstrap.cc:492] T 897e88ac93484a11829189e027de8bae P 0c4821d9898d42799c5789d4d2da3f63: No bootstrap required, opened a new log
I20240503 01:59:35.827690 7908 ts_tablet_manager.cc:1367] T 897e88ac93484a11829189e027de8bae P 0c4821d9898d42799c5789d4d2da3f63: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 01:59:35.828063 7908 raft_consensus.cc:348] T 897e88ac93484a11829189e027de8bae P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } }
I20240503 01:59:35.828163 7908 raft_consensus.cc:374] T 897e88ac93484a11829189e027de8bae P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:35.828189 7908 raft_consensus.cc:729] T 897e88ac93484a11829189e027de8bae P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c4821d9898d42799c5789d4d2da3f63, State: Initialized, Role: FOLLOWER
I20240503 01:59:35.828284 7908 consensus_queue.cc:260] T 897e88ac93484a11829189e027de8bae P 0c4821d9898d42799c5789d4d2da3f63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } }
I20240503 01:59:35.828506 7908 ts_tablet_manager.cc:1398] T 897e88ac93484a11829189e027de8bae P 0c4821d9898d42799c5789d4d2da3f63: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:35.830142 6738 tablet_service.cc:1467] Processing CreateTablet for tablet f11a8034585a4d57be0e9b1e5e8bcd85 (DEFAULT_TABLE table=rebalance_test_table_9 [id=27f301d87e884061bddb2605833c29a6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:35.830195 6474 tablet_service.cc:1467] Processing CreateTablet for tablet 67150a2f5cb147b8b57a8c8040b567e1 (DEFAULT_TABLE table=rebalance_test_table_9 [id=27f301d87e884061bddb2605833c29a6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:35.830268 7906 tablet_bootstrap.cc:492] T 897e88ac93484a11829189e027de8bae P b6b1d5677b8d47788b7968f64b3dac3c: Bootstrap starting.
I20240503 01:59:35.830401 6738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f11a8034585a4d57be0e9b1e5e8bcd85. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:35.830463 6474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67150a2f5cb147b8b57a8c8040b567e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:35.831094 7906 tablet_bootstrap.cc:654] T 897e88ac93484a11829189e027de8bae P b6b1d5677b8d47788b7968f64b3dac3c: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:35.832140 7906 tablet_bootstrap.cc:492] T 897e88ac93484a11829189e027de8bae P b6b1d5677b8d47788b7968f64b3dac3c: No bootstrap required, opened a new log
I20240503 01:59:35.832219 7906 ts_tablet_manager.cc:1367] T 897e88ac93484a11829189e027de8bae P b6b1d5677b8d47788b7968f64b3dac3c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:35.832629 7906 raft_consensus.cc:348] T 897e88ac93484a11829189e027de8bae P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } }
I20240503 01:59:35.832792 7906 raft_consensus.cc:374] T 897e88ac93484a11829189e027de8bae P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:35.832743 6344 tablet_service.cc:1467] Processing CreateTablet for tablet 67150a2f5cb147b8b57a8c8040b567e1 (DEFAULT_TABLE table=rebalance_test_table_9 [id=27f301d87e884061bddb2605833c29a6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:35.832823 7906 raft_consensus.cc:729] T 897e88ac93484a11829189e027de8bae P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6b1d5677b8d47788b7968f64b3dac3c, State: Initialized, Role: FOLLOWER
I20240503 01:59:35.832942 7906 consensus_queue.cc:260] T 897e88ac93484a11829189e027de8bae P b6b1d5677b8d47788b7968f64b3dac3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } }
I20240503 01:59:35.833024 6344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67150a2f5cb147b8b57a8c8040b567e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:35.833209 7906 ts_tablet_manager.cc:1398] T 897e88ac93484a11829189e027de8bae P b6b1d5677b8d47788b7968f64b3dac3c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:35.833395 7906 tablet_bootstrap.cc:492] T f11a8034585a4d57be0e9b1e5e8bcd85 P b6b1d5677b8d47788b7968f64b3dac3c: Bootstrap starting.
I20240503 01:59:35.834640 7904 tablet_bootstrap.cc:492] T 67150a2f5cb147b8b57a8c8040b567e1 P 4127249ba42644388d225d614c935a25: Bootstrap starting.
I20240503 01:59:35.835337 7906 tablet_bootstrap.cc:654] T f11a8034585a4d57be0e9b1e5e8bcd85 P b6b1d5677b8d47788b7968f64b3dac3c: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:35.835484 7904 tablet_bootstrap.cc:654] T 67150a2f5cb147b8b57a8c8040b567e1 P 4127249ba42644388d225d614c935a25: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:35.835922 6476 tablet_service.cc:1467] Processing CreateTablet for tablet f11a8034585a4d57be0e9b1e5e8bcd85 (DEFAULT_TABLE table=rebalance_test_table_9 [id=27f301d87e884061bddb2605833c29a6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:35.836165 6476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f11a8034585a4d57be0e9b1e5e8bcd85. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:35.836187 6606 tablet_service.cc:1467] Processing CreateTablet for tablet 67150a2f5cb147b8b57a8c8040b567e1 (DEFAULT_TABLE table=rebalance_test_table_9 [id=27f301d87e884061bddb2605833c29a6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:35.836462 6606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67150a2f5cb147b8b57a8c8040b567e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:35.838220 7908 tablet_bootstrap.cc:492] T 67150a2f5cb147b8b57a8c8040b567e1 P 0c4821d9898d42799c5789d4d2da3f63: Bootstrap starting.
I20240503 01:59:35.839076 7907 tablet_bootstrap.cc:492] T 67150a2f5cb147b8b57a8c8040b567e1 P 79163c69f6f04a9abd0208d74e1c61be: Bootstrap starting.
I20240503 01:59:35.839164 7908 tablet_bootstrap.cc:654] T 67150a2f5cb147b8b57a8c8040b567e1 P 0c4821d9898d42799c5789d4d2da3f63: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:35.839848 7907 tablet_bootstrap.cc:654] T 67150a2f5cb147b8b57a8c8040b567e1 P 79163c69f6f04a9abd0208d74e1c61be: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:35.839848 7904 tablet_bootstrap.cc:492] T 67150a2f5cb147b8b57a8c8040b567e1 P 4127249ba42644388d225d614c935a25: No bootstrap required, opened a new log
I20240503 01:59:35.839931 7904 ts_tablet_manager.cc:1367] T 67150a2f5cb147b8b57a8c8040b567e1 P 4127249ba42644388d225d614c935a25: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20240503 01:59:35.839965 6607 tablet_service.cc:1467] Processing CreateTablet for tablet f11a8034585a4d57be0e9b1e5e8bcd85 (DEFAULT_TABLE table=rebalance_test_table_9 [id=27f301d87e884061bddb2605833c29a6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:35.840176 6607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f11a8034585a4d57be0e9b1e5e8bcd85. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:35.840324 7904 raft_consensus.cc:348] T 67150a2f5cb147b8b57a8c8040b567e1 P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } }
I20240503 01:59:35.840448 7904 raft_consensus.cc:374] T 67150a2f5cb147b8b57a8c8040b567e1 P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:35.840482 7904 raft_consensus.cc:729] T 67150a2f5cb147b8b57a8c8040b567e1 P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4127249ba42644388d225d614c935a25, State: Initialized, Role: FOLLOWER
I20240503 01:59:35.840591 7904 consensus_queue.cc:260] T 67150a2f5cb147b8b57a8c8040b567e1 P 4127249ba42644388d225d614c935a25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } }
I20240503 01:59:35.840865 7904 ts_tablet_manager.cc:1398] T 67150a2f5cb147b8b57a8c8040b567e1 P 4127249ba42644388d225d614c935a25: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:35.841779 7908 tablet_bootstrap.cc:492] T 67150a2f5cb147b8b57a8c8040b567e1 P 0c4821d9898d42799c5789d4d2da3f63: No bootstrap required, opened a new log
I20240503 01:59:35.841862 7908 ts_tablet_manager.cc:1367] T 67150a2f5cb147b8b57a8c8040b567e1 P 0c4821d9898d42799c5789d4d2da3f63: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 01:59:35.842257 7908 raft_consensus.cc:348] T 67150a2f5cb147b8b57a8c8040b567e1 P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } }
I20240503 01:59:35.842382 7908 raft_consensus.cc:374] T 67150a2f5cb147b8b57a8c8040b567e1 P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:35.842427 7908 raft_consensus.cc:729] T 67150a2f5cb147b8b57a8c8040b567e1 P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c4821d9898d42799c5789d4d2da3f63, State: Initialized, Role: FOLLOWER
I20240503 01:59:35.842569 7908 consensus_queue.cc:260] T 67150a2f5cb147b8b57a8c8040b567e1 P 0c4821d9898d42799c5789d4d2da3f63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } }
I20240503 01:59:35.842728 7907 tablet_bootstrap.cc:492] T 67150a2f5cb147b8b57a8c8040b567e1 P 79163c69f6f04a9abd0208d74e1c61be: No bootstrap required, opened a new log
I20240503 01:59:35.842808 7907 ts_tablet_manager.cc:1367] T 67150a2f5cb147b8b57a8c8040b567e1 P 79163c69f6f04a9abd0208d74e1c61be: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 01:59:35.842953 7906 tablet_bootstrap.cc:492] T f11a8034585a4d57be0e9b1e5e8bcd85 P b6b1d5677b8d47788b7968f64b3dac3c: No bootstrap required, opened a new log
I20240503 01:59:35.843030 7906 ts_tablet_manager.cc:1367] T f11a8034585a4d57be0e9b1e5e8bcd85 P b6b1d5677b8d47788b7968f64b3dac3c: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.002s
I20240503 01:59:35.843255 7907 raft_consensus.cc:348] T 67150a2f5cb147b8b57a8c8040b567e1 P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } }
I20240503 01:59:35.843344 7907 raft_consensus.cc:374] T 67150a2f5cb147b8b57a8c8040b567e1 P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:35.843379 7907 raft_consensus.cc:729] T 67150a2f5cb147b8b57a8c8040b567e1 P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79163c69f6f04a9abd0208d74e1c61be, State: Initialized, Role: FOLLOWER
I20240503 01:59:35.843463 7906 raft_consensus.cc:348] T f11a8034585a4d57be0e9b1e5e8bcd85 P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } }
I20240503 01:59:35.843497 7907 consensus_queue.cc:260] T 67150a2f5cb147b8b57a8c8040b567e1 P 79163c69f6f04a9abd0208d74e1c61be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } }
I20240503 01:59:35.843547 7906 raft_consensus.cc:374] T f11a8034585a4d57be0e9b1e5e8bcd85 P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:35.843578 7906 raft_consensus.cc:729] T f11a8034585a4d57be0e9b1e5e8bcd85 P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6b1d5677b8d47788b7968f64b3dac3c, State: Initialized, Role: FOLLOWER
I20240503 01:59:35.843729 7906 consensus_queue.cc:260] T f11a8034585a4d57be0e9b1e5e8bcd85 P b6b1d5677b8d47788b7968f64b3dac3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } }
I20240503 01:59:35.843798 7907 ts_tablet_manager.cc:1398] T 67150a2f5cb147b8b57a8c8040b567e1 P 79163c69f6f04a9abd0208d74e1c61be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:35.843957 7907 tablet_bootstrap.cc:492] T f11a8034585a4d57be0e9b1e5e8bcd85 P 79163c69f6f04a9abd0208d74e1c61be: Bootstrap starting.
I20240503 01:59:35.843972 7908 ts_tablet_manager.cc:1398] T 67150a2f5cb147b8b57a8c8040b567e1 P 0c4821d9898d42799c5789d4d2da3f63: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:35.844125 7908 tablet_bootstrap.cc:492] T f11a8034585a4d57be0e9b1e5e8bcd85 P 0c4821d9898d42799c5789d4d2da3f63: Bootstrap starting.
I20240503 01:59:35.844547 7906 ts_tablet_manager.cc:1398] T f11a8034585a4d57be0e9b1e5e8bcd85 P b6b1d5677b8d47788b7968f64b3dac3c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:35.845880 7908 tablet_bootstrap.cc:654] T f11a8034585a4d57be0e9b1e5e8bcd85 P 0c4821d9898d42799c5789d4d2da3f63: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:35.846146 7907 tablet_bootstrap.cc:654] T f11a8034585a4d57be0e9b1e5e8bcd85 P 79163c69f6f04a9abd0208d74e1c61be: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:35.847409 7907 tablet_bootstrap.cc:492] T f11a8034585a4d57be0e9b1e5e8bcd85 P 79163c69f6f04a9abd0208d74e1c61be: No bootstrap required, opened a new log
I20240503 01:59:35.847487 7907 ts_tablet_manager.cc:1367] T f11a8034585a4d57be0e9b1e5e8bcd85 P 79163c69f6f04a9abd0208d74e1c61be: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 01:59:35.848109 7907 raft_consensus.cc:348] T f11a8034585a4d57be0e9b1e5e8bcd85 P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } }
I20240503 01:59:35.848191 7907 raft_consensus.cc:374] T f11a8034585a4d57be0e9b1e5e8bcd85 P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:35.848223 7907 raft_consensus.cc:729] T f11a8034585a4d57be0e9b1e5e8bcd85 P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79163c69f6f04a9abd0208d74e1c61be, State: Initialized, Role: FOLLOWER
I20240503 01:59:35.848335 7907 consensus_queue.cc:260] T f11a8034585a4d57be0e9b1e5e8bcd85 P 79163c69f6f04a9abd0208d74e1c61be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } }
I20240503 01:59:35.848590 7907 ts_tablet_manager.cc:1398] T f11a8034585a4d57be0e9b1e5e8bcd85 P 79163c69f6f04a9abd0208d74e1c61be: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:35.849277 7908 tablet_bootstrap.cc:492] T f11a8034585a4d57be0e9b1e5e8bcd85 P 0c4821d9898d42799c5789d4d2da3f63: No bootstrap required, opened a new log
I20240503 01:59:35.849366 7908 ts_tablet_manager.cc:1367] T f11a8034585a4d57be0e9b1e5e8bcd85 P 0c4821d9898d42799c5789d4d2da3f63: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240503 01:59:35.849864 7908 raft_consensus.cc:348] T f11a8034585a4d57be0e9b1e5e8bcd85 P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } }
I20240503 01:59:35.849965 7908 raft_consensus.cc:374] T f11a8034585a4d57be0e9b1e5e8bcd85 P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:35.850394 7908 raft_consensus.cc:729] T f11a8034585a4d57be0e9b1e5e8bcd85 P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c4821d9898d42799c5789d4d2da3f63, State: Initialized, Role: FOLLOWER
I20240503 01:59:35.850557 7908 consensus_queue.cc:260] T f11a8034585a4d57be0e9b1e5e8bcd85 P 0c4821d9898d42799c5789d4d2da3f63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } }
I20240503 01:59:35.850896 7908 ts_tablet_manager.cc:1398] T f11a8034585a4d57be0e9b1e5e8bcd85 P 0c4821d9898d42799c5789d4d2da3f63: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:35.898200 7877 raft_consensus.cc:482] T 67150a2f5cb147b8b57a8c8040b567e1 P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:35.898324 7877 raft_consensus.cc:504] T 67150a2f5cb147b8b57a8c8040b567e1 P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } }
I20240503 01:59:35.898720 7877 leader_election.cc:288] T 67150a2f5cb147b8b57a8c8040b567e1 P 4127249ba42644388d225d614c935a25 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130:34189), 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131:44005)
I20240503 01:59:35.899026 6625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67150a2f5cb147b8b57a8c8040b567e1" candidate_uuid: "4127249ba42644388d225d614c935a25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79163c69f6f04a9abd0208d74e1c61be" is_pre_election: true
I20240503 01:59:35.899056 6494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67150a2f5cb147b8b57a8c8040b567e1" candidate_uuid: "4127249ba42644388d225d614c935a25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c4821d9898d42799c5789d4d2da3f63" is_pre_election: true
I20240503 01:59:35.899152 6625 raft_consensus.cc:2454] T 67150a2f5cb147b8b57a8c8040b567e1 P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4127249ba42644388d225d614c935a25 in term 0.
I20240503 01:59:35.899185 6494 raft_consensus.cc:2454] T 67150a2f5cb147b8b57a8c8040b567e1 P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4127249ba42644388d225d614c935a25 in term 0.
I20240503 01:59:35.899415 6299 leader_election.cc:302] T 67150a2f5cb147b8b57a8c8040b567e1 P 4127249ba42644388d225d614c935a25 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4127249ba42644388d225d614c935a25, 79163c69f6f04a9abd0208d74e1c61be; no voters:
I20240503 01:59:35.899561 7877 raft_consensus.cc:2781] T 67150a2f5cb147b8b57a8c8040b567e1 P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:35.899626 7877 raft_consensus.cc:482] T 67150a2f5cb147b8b57a8c8040b567e1 P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:35.899716 7877 raft_consensus.cc:3037] T 67150a2f5cb147b8b57a8c8040b567e1 P 4127249ba42644388d225d614c935a25 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:35.900512 7877 raft_consensus.cc:504] T 67150a2f5cb147b8b57a8c8040b567e1 P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } }
I20240503 01:59:35.900867 7877 leader_election.cc:288] T 67150a2f5cb147b8b57a8c8040b567e1 P 4127249ba42644388d225d614c935a25 [CANDIDATE]: Term 1 election: Requested vote from peers 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130:34189), 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131:44005)
I20240503 01:59:35.901144 6494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67150a2f5cb147b8b57a8c8040b567e1" candidate_uuid: "4127249ba42644388d225d614c935a25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c4821d9898d42799c5789d4d2da3f63"
I20240503 01:59:35.901226 6625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67150a2f5cb147b8b57a8c8040b567e1" candidate_uuid: "4127249ba42644388d225d614c935a25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79163c69f6f04a9abd0208d74e1c61be"
I20240503 01:59:35.901265 6494 raft_consensus.cc:3037] T 67150a2f5cb147b8b57a8c8040b567e1 P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:35.901343 6625 raft_consensus.cc:3037] T 67150a2f5cb147b8b57a8c8040b567e1 P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:35.902249 6494 raft_consensus.cc:2454] T 67150a2f5cb147b8b57a8c8040b567e1 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4127249ba42644388d225d614c935a25 in term 1.
I20240503 01:59:35.902256 6625 raft_consensus.cc:2454] T 67150a2f5cb147b8b57a8c8040b567e1 P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4127249ba42644388d225d614c935a25 in term 1.
I20240503 01:59:35.902552 6299 leader_election.cc:302] T 67150a2f5cb147b8b57a8c8040b567e1 P 4127249ba42644388d225d614c935a25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4127249ba42644388d225d614c935a25, 79163c69f6f04a9abd0208d74e1c61be; no voters:
I20240503 01:59:35.902707 7877 raft_consensus.cc:2781] T 67150a2f5cb147b8b57a8c8040b567e1 P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:35.902806 7877 raft_consensus.cc:686] T 67150a2f5cb147b8b57a8c8040b567e1 P 4127249ba42644388d225d614c935a25 [term 1 LEADER]: Becoming Leader. State: Replica: 4127249ba42644388d225d614c935a25, State: Running, Role: LEADER
I20240503 01:59:35.902983 7877 consensus_queue.cc:237] T 67150a2f5cb147b8b57a8c8040b567e1 P 4127249ba42644388d225d614c935a25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } }
I20240503 01:59:35.903707 7361 raft_consensus.cc:482] T 897e88ac93484a11829189e027de8bae P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:35.903790 7361 raft_consensus.cc:504] T 897e88ac93484a11829189e027de8bae P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } }
I20240503 01:59:35.904098 7361 leader_election.cc:288] T 897e88ac93484a11829189e027de8bae P 0c4821d9898d42799c5789d4d2da3f63 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131:44005), b6b1d5677b8d47788b7968f64b3dac3c (127.0.99.132:46067)
I20240503 01:59:35.904402 6625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "897e88ac93484a11829189e027de8bae" candidate_uuid: "0c4821d9898d42799c5789d4d2da3f63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79163c69f6f04a9abd0208d74e1c61be" is_pre_election: true
I20240503 01:59:35.904456 6757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "897e88ac93484a11829189e027de8bae" candidate_uuid: "0c4821d9898d42799c5789d4d2da3f63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" is_pre_election: true
I20240503 01:59:35.904537 6625 raft_consensus.cc:2454] T 897e88ac93484a11829189e027de8bae P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c4821d9898d42799c5789d4d2da3f63 in term 0.
I20240503 01:59:35.904574 6757 raft_consensus.cc:2454] T 897e88ac93484a11829189e027de8bae P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c4821d9898d42799c5789d4d2da3f63 in term 0.
I20240503 01:59:35.904884 6430 leader_election.cc:302] T 897e88ac93484a11829189e027de8bae P 0c4821d9898d42799c5789d4d2da3f63 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c4821d9898d42799c5789d4d2da3f63, 79163c69f6f04a9abd0208d74e1c61be; no voters:
I20240503 01:59:35.905055 7361 raft_consensus.cc:2781] T 897e88ac93484a11829189e027de8bae P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:35.904954 6223 catalog_manager.cc:5484] T 67150a2f5cb147b8b57a8c8040b567e1 P 4127249ba42644388d225d614c935a25 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4127249ba42644388d225d614c935a25 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "4127249ba42644388d225d614c935a25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:35.905112 7361 raft_consensus.cc:482] T 897e88ac93484a11829189e027de8bae P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:35.905158 7361 raft_consensus.cc:3037] T 897e88ac93484a11829189e027de8bae P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:35.905932 7361 raft_consensus.cc:504] T 897e88ac93484a11829189e027de8bae P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } }
I20240503 01:59:35.906287 7361 leader_election.cc:288] T 897e88ac93484a11829189e027de8bae P 0c4821d9898d42799c5789d4d2da3f63 [CANDIDATE]: Term 1 election: Requested vote from peers 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131:44005), b6b1d5677b8d47788b7968f64b3dac3c (127.0.99.132:46067)
I20240503 01:59:35.906534 6625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "897e88ac93484a11829189e027de8bae" candidate_uuid: "0c4821d9898d42799c5789d4d2da3f63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79163c69f6f04a9abd0208d74e1c61be"
I20240503 01:59:35.906641 6625 raft_consensus.cc:3037] T 897e88ac93484a11829189e027de8bae P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:35.907377 6757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "897e88ac93484a11829189e027de8bae" candidate_uuid: "0c4821d9898d42799c5789d4d2da3f63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6b1d5677b8d47788b7968f64b3dac3c"
I20240503 01:59:35.907485 6757 raft_consensus.cc:3037] T 897e88ac93484a11829189e027de8bae P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:35.907548 6625 raft_consensus.cc:2454] T 897e88ac93484a11829189e027de8bae P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c4821d9898d42799c5789d4d2da3f63 in term 1.
I20240503 01:59:35.907843 6430 leader_election.cc:302] T 897e88ac93484a11829189e027de8bae P 0c4821d9898d42799c5789d4d2da3f63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c4821d9898d42799c5789d4d2da3f63, 79163c69f6f04a9abd0208d74e1c61be; no voters:
I20240503 01:59:35.908044 7361 raft_consensus.cc:2781] T 897e88ac93484a11829189e027de8bae P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:35.908135 7361 raft_consensus.cc:686] T 897e88ac93484a11829189e027de8bae P 0c4821d9898d42799c5789d4d2da3f63 [term 1 LEADER]: Becoming Leader. State: Replica: 0c4821d9898d42799c5789d4d2da3f63, State: Running, Role: LEADER
I20240503 01:59:35.908259 7361 consensus_queue.cc:237] T 897e88ac93484a11829189e027de8bae P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } }
I20240503 01:59:35.908524 6757 raft_consensus.cc:2454] T 897e88ac93484a11829189e027de8bae P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c4821d9898d42799c5789d4d2da3f63 in term 1.
I20240503 01:59:35.909785 6223 catalog_manager.cc:5484] T 897e88ac93484a11829189e027de8bae P 0c4821d9898d42799c5789d4d2da3f63 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "0c4821d9898d42799c5789d4d2da3f63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:35.977629 7957 raft_consensus.cc:482] T f11a8034585a4d57be0e9b1e5e8bcd85 P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:35.977770 7957 raft_consensus.cc:504] T f11a8034585a4d57be0e9b1e5e8bcd85 P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } }
I20240503 01:59:35.978222 7957 leader_election.cc:288] T f11a8034585a4d57be0e9b1e5e8bcd85 P b6b1d5677b8d47788b7968f64b3dac3c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130:34189), 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131:44005)
I20240503 01:59:35.978657 6494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f11a8034585a4d57be0e9b1e5e8bcd85" candidate_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c4821d9898d42799c5789d4d2da3f63" is_pre_election: true
I20240503 01:59:35.978657 6625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f11a8034585a4d57be0e9b1e5e8bcd85" candidate_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79163c69f6f04a9abd0208d74e1c61be" is_pre_election: true
I20240503 01:59:35.978801 6494 raft_consensus.cc:2454] T f11a8034585a4d57be0e9b1e5e8bcd85 P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6b1d5677b8d47788b7968f64b3dac3c in term 0.
I20240503 01:59:35.978801 6625 raft_consensus.cc:2454] T f11a8034585a4d57be0e9b1e5e8bcd85 P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6b1d5677b8d47788b7968f64b3dac3c in term 0.
I20240503 01:59:35.979084 6692 leader_election.cc:302] T f11a8034585a4d57be0e9b1e5e8bcd85 P b6b1d5677b8d47788b7968f64b3dac3c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79163c69f6f04a9abd0208d74e1c61be, b6b1d5677b8d47788b7968f64b3dac3c; no voters:
I20240503 01:59:35.979238 7957 raft_consensus.cc:2781] T f11a8034585a4d57be0e9b1e5e8bcd85 P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:35.979297 7957 raft_consensus.cc:482] T f11a8034585a4d57be0e9b1e5e8bcd85 P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:35.979351 7957 raft_consensus.cc:3037] T f11a8034585a4d57be0e9b1e5e8bcd85 P b6b1d5677b8d47788b7968f64b3dac3c [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:35.980237 7957 raft_consensus.cc:504] T f11a8034585a4d57be0e9b1e5e8bcd85 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } }
I20240503 01:59:35.980551 7957 leader_election.cc:288] T f11a8034585a4d57be0e9b1e5e8bcd85 P b6b1d5677b8d47788b7968f64b3dac3c [CANDIDATE]: Term 1 election: Requested vote from peers 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130:34189), 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131:44005)
I20240503 01:59:35.980883 6494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f11a8034585a4d57be0e9b1e5e8bcd85" candidate_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c4821d9898d42799c5789d4d2da3f63"
I20240503 01:59:35.980911 6625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f11a8034585a4d57be0e9b1e5e8bcd85" candidate_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79163c69f6f04a9abd0208d74e1c61be"
I20240503 01:59:35.981010 6625 raft_consensus.cc:3037] T f11a8034585a4d57be0e9b1e5e8bcd85 P 79163c69f6f04a9abd0208d74e1c61be [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:35.981010 6494 raft_consensus.cc:3037] T f11a8034585a4d57be0e9b1e5e8bcd85 P 0c4821d9898d42799c5789d4d2da3f63 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:35.981943 6625 raft_consensus.cc:2454] T f11a8034585a4d57be0e9b1e5e8bcd85 P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6b1d5677b8d47788b7968f64b3dac3c in term 1.
I20240503 01:59:35.981949 6494 raft_consensus.cc:2454] T f11a8034585a4d57be0e9b1e5e8bcd85 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6b1d5677b8d47788b7968f64b3dac3c in term 1.
I20240503 01:59:35.982244 6692 leader_election.cc:302] T f11a8034585a4d57be0e9b1e5e8bcd85 P b6b1d5677b8d47788b7968f64b3dac3c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79163c69f6f04a9abd0208d74e1c61be, b6b1d5677b8d47788b7968f64b3dac3c; no voters:
I20240503 01:59:35.982386 7957 raft_consensus.cc:2781] T f11a8034585a4d57be0e9b1e5e8bcd85 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:35.982477 7957 raft_consensus.cc:686] T f11a8034585a4d57be0e9b1e5e8bcd85 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 LEADER]: Becoming Leader. State: Replica: b6b1d5677b8d47788b7968f64b3dac3c, State: Running, Role: LEADER
I20240503 01:59:35.982640 7957 consensus_queue.cc:237] T f11a8034585a4d57be0e9b1e5e8bcd85 P b6b1d5677b8d47788b7968f64b3dac3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } }
I20240503 01:59:35.984417 6223 catalog_manager.cc:5484] T f11a8034585a4d57be0e9b1e5e8bcd85 P b6b1d5677b8d47788b7968f64b3dac3c reported cstate change: term changed from 0 to 1, leader changed from <none> to b6b1d5677b8d47788b7968f64b3dac3c (127.0.99.132). New cstate: current_term: 1 leader_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } health_report { overall_health: HEALTHY } } }
W20240503 01:59:36.120213 7972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:36.120455 7972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:36.123801 7972 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_9'
I20240503 01:59:36.197189 6494 raft_consensus.cc:1261] T 67150a2f5cb147b8b57a8c8040b567e1 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Refusing update from remote peer 4127249ba42644388d225d614c935a25: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:36.197803 7877 consensus_queue.cc:1035] T 67150a2f5cb147b8b57a8c8040b567e1 P 4127249ba42644388d225d614c935a25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:36.198853 6625 raft_consensus.cc:1261] T 67150a2f5cb147b8b57a8c8040b567e1 P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Refusing update from remote peer 4127249ba42644388d225d614c935a25: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:36.200525 6757 raft_consensus.cc:1261] T 897e88ac93484a11829189e027de8bae P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Refusing update from remote peer 0c4821d9898d42799c5789d4d2da3f63: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:36.200947 7361 consensus_queue.cc:1035] T 897e88ac93484a11829189e027de8bae P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:36.203033 6627 raft_consensus.cc:1261] T f11a8034585a4d57be0e9b1e5e8bcd85 P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Refusing update from remote peer b6b1d5677b8d47788b7968f64b3dac3c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:36.203446 7448 consensus_queue.cc:1035] T f11a8034585a4d57be0e9b1e5e8bcd85 P b6b1d5677b8d47788b7968f64b3dac3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:36.204567 6495 raft_consensus.cc:1261] T f11a8034585a4d57be0e9b1e5e8bcd85 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Refusing update from remote peer b6b1d5677b8d47788b7968f64b3dac3c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:36.205036 7957 consensus_queue.cc:1035] T f11a8034585a4d57be0e9b1e5e8bcd85 P b6b1d5677b8d47788b7968f64b3dac3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:36.207700 7877 consensus_queue.cc:1035] T 67150a2f5cb147b8b57a8c8040b567e1 P 4127249ba42644388d225d614c935a25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:36.213599 6626 raft_consensus.cc:1261] T 897e88ac93484a11829189e027de8bae P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Refusing update from remote peer 0c4821d9898d42799c5789d4d2da3f63: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:36.213975 7918 consensus_queue.cc:1035] T 897e88ac93484a11829189e027de8bae P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:36.231395 7881 mvcc.cc:205] Tried to move back new op lower bound from 7023417656098590720 to 7023417654899527680. Current Snapshot: MvccSnapshot[applied={T|T < 7023417656098590720}]
INSERT report
rows total: 2000
time total: 101.743 ms
time per row: 0.0508717 ms
I20240503 01:59:36.267406 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task42rE7p/build/debug/bin/kudu
/tmp/dist-test-task42rE7p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:41713
--local_ip_for_outbound_sockets=127.0.99.133
--tserver_master_addrs=127.0.99.190:38597
--webserver_port=40469
--webserver_interface=127.0.99.133
--builtin_ntp_servers=127.0.99.148:34293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
W20240503 01:59:36.377259 8011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:36.377514 8011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:59:36.377581 8011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:36.380681 8011 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:59:36.380748 8011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:59:36.380815 8011 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240503 01:59:36.384683 8011 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:34293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:41713
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=40469
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.0.99.190:38597
--never_fsync=true
--heap_profile_path=/tmp/kudu.8011
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:59:36.385574 8011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:59:36.386665 8011 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:36.395712 8016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:36.396582 8019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:36.397486 8017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:36.398818 8011 server_base.cc:1034] running on GCE node
I20240503 01:59:36.399223 8011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:59:36.399734 8011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:59:36.400898 8011 hybrid_clock.cc:648] HybridClock initialized: now 1714701576400872 us; error 44 us; skew 500 ppm
I20240503 01:59:36.402855 8011 webserver.cc:460] Webserver started at http://127.0.99.133:40469/ using document root <none> and password file <none>
I20240503 01:59:36.403376 8011 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:36.403445 8011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:36.406826 8011 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240503 01:59:36.408773 8025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:36.409758 8011 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:36.409859 8011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "9264f24d028347d98f7e4e2d1aedeb79"
format_stamp: "Formatted at 2024-05-03 01:59:26 on dist-test-slave-dtz3"
I20240503 01:59:36.410248 8011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:36.437103 8011 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:36.437670 8011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:59:36.437798 8011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:36.438246 8011 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:36.439111 8011 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:59:36.439162 8011 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:36.439219 8011 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:59:36.439250 8011 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:36.448992 8011 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:41713
I20240503 01:59:36.449028 8138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:41713 every 8 connection(s)
I20240503 01:59:36.450234 8011 server_base.cc:1166] Dumped server information to /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 01:59:36.453356 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task42rE7p/build/debug/bin/kudu as pid 8011
I20240503 01:59:36.453733 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task42rE7p/build/debug/bin/kudu
/tmp/dist-test-task42rE7p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.134:35795
--local_ip_for_outbound_sockets=127.0.99.134
--tserver_master_addrs=127.0.99.190:38597
--webserver_port=43753
--webserver_interface=127.0.99.134
--builtin_ntp_servers=127.0.99.148:34293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240503 01:59:36.459307 8139 heartbeater.cc:347] Connected to a master server at 127.0.99.190:38597
I20240503 01:59:36.459496 8139 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:36.459947 8139 heartbeater.cc:511] Master 127.0.99.190:38597 requested a full tablet report, sending...
I20240503 01:59:36.461156 6223 ts_manager.cc:194] Re-registered known tserver with Master: 9264f24d028347d98f7e4e2d1aedeb79 (127.0.99.133:41713)
I20240503 01:59:36.461967 6223 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:37139
W20240503 01:59:36.561900 8142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:36.562090 8142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:59:36.562135 8142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:36.565742 8142 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:59:36.565829 8142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:59:36.565924 8142 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.134
I20240503 01:59:36.569433 8142 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:34293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.134:35795
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.0.99.134
--webserver_port=43753
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.0.99.190:38597
--never_fsync=true
--heap_profile_path=/tmp/kudu.8142
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.134
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:59:36.570233 8142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:59:36.571309 8142 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:36.580816 8148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:36.581471 8150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:36.582275 8147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:36.583642 8142 server_base.cc:1034] running on GCE node
I20240503 01:59:36.584014 8142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:59:36.584563 8142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:59:36.585718 8142 hybrid_clock.cc:648] HybridClock initialized: now 1714701576585701 us; error 28 us; skew 500 ppm
I20240503 01:59:36.592634 8142 webserver.cc:460] Webserver started at http://127.0.99.134:43753/ using document root <none> and password file <none>
I20240503 01:59:36.593223 8142 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:36.593318 8142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:36.597105 8142 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20240503 01:59:36.599062 8156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:36.600072 8142 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:36.600163 8142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "65d4632bcdd54ba596acdbbe1b5ef8af"
format_stamp: "Formatted at 2024-05-03 01:59:26 on dist-test-slave-dtz3"
I20240503 01:59:36.600528 8142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:36.613978 8142 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:36.614462 8142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:59:36.614629 8142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:36.615113 8142 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:36.615931 8142 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:59:36.615979 8142 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:36.616017 8142 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:59:36.616042 8142 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:36.623673 8142 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.134:35795
I20240503 01:59:36.623723 8269 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.134:35795 every 8 connection(s)
I20240503 01:59:36.624624 8142 server_base.cc:1166] Dumped server information to /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 01:59:36.629323 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task42rE7p/build/debug/bin/kudu as pid 8142
I20240503 01:59:36.629705 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task42rE7p/build/debug/bin/kudu
/tmp/dist-test-task42rE7p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.135:34035
--local_ip_for_outbound_sockets=127.0.99.135
--tserver_master_addrs=127.0.99.190:38597
--webserver_port=37065
--webserver_interface=127.0.99.135
--builtin_ntp_servers=127.0.99.148:34293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240503 01:59:36.632956 8270 heartbeater.cc:347] Connected to a master server at 127.0.99.190:38597
I20240503 01:59:36.633170 8270 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:36.633683 8270 heartbeater.cc:511] Master 127.0.99.190:38597 requested a full tablet report, sending...
I20240503 01:59:36.634688 6223 ts_manager.cc:194] Re-registered known tserver with Master: 65d4632bcdd54ba596acdbbe1b5ef8af (127.0.99.134:35795)
I20240503 01:59:36.635313 6223 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.134:57939
W20240503 01:59:36.738233 8273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:36.738427 8273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:59:36.738485 8273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:36.741818 8273 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:59:36.741895 8273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:59:36.742002 8273 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.135
I20240503 01:59:36.745644 8273 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:34293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.135:34035
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.0.99.135
--webserver_port=37065
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.0.99.190:38597
--never_fsync=true
--heap_profile_path=/tmp/kudu.8273
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.135
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:59:36.746438 8273 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:59:36.747680 8273 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:36.756345 8278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:36.757218 8281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:36.758425 8279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:36.759486 8273 server_base.cc:1034] running on GCE node
I20240503 01:59:36.759933 8273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:59:36.760500 8273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:59:36.761637 8273 hybrid_clock.cc:648] HybridClock initialized: now 1714701576761606 us; error 44 us; skew 500 ppm
I20240503 01:59:36.763442 8273 webserver.cc:460] Webserver started at http://127.0.99.135:37065/ using document root <none> and password file <none>
I20240503 01:59:36.763959 8273 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:36.764027 8273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:36.767186 8273 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:36.768987 8287 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:36.770095 8273 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:36.770181 8273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "de2f076398ee44008622fb5db2a75952"
format_stamp: "Formatted at 2024-05-03 01:59:26 on dist-test-slave-dtz3"
I20240503 01:59:36.770610 8273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:36.782171 8273 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:36.782716 8273 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:59:36.782858 8273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:36.783345 8273 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:36.784165 8273 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:59:36.784215 8273 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:36.784251 8273 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:59:36.784286 8273 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:36.793720 8400 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.135:34035 every 8 connection(s)
I20240503 01:59:36.793699 8273 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.135:34035
I20240503 01:59:36.794978 8273 server_base.cc:1166] Dumped server information to /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 01:59:36.795250 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task42rE7p/build/debug/bin/kudu as pid 8273
I20240503 01:59:36.795727 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task42rE7p/build/debug/bin/kudu
/tmp/dist-test-task42rE7p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.136:46759
--local_ip_for_outbound_sockets=127.0.99.136
--tserver_master_addrs=127.0.99.190:38597
--webserver_port=43353
--webserver_interface=127.0.99.136
--builtin_ntp_servers=127.0.99.148:34293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240503 01:59:36.812232 8401 heartbeater.cc:347] Connected to a master server at 127.0.99.190:38597
I20240503 01:59:36.812463 8401 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:36.813091 8401 heartbeater.cc:511] Master 127.0.99.190:38597 requested a full tablet report, sending...
I20240503 01:59:36.814057 6223 ts_manager.cc:194] Re-registered known tserver with Master: de2f076398ee44008622fb5db2a75952 (127.0.99.135:34035)
I20240503 01:59:36.814615 6223 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.135:44859
W20240503 01:59:36.909581 8403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:36.909813 8403 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:59:36.909873 8403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:36.912920 8403 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:59:36.912977 8403 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:59:36.913076 8403 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.136
I20240503 01:59:36.916697 8403 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:34293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.136:46759
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.0.99.136
--webserver_port=43353
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.0.99.190:38597
--never_fsync=true
--heap_profile_path=/tmp/kudu.8403
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.136
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:59:36.917516 8403 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:59:36.918594 8403 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:36.927570 8409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:36.928395 8412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:36.929612 8410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:36.930680 8403 server_base.cc:1034] running on GCE node
I20240503 01:59:36.931103 8403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:59:36.931643 8403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:59:36.932791 8403 hybrid_clock.cc:648] HybridClock initialized: now 1714701576932763 us; error 40 us; skew 500 ppm
I20240503 01:59:36.934711 8403 webserver.cc:460] Webserver started at http://127.0.99.136:43353/ using document root <none> and password file <none>
I20240503 01:59:36.935268 8403 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:36.935357 8403 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:36.938681 8403 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240503 01:59:36.940445 8418 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:36.941475 8403 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:36.941576 8403 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "7a1404fe1c1b4b388b821c8261b53104"
format_stamp: "Formatted at 2024-05-03 01:59:27 on dist-test-slave-dtz3"
I20240503 01:59:36.942041 8403 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:36.952955 8403 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:36.953466 8403 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:59:36.953603 8403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:36.954099 8403 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:36.954898 8403 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:59:36.954946 8403 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:36.955003 8403 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:59:36.955041 8403 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:36.965279 8403 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.136:46759
I20240503 01:59:36.965307 8531 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.136:46759 every 8 connection(s)
I20240503 01:59:36.966255 8403 server_base.cc:1166] Dumped server information to /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 01:59:36.971637 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task42rE7p/build/debug/bin/kudu as pid 8403
I20240503 01:59:36.978614 8532 heartbeater.cc:347] Connected to a master server at 127.0.99.190:38597
I20240503 01:59:36.978863 8532 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:36.979537 8532 heartbeater.cc:511] Master 127.0.99.190:38597 requested a full tablet report, sending...
I20240503 01:59:36.980759 6223 ts_manager.cc:194] Re-registered known tserver with Master: 7a1404fe1c1b4b388b821c8261b53104 (127.0.99.136:46759)
I20240503 01:59:36.981571 6223 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.136:34649
W20240503 01:59:37.080330 8536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:37.080572 8536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:37.083716 8536 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 01:59:37.218087 8204 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:37.218338 6344 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 01:59:37.218976 6476 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 01:59:37.226364 6738 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:37.243299 8073 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:37.243641 8466 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:37.245893 6476 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 01:59:37.250998 8204 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:37.262468 8335 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:37.276517 6607 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 01:59:37.282660 6738 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:37.291316 6344 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 01:59:37.295380 8073 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:37.297390 8466 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:37.306658 8335 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:37.325824 6607 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 01:59:37.426735 8204 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:37.429780 6344 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 01:59:37.430066 6476 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 01:59:37.439350 6738 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:37.457643 8335 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:37.463716 8139 heartbeater.cc:503] Master 127.0.99.190:38597 was elected leader, sending a full tablet report...
I20240503 01:59:37.468945 8073 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:37.469125 8466 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:37.473101 6607 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240503 01:59:37.544994 8655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:37.545248 8655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:37.548470 8655 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 01:59:37.608976 8204 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:37.615272 6476 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 01:59:37.622521 6607 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 01:59:37.630127 8073 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:37.632889 8466 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:37.637275 8270 heartbeater.cc:503] Master 127.0.99.190:38597 was elected leader, sending a full tablet report...
I20240503 01:59:37.650543 8335 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:37.659128 6738 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:37.681887 6344 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 01:59:37.695823 6607 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 01:59:37.695999 6476 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 01:59:37.698951 8073 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:37.700105 6344 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 01:59:37.711529 8335 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:37.722718 8204 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:37.729319 6738 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:37.733093 8466 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:37.775604 6494 consensus_queue.cc:237] T e5ed3e729bc64ebeac216ce5076b9fd9 P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: true } }
I20240503 01:59:37.777261 6365 raft_consensus.cc:1261] T e5ed3e729bc64ebeac216ce5076b9fd9 P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Refusing update from remote peer 0c4821d9898d42799c5789d4d2da3f63: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:37.777664 8773 consensus_queue.cc:1035] T e5ed3e729bc64ebeac216ce5076b9fd9 P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:37.777822 6757 raft_consensus.cc:1261] T e5ed3e729bc64ebeac216ce5076b9fd9 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Refusing update from remote peer 0c4821d9898d42799c5789d4d2da3f63: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:37.778913 8773 consensus_queue.cc:1035] T e5ed3e729bc64ebeac216ce5076b9fd9 P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:37.779678 7917 raft_consensus.cc:2932] T e5ed3e729bc64ebeac216ce5076b9fd9 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER de2f076398ee44008622fb5db2a75952 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: true } } }
I20240503 01:59:37.780755 6365 raft_consensus.cc:2932] T e5ed3e729bc64ebeac216ce5076b9fd9 P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER de2f076398ee44008622fb5db2a75952 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: true } } }
I20240503 01:59:37.783068 6216 catalog_manager.cc:5484] T e5ed3e729bc64ebeac216ce5076b9fd9 P 0c4821d9898d42799c5789d4d2da3f63 reported cstate change: config changed from index -1 to 4, NON_VOTER de2f076398ee44008622fb5db2a75952 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "0c4821d9898d42799c5789d4d2da3f63" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:37.783730 6757 raft_consensus.cc:2932] T e5ed3e729bc64ebeac216ce5076b9fd9 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER de2f076398ee44008622fb5db2a75952 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: true } } }
W20240503 01:59:37.785049 6432 consensus_peers.cc:485] T e5ed3e729bc64ebeac216ce5076b9fd9 P 0c4821d9898d42799c5789d4d2da3f63 -> Peer de2f076398ee44008622fb5db2a75952 (127.0.99.135:34035): Couldn't send request to peer de2f076398ee44008622fb5db2a75952. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e5ed3e729bc64ebeac216ce5076b9fd9. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:37.804149 6365 consensus_queue.cc:237] T 23b3b3d386544e5d9bfae8041cdf01a0 P 4127249ba42644388d225d614c935a25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: true } }
I20240503 01:59:37.805799 6627 raft_consensus.cc:1261] T 23b3b3d386544e5d9bfae8041cdf01a0 P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Refusing update from remote peer 4127249ba42644388d225d614c935a25: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:37.805812 6757 raft_consensus.cc:1261] T 23b3b3d386544e5d9bfae8041cdf01a0 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Refusing update from remote peer 4127249ba42644388d225d614c935a25: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:37.806226 8790 consensus_queue.cc:1035] T 23b3b3d386544e5d9bfae8041cdf01a0 P 4127249ba42644388d225d614c935a25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:37.806447 7877 consensus_queue.cc:1035] T 23b3b3d386544e5d9bfae8041cdf01a0 P 4127249ba42644388d225d614c935a25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:37.808262 7877 raft_consensus.cc:2932] T 23b3b3d386544e5d9bfae8041cdf01a0 P 4127249ba42644388d225d614c935a25 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9264f24d028347d98f7e4e2d1aedeb79 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: true } } }
W20240503 01:59:37.811144 6298 consensus_peers.cc:485] T 23b3b3d386544e5d9bfae8041cdf01a0 P 4127249ba42644388d225d614c935a25 -> Peer 9264f24d028347d98f7e4e2d1aedeb79 (127.0.99.133:41713): Couldn't send request to peer 9264f24d028347d98f7e4e2d1aedeb79. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 23b3b3d386544e5d9bfae8041cdf01a0. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:37.811209 6220 catalog_manager.cc:5484] T 23b3b3d386544e5d9bfae8041cdf01a0 P 4127249ba42644388d225d614c935a25 reported cstate change: config changed from index -1 to 4, NON_VOTER 9264f24d028347d98f7e4e2d1aedeb79 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "4127249ba42644388d225d614c935a25" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:37.812009 6757 raft_consensus.cc:2932] T 23b3b3d386544e5d9bfae8041cdf01a0 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9264f24d028347d98f7e4e2d1aedeb79 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: true } } }
I20240503 01:59:37.816759 8401 heartbeater.cc:503] Master 127.0.99.190:38597 was elected leader, sending a full tablet report...
I20240503 01:59:37.822314 6627 consensus_queue.cc:237] T 21d27cca81e8455e9934e1cc604fc2b2 P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: true } }
I20240503 01:59:37.822356 6625 raft_consensus.cc:2932] T 23b3b3d386544e5d9bfae8041cdf01a0 P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9264f24d028347d98f7e4e2d1aedeb79 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: true } } }
I20240503 01:59:37.825649 6494 raft_consensus.cc:1261] T 21d27cca81e8455e9934e1cc604fc2b2 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:37.826171 6365 raft_consensus.cc:1261] T 21d27cca81e8455e9934e1cc604fc2b2 P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:37.826180 7647 consensus_queue.cc:1035] T 21d27cca81e8455e9934e1cc604fc2b2 P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:37.826978 7647 consensus_queue.cc:1035] T 21d27cca81e8455e9934e1cc604fc2b2 P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:37.828958 8801 raft_consensus.cc:2932] T 21d27cca81e8455e9934e1cc604fc2b2 P 79163c69f6f04a9abd0208d74e1c61be [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 7a1404fe1c1b4b388b821c8261b53104 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: true } } }
I20240503 01:59:37.829296 6494 raft_consensus.cc:2932] T 21d27cca81e8455e9934e1cc604fc2b2 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 7a1404fe1c1b4b388b821c8261b53104 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: true } } }
I20240503 01:59:37.832409 6220 catalog_manager.cc:5484] T 21d27cca81e8455e9934e1cc604fc2b2 P 79163c69f6f04a9abd0208d74e1c61be reported cstate change: config changed from index -1 to 4, NON_VOTER 7a1404fe1c1b4b388b821c8261b53104 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "79163c69f6f04a9abd0208d74e1c61be" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 01:59:37.836733 6561 consensus_peers.cc:485] T 21d27cca81e8455e9934e1cc604fc2b2 P 79163c69f6f04a9abd0208d74e1c61be -> Peer 7a1404fe1c1b4b388b821c8261b53104 (127.0.99.136:46759): Couldn't send request to peer 7a1404fe1c1b4b388b821c8261b53104. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 21d27cca81e8455e9934e1cc604fc2b2. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:37.837837 6365 raft_consensus.cc:2932] T 21d27cca81e8455e9934e1cc604fc2b2 P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 7a1404fe1c1b4b388b821c8261b53104 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: true } } }
I20240503 01:59:37.844717 6365 consensus_queue.cc:237] T 470b553976064bdd8c321799ac718274 P 4127249ba42644388d225d614c935a25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: true } }
I20240503 01:59:37.846436 6494 raft_consensus.cc:1261] T 470b553976064bdd8c321799ac718274 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Refusing update from remote peer 4127249ba42644388d225d614c935a25: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:37.846807 8701 consensus_queue.cc:1035] T 470b553976064bdd8c321799ac718274 P 4127249ba42644388d225d614c935a25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:37.848131 6757 raft_consensus.cc:1261] T 470b553976064bdd8c321799ac718274 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Refusing update from remote peer 4127249ba42644388d225d614c935a25: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:37.848707 7877 raft_consensus.cc:2932] T 470b553976064bdd8c321799ac718274 P 4127249ba42644388d225d614c935a25 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 65d4632bcdd54ba596acdbbe1b5ef8af (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: true } } }
I20240503 01:59:37.849160 6494 raft_consensus.cc:2932] T 470b553976064bdd8c321799ac718274 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 65d4632bcdd54ba596acdbbe1b5ef8af (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: true } } }
I20240503 01:59:37.849781 8790 consensus_queue.cc:1035] T 470b553976064bdd8c321799ac718274 P 4127249ba42644388d225d614c935a25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:37.850869 6757 raft_consensus.cc:2932] T 470b553976064bdd8c321799ac718274 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 65d4632bcdd54ba596acdbbe1b5ef8af (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: true } } }
I20240503 01:59:37.852823 6220 catalog_manager.cc:5484] T 470b553976064bdd8c321799ac718274 P 0c4821d9898d42799c5789d4d2da3f63 reported cstate change: config changed from index -1 to 4, NON_VOTER 65d4632bcdd54ba596acdbbe1b5ef8af (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "4127249ba42644388d225d614c935a25" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: true } } }
W20240503 01:59:37.857010 6298 consensus_peers.cc:485] T 470b553976064bdd8c321799ac718274 P 4127249ba42644388d225d614c935a25 -> Peer 65d4632bcdd54ba596acdbbe1b5ef8af (127.0.99.134:35795): Couldn't send request to peer 65d4632bcdd54ba596acdbbe1b5ef8af. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 470b553976064bdd8c321799ac718274. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:37.860870 6625 consensus_queue.cc:237] T 0bdcea4e59ec4e2da63b9316f9d718be P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: true } }
I20240503 01:59:37.862478 6757 raft_consensus.cc:1261] T 0bdcea4e59ec4e2da63b9316f9d718be P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:37.862603 6365 raft_consensus.cc:1261] T 0bdcea4e59ec4e2da63b9316f9d718be P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:37.862864 7647 consensus_queue.cc:1035] T 0bdcea4e59ec4e2da63b9316f9d718be P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:37.863201 7647 consensus_queue.cc:1035] T 0bdcea4e59ec4e2da63b9316f9d718be P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:37.865585 8800 raft_consensus.cc:2932] T 0bdcea4e59ec4e2da63b9316f9d718be P 79163c69f6f04a9abd0208d74e1c61be [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER de2f076398ee44008622fb5db2a75952 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: true } } }
I20240503 01:59:37.866880 6757 raft_consensus.cc:2932] T 0bdcea4e59ec4e2da63b9316f9d718be P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER de2f076398ee44008622fb5db2a75952 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: true } } }
W20240503 01:59:37.867630 6563 consensus_peers.cc:485] T 0bdcea4e59ec4e2da63b9316f9d718be P 79163c69f6f04a9abd0208d74e1c61be -> Peer de2f076398ee44008622fb5db2a75952 (127.0.99.135:34035): Couldn't send request to peer de2f076398ee44008622fb5db2a75952. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0bdcea4e59ec4e2da63b9316f9d718be. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:37.868443 6365 raft_consensus.cc:2932] T 0bdcea4e59ec4e2da63b9316f9d718be P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER de2f076398ee44008622fb5db2a75952 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: true } } }
I20240503 01:59:37.869655 6223 catalog_manager.cc:5484] T 0bdcea4e59ec4e2da63b9316f9d718be P b6b1d5677b8d47788b7968f64b3dac3c reported cstate change: config changed from index -1 to 4, NON_VOTER de2f076398ee44008622fb5db2a75952 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "79163c69f6f04a9abd0208d74e1c61be" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: true } } }
I20240503 01:59:37.880070 6757 consensus_queue.cc:237] T f11a8034585a4d57be0e9b1e5e8bcd85 P b6b1d5677b8d47788b7968f64b3dac3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: true } }
I20240503 01:59:37.881807 6625 raft_consensus.cc:1261] T f11a8034585a4d57be0e9b1e5e8bcd85 P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Refusing update from remote peer b6b1d5677b8d47788b7968f64b3dac3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:37.882148 6494 raft_consensus.cc:1261] T f11a8034585a4d57be0e9b1e5e8bcd85 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Refusing update from remote peer b6b1d5677b8d47788b7968f64b3dac3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:37.882253 8827 consensus_queue.cc:1035] T f11a8034585a4d57be0e9b1e5e8bcd85 P b6b1d5677b8d47788b7968f64b3dac3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:37.882529 7448 consensus_queue.cc:1035] T f11a8034585a4d57be0e9b1e5e8bcd85 P b6b1d5677b8d47788b7968f64b3dac3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:37.884378 7448 raft_consensus.cc:2932] T f11a8034585a4d57be0e9b1e5e8bcd85 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9264f24d028347d98f7e4e2d1aedeb79 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: true } } }
W20240503 01:59:37.887020 6691 consensus_peers.cc:485] T f11a8034585a4d57be0e9b1e5e8bcd85 P b6b1d5677b8d47788b7968f64b3dac3c -> Peer 9264f24d028347d98f7e4e2d1aedeb79 (127.0.99.133:41713): Couldn't send request to peer 9264f24d028347d98f7e4e2d1aedeb79. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f11a8034585a4d57be0e9b1e5e8bcd85. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:37.887041 6223 catalog_manager.cc:5484] T f11a8034585a4d57be0e9b1e5e8bcd85 P b6b1d5677b8d47788b7968f64b3dac3c reported cstate change: config changed from index -1 to 4, NON_VOTER 9264f24d028347d98f7e4e2d1aedeb79 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:37.887647 6625 raft_consensus.cc:2932] T f11a8034585a4d57be0e9b1e5e8bcd85 P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9264f24d028347d98f7e4e2d1aedeb79 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: true } } }
I20240503 01:59:37.887719 6494 raft_consensus.cc:2932] T f11a8034585a4d57be0e9b1e5e8bcd85 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9264f24d028347d98f7e4e2d1aedeb79 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: true } } }
I20240503 01:59:37.897784 6625 consensus_queue.cc:237] T d6ee8175874c4063986033b0a80ccb82 P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: true } }
I20240503 01:59:37.899303 6757 raft_consensus.cc:1261] T d6ee8175874c4063986033b0a80ccb82 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:37.899682 6494 raft_consensus.cc:1261] T d6ee8175874c4063986033b0a80ccb82 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:37.899667 7647 consensus_queue.cc:1035] T d6ee8175874c4063986033b0a80ccb82 P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:37.900125 7647 consensus_queue.cc:1035] T d6ee8175874c4063986033b0a80ccb82 P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 01:59:37.900267 6561 consensus_peers.cc:485] T d6ee8175874c4063986033b0a80ccb82 P 79163c69f6f04a9abd0208d74e1c61be -> Peer 7a1404fe1c1b4b388b821c8261b53104 (127.0.99.136:46759): Couldn't send request to peer 7a1404fe1c1b4b388b821c8261b53104. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d6ee8175874c4063986033b0a80ccb82. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:37.901532 7647 raft_consensus.cc:2932] T d6ee8175874c4063986033b0a80ccb82 P 79163c69f6f04a9abd0208d74e1c61be [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 7a1404fe1c1b4b388b821c8261b53104 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: true } } }
I20240503 01:59:37.901921 6757 raft_consensus.cc:2932] T d6ee8175874c4063986033b0a80ccb82 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 7a1404fe1c1b4b388b821c8261b53104 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: true } } }
I20240503 01:59:37.903882 6494 raft_consensus.cc:2932] T d6ee8175874c4063986033b0a80ccb82 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 7a1404fe1c1b4b388b821c8261b53104 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: true } } }
I20240503 01:59:37.903885 6220 catalog_manager.cc:5484] T d6ee8175874c4063986033b0a80ccb82 P 79163c69f6f04a9abd0208d74e1c61be reported cstate change: config changed from index -1 to 4, NON_VOTER 7a1404fe1c1b4b388b821c8261b53104 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "79163c69f6f04a9abd0208d74e1c61be" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:37.913611 6494 consensus_queue.cc:237] T 2ac8004211e64c3481e75377092fa93b P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: true } }
I20240503 01:59:37.914947 6757 raft_consensus.cc:1261] T 2ac8004211e64c3481e75377092fa93b P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Refusing update from remote peer 0c4821d9898d42799c5789d4d2da3f63: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:37.915127 6365 raft_consensus.cc:1261] T 2ac8004211e64c3481e75377092fa93b P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Refusing update from remote peer 0c4821d9898d42799c5789d4d2da3f63: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:37.915453 8779 consensus_queue.cc:1035] T 2ac8004211e64c3481e75377092fa93b P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:37.915612 7917 consensus_queue.cc:1035] T 2ac8004211e64c3481e75377092fa93b P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:37.917622 8779 raft_consensus.cc:2932] T 2ac8004211e64c3481e75377092fa93b P 0c4821d9898d42799c5789d4d2da3f63 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9264f24d028347d98f7e4e2d1aedeb79 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: true } } }
I20240503 01:59:37.918434 6757 raft_consensus.cc:2932] T 2ac8004211e64c3481e75377092fa93b P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9264f24d028347d98f7e4e2d1aedeb79 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: true } } }
I20240503 01:59:37.920112 6365 raft_consensus.cc:2932] T 2ac8004211e64c3481e75377092fa93b P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9264f24d028347d98f7e4e2d1aedeb79 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: true } } }
W20240503 01:59:37.921550 6429 consensus_peers.cc:485] T 2ac8004211e64c3481e75377092fa93b P 0c4821d9898d42799c5789d4d2da3f63 -> Peer 9264f24d028347d98f7e4e2d1aedeb79 (127.0.99.133:41713): Couldn't send request to peer 9264f24d028347d98f7e4e2d1aedeb79. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2ac8004211e64c3481e75377092fa93b. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:37.921844 6217 catalog_manager.cc:5484] T 2ac8004211e64c3481e75377092fa93b P b6b1d5677b8d47788b7968f64b3dac3c reported cstate change: config changed from index -1 to 4, NON_VOTER 9264f24d028347d98f7e4e2d1aedeb79 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "0c4821d9898d42799c5789d4d2da3f63" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: true } } }
W20240503 01:59:37.932896 8777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:37.933166 8777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240503 01:59:37.934381 6494 consensus_queue.cc:237] T ff8f06c5ad8442ee942bf900212fe858 P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: true } }
I20240503 01:59:37.935920 6625 raft_consensus.cc:1261] T ff8f06c5ad8442ee942bf900212fe858 P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Refusing update from remote peer 0c4821d9898d42799c5789d4d2da3f63: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:37.935954 6365 raft_consensus.cc:1261] T ff8f06c5ad8442ee942bf900212fe858 P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Refusing update from remote peer 0c4821d9898d42799c5789d4d2da3f63: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:37.936369 7917 consensus_queue.cc:1035] T ff8f06c5ad8442ee942bf900212fe858 P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:37.936735 7917 consensus_queue.cc:1035] T ff8f06c5ad8442ee942bf900212fe858 P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 01:59:37.937147 8777 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 01:59:37.938616 7917 raft_consensus.cc:2932] T ff8f06c5ad8442ee942bf900212fe858 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 65d4632bcdd54ba596acdbbe1b5ef8af (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: true } } }
I20240503 01:59:37.939076 6625 raft_consensus.cc:2932] T ff8f06c5ad8442ee942bf900212fe858 P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 65d4632bcdd54ba596acdbbe1b5ef8af (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: true } } }
W20240503 01:59:37.940268 6429 consensus_peers.cc:485] T ff8f06c5ad8442ee942bf900212fe858 P 0c4821d9898d42799c5789d4d2da3f63 -> Peer 65d4632bcdd54ba596acdbbe1b5ef8af (127.0.99.134:35795): Couldn't send request to peer 65d4632bcdd54ba596acdbbe1b5ef8af. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ff8f06c5ad8442ee942bf900212fe858. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:37.941402 6223 catalog_manager.cc:5484] T ff8f06c5ad8442ee942bf900212fe858 P 0c4821d9898d42799c5789d4d2da3f63 reported cstate change: config changed from index -1 to 4, NON_VOTER 65d4632bcdd54ba596acdbbe1b5ef8af (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "0c4821d9898d42799c5789d4d2da3f63" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:37.945588 6365 raft_consensus.cc:2932] T ff8f06c5ad8442ee942bf900212fe858 P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 65d4632bcdd54ba596acdbbe1b5ef8af (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: true } } }
I20240503 01:59:37.951859 6494 consensus_queue.cc:237] T 1d392e08c15a4d4098a350287b3163ef P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: true } }
I20240503 01:59:37.953320 6757 raft_consensus.cc:1261] T 1d392e08c15a4d4098a350287b3163ef P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Refusing update from remote peer 0c4821d9898d42799c5789d4d2da3f63: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:37.953380 6625 raft_consensus.cc:1261] T 1d392e08c15a4d4098a350287b3163ef P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Refusing update from remote peer 0c4821d9898d42799c5789d4d2da3f63: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 01:59:37.953552 6432 consensus_peers.cc:485] T 1d392e08c15a4d4098a350287b3163ef P 0c4821d9898d42799c5789d4d2da3f63 -> Peer de2f076398ee44008622fb5db2a75952 (127.0.99.135:34035): Couldn't send request to peer de2f076398ee44008622fb5db2a75952. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1d392e08c15a4d4098a350287b3163ef. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:37.953789 8774 consensus_queue.cc:1035] T 1d392e08c15a4d4098a350287b3163ef P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:37.953938 7917 consensus_queue.cc:1035] T 1d392e08c15a4d4098a350287b3163ef P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:37.955554 8774 raft_consensus.cc:2932] T 1d392e08c15a4d4098a350287b3163ef P 0c4821d9898d42799c5789d4d2da3f63 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER de2f076398ee44008622fb5db2a75952 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: true } } }
I20240503 01:59:37.955943 6757 raft_consensus.cc:2932] T 1d392e08c15a4d4098a350287b3163ef P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER de2f076398ee44008622fb5db2a75952 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: true } } }
I20240503 01:59:37.957978 6220 catalog_manager.cc:5484] T 1d392e08c15a4d4098a350287b3163ef P 0c4821d9898d42799c5789d4d2da3f63 reported cstate change: config changed from index -1 to 4, NON_VOTER de2f076398ee44008622fb5db2a75952 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "0c4821d9898d42799c5789d4d2da3f63" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:37.959555 6625 raft_consensus.cc:2932] T 1d392e08c15a4d4098a350287b3163ef P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER de2f076398ee44008622fb5db2a75952 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: true } } }
I20240503 01:59:37.976828 6365 consensus_queue.cc:237] T 67150a2f5cb147b8b57a8c8040b567e1 P 4127249ba42644388d225d614c935a25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: true } }
I20240503 01:59:37.978299 6494 raft_consensus.cc:1261] T 67150a2f5cb147b8b57a8c8040b567e1 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Refusing update from remote peer 4127249ba42644388d225d614c935a25: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:37.978392 6625 raft_consensus.cc:1261] T 67150a2f5cb147b8b57a8c8040b567e1 P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Refusing update from remote peer 4127249ba42644388d225d614c935a25: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 01:59:37.978534 6298 consensus_peers.cc:485] T 67150a2f5cb147b8b57a8c8040b567e1 P 4127249ba42644388d225d614c935a25 -> Peer 65d4632bcdd54ba596acdbbe1b5ef8af (127.0.99.134:35795): Couldn't send request to peer 65d4632bcdd54ba596acdbbe1b5ef8af. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 67150a2f5cb147b8b57a8c8040b567e1. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:37.978708 8790 consensus_queue.cc:1035] T 67150a2f5cb147b8b57a8c8040b567e1 P 4127249ba42644388d225d614c935a25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:37.978884 7877 consensus_queue.cc:1035] T 67150a2f5cb147b8b57a8c8040b567e1 P 4127249ba42644388d225d614c935a25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:37.980729 8790 raft_consensus.cc:2932] T 67150a2f5cb147b8b57a8c8040b567e1 P 4127249ba42644388d225d614c935a25 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 65d4632bcdd54ba596acdbbe1b5ef8af (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: true } } }
I20240503 01:59:37.981137 6625 raft_consensus.cc:2932] T 67150a2f5cb147b8b57a8c8040b567e1 P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 65d4632bcdd54ba596acdbbe1b5ef8af (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: true } } }
I20240503 01:59:37.982193 6494 raft_consensus.cc:2932] T 67150a2f5cb147b8b57a8c8040b567e1 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 65d4632bcdd54ba596acdbbe1b5ef8af (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: true } } }
I20240503 01:59:37.983722 6223 catalog_manager.cc:5484] T 67150a2f5cb147b8b57a8c8040b567e1 P 79163c69f6f04a9abd0208d74e1c61be reported cstate change: config changed from index -1 to 4, NON_VOTER 65d4632bcdd54ba596acdbbe1b5ef8af (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "4127249ba42644388d225d614c935a25" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: true } } }
I20240503 01:59:37.985502 8532 heartbeater.cc:503] Master 127.0.99.190:38597 was elected leader, sending a full tablet report...
I20240503 01:59:37.996217 6625 consensus_queue.cc:237] T a9a74e04ce3745b2b479e96a3f3dc86e P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: true } }
I20240503 01:59:37.997850 6757 raft_consensus.cc:1261] T a9a74e04ce3745b2b479e96a3f3dc86e P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:37.998073 6365 raft_consensus.cc:1261] T a9a74e04ce3745b2b479e96a3f3dc86e P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:37.998281 8801 consensus_queue.cc:1035] T a9a74e04ce3745b2b479e96a3f3dc86e P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:37.998498 8800 consensus_queue.cc:1035] T a9a74e04ce3745b2b479e96a3f3dc86e P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 01:59:37.998697 6561 consensus_peers.cc:485] T a9a74e04ce3745b2b479e96a3f3dc86e P 79163c69f6f04a9abd0208d74e1c61be -> Peer 7a1404fe1c1b4b388b821c8261b53104 (127.0.99.136:46759): Couldn't send request to peer 7a1404fe1c1b4b388b821c8261b53104. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a9a74e04ce3745b2b479e96a3f3dc86e. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:38.000782 7647 raft_consensus.cc:2932] T a9a74e04ce3745b2b479e96a3f3dc86e P 79163c69f6f04a9abd0208d74e1c61be [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 7a1404fe1c1b4b388b821c8261b53104 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: true } } }
I20240503 01:59:38.001354 6365 raft_consensus.cc:2932] T a9a74e04ce3745b2b479e96a3f3dc86e P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 7a1404fe1c1b4b388b821c8261b53104 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: true } } }
I20240503 01:59:38.001444 6757 raft_consensus.cc:2932] T a9a74e04ce3745b2b479e96a3f3dc86e P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 7a1404fe1c1b4b388b821c8261b53104 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: true } } }
I20240503 01:59:38.003966 6220 catalog_manager.cc:5484] T a9a74e04ce3745b2b479e96a3f3dc86e P 79163c69f6f04a9abd0208d74e1c61be reported cstate change: config changed from index -1 to 4, NON_VOTER 7a1404fe1c1b4b388b821c8261b53104 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "79163c69f6f04a9abd0208d74e1c61be" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:38.014317 6365 consensus_queue.cc:237] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 4127249ba42644388d225d614c935a25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: true } }
I20240503 01:59:38.016252 6494 raft_consensus.cc:1261] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Refusing update from remote peer 4127249ba42644388d225d614c935a25: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:38.016420 6625 raft_consensus.cc:1261] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Refusing update from remote peer 4127249ba42644388d225d614c935a25: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:38.016721 7877 consensus_queue.cc:1035] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 4127249ba42644388d225d614c935a25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 01:59:38.017020 6298 consensus_peers.cc:485] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 4127249ba42644388d225d614c935a25 -> Peer 9264f24d028347d98f7e4e2d1aedeb79 (127.0.99.133:41713): Couldn't send request to peer 9264f24d028347d98f7e4e2d1aedeb79. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2088c84e4eef4ccfbc70d7b668b8f2cf. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:38.017509 8701 consensus_queue.cc:1035] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 4127249ba42644388d225d614c935a25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:38.021806 8701 raft_consensus.cc:2932] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 4127249ba42644388d225d614c935a25 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9264f24d028347d98f7e4e2d1aedeb79 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: true } } }
I20240503 01:59:38.022203 6625 raft_consensus.cc:2932] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9264f24d028347d98f7e4e2d1aedeb79 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: true } } }
I20240503 01:59:38.022212 6494 raft_consensus.cc:2932] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9264f24d028347d98f7e4e2d1aedeb79 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: true } } }
I20240503 01:59:38.025022 6223 catalog_manager.cc:5484] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 79163c69f6f04a9abd0208d74e1c61be reported cstate change: config changed from index -1 to 4, NON_VOTER 9264f24d028347d98f7e4e2d1aedeb79 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "4127249ba42644388d225d614c935a25" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: true } } }
I20240503 01:59:38.035481 6625 consensus_queue.cc:237] T a5926dae08184952bc693342c2c61ebf P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: true } }
I20240503 01:59:38.037539 6757 raft_consensus.cc:1261] T a5926dae08184952bc693342c2c61ebf P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 01:59:38.038002 6563 consensus_peers.cc:485] T a5926dae08184952bc693342c2c61ebf P 79163c69f6f04a9abd0208d74e1c61be -> Peer de2f076398ee44008622fb5db2a75952 (127.0.99.135:34035): Couldn't send request to peer de2f076398ee44008622fb5db2a75952. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a5926dae08184952bc693342c2c61ebf. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:38.038261 8800 consensus_queue.cc:1035] T a5926dae08184952bc693342c2c61ebf P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:38.039237 6365 raft_consensus.cc:1261] T a5926dae08184952bc693342c2c61ebf P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:38.039637 8800 consensus_queue.cc:1035] T a5926dae08184952bc693342c2c61ebf P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:38.040849 8800 raft_consensus.cc:2932] T a5926dae08184952bc693342c2c61ebf P 79163c69f6f04a9abd0208d74e1c61be [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER de2f076398ee44008622fb5db2a75952 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: true } } }
I20240503 01:59:38.042989 6757 raft_consensus.cc:2932] T a5926dae08184952bc693342c2c61ebf P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER de2f076398ee44008622fb5db2a75952 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: true } } }
I20240503 01:59:38.045979 6218 catalog_manager.cc:5484] T a5926dae08184952bc693342c2c61ebf P 79163c69f6f04a9abd0208d74e1c61be reported cstate change: config changed from index -1 to 4, NON_VOTER de2f076398ee44008622fb5db2a75952 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "79163c69f6f04a9abd0208d74e1c61be" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:38.046983 6365 raft_consensus.cc:2932] T a5926dae08184952bc693342c2c61ebf P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER de2f076398ee44008622fb5db2a75952 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: true } } }
I20240503 01:59:38.060528 6757 consensus_queue.cc:237] T 2c8c328220e64e168c74fea7619f0c3b P b6b1d5677b8d47788b7968f64b3dac3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: true } }
I20240503 01:59:38.062083 6625 raft_consensus.cc:1261] T 2c8c328220e64e168c74fea7619f0c3b P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Refusing update from remote peer b6b1d5677b8d47788b7968f64b3dac3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:38.063112 8827 consensus_queue.cc:1035] T 2c8c328220e64e168c74fea7619f0c3b P b6b1d5677b8d47788b7968f64b3dac3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:38.063356 6494 raft_consensus.cc:1261] T 2c8c328220e64e168c74fea7619f0c3b P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Refusing update from remote peer b6b1d5677b8d47788b7968f64b3dac3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:38.064047 8833 consensus_queue.cc:1035] T 2c8c328220e64e168c74fea7619f0c3b P b6b1d5677b8d47788b7968f64b3dac3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:38.066984 6494 raft_consensus.cc:2932] T 2c8c328220e64e168c74fea7619f0c3b P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 65d4632bcdd54ba596acdbbe1b5ef8af (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: true } } }
I20240503 01:59:38.067221 6476 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 01:59:38.066517 8833 raft_consensus.cc:2932] T 2c8c328220e64e168c74fea7619f0c3b P b6b1d5677b8d47788b7968f64b3dac3c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 65d4632bcdd54ba596acdbbe1b5ef8af (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: true } } }
I20240503 01:59:38.069876 6216 catalog_manager.cc:5484] T 2c8c328220e64e168c74fea7619f0c3b P 0c4821d9898d42799c5789d4d2da3f63 reported cstate change: config changed from index -1 to 4, NON_VOTER 65d4632bcdd54ba596acdbbe1b5ef8af (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: true } } }
I20240503 01:59:38.075179 6625 raft_consensus.cc:2932] T 2c8c328220e64e168c74fea7619f0c3b P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 65d4632bcdd54ba596acdbbe1b5ef8af (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: true } } }
W20240503 01:59:38.076017 6691 consensus_peers.cc:485] T 2c8c328220e64e168c74fea7619f0c3b P b6b1d5677b8d47788b7968f64b3dac3c -> Peer 65d4632bcdd54ba596acdbbe1b5ef8af (127.0.99.134:35795): Couldn't send request to peer 65d4632bcdd54ba596acdbbe1b5ef8af. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2c8c328220e64e168c74fea7619f0c3b. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:38.080287 8204 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:38.081866 6344 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 01:59:38.091383 8073 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:38.095371 8466 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:38.096341 6738 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:38.097038 8335 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:38.106587 6495 consensus_queue.cc:237] T 120d5dd4f26c42399702141094ce7982 P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: true } }
I20240503 01:59:38.108103 6757 raft_consensus.cc:1261] T 120d5dd4f26c42399702141094ce7982 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Refusing update from remote peer 0c4821d9898d42799c5789d4d2da3f63: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:38.108449 8773 consensus_queue.cc:1035] T 120d5dd4f26c42399702141094ce7982 P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:38.108980 6365 raft_consensus.cc:1261] T 120d5dd4f26c42399702141094ce7982 P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Refusing update from remote peer 0c4821d9898d42799c5789d4d2da3f63: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:38.109524 8773 consensus_queue.cc:1035] T 120d5dd4f26c42399702141094ce7982 P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:38.110373 6607 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 01:59:38.111747 8773 raft_consensus.cc:2932] T 120d5dd4f26c42399702141094ce7982 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 7a1404fe1c1b4b388b821c8261b53104 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: true } } }
W20240503 01:59:38.112862 6430 consensus_peers.cc:485] T 120d5dd4f26c42399702141094ce7982 P 0c4821d9898d42799c5789d4d2da3f63 -> Peer 7a1404fe1c1b4b388b821c8261b53104 (127.0.99.136:46759): Couldn't send request to peer 7a1404fe1c1b4b388b821c8261b53104. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 120d5dd4f26c42399702141094ce7982. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:38.113195 6365 raft_consensus.cc:2932] T 120d5dd4f26c42399702141094ce7982 P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 7a1404fe1c1b4b388b821c8261b53104 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: true } } }
I20240503 01:59:38.117570 6220 catalog_manager.cc:5484] T 120d5dd4f26c42399702141094ce7982 P 4127249ba42644388d225d614c935a25 reported cstate change: config changed from index -1 to 4, NON_VOTER 7a1404fe1c1b4b388b821c8261b53104 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "0c4821d9898d42799c5789d4d2da3f63" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: true } } }
I20240503 01:59:38.121099 6757 raft_consensus.cc:2932] T 120d5dd4f26c42399702141094ce7982 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 7a1404fe1c1b4b388b821c8261b53104 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: true } } }
Runtime error: ksck discovered errors
I20240503 01:59:38.155856 6625 consensus_queue.cc:237] T a388d59f6ba94e91958f55b340343fa6 P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: true } }
I20240503 01:59:38.157475 6495 raft_consensus.cc:1261] T a388d59f6ba94e91958f55b340343fa6 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:38.157500 6365 raft_consensus.cc:1261] T a388d59f6ba94e91958f55b340343fa6 P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:38.157858 7647 consensus_queue.cc:1035] T a388d59f6ba94e91958f55b340343fa6 P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:38.158404 8801 consensus_queue.cc:1035] T a388d59f6ba94e91958f55b340343fa6 P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:38.160248 8801 raft_consensus.cc:2932] T a388d59f6ba94e91958f55b340343fa6 P 79163c69f6f04a9abd0208d74e1c61be [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9264f24d028347d98f7e4e2d1aedeb79 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: true } } }
I20240503 01:59:38.160588 6365 raft_consensus.cc:2932] T a388d59f6ba94e91958f55b340343fa6 P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9264f24d028347d98f7e4e2d1aedeb79 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: true } } }
W20240503 01:59:38.161870 6560 consensus_peers.cc:485] T a388d59f6ba94e91958f55b340343fa6 P 79163c69f6f04a9abd0208d74e1c61be -> Peer 9264f24d028347d98f7e4e2d1aedeb79 (127.0.99.133:41713): Couldn't send request to peer 9264f24d028347d98f7e4e2d1aedeb79. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a388d59f6ba94e91958f55b340343fa6. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:38.163067 6216 catalog_manager.cc:5484] T a388d59f6ba94e91958f55b340343fa6 P 79163c69f6f04a9abd0208d74e1c61be reported cstate change: config changed from index -1 to 4, NON_VOTER 9264f24d028347d98f7e4e2d1aedeb79 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "79163c69f6f04a9abd0208d74e1c61be" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:38.169528 6495 raft_consensus.cc:2932] T a388d59f6ba94e91958f55b340343fa6 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9264f24d028347d98f7e4e2d1aedeb79 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: true } } }
I20240503 01:59:38.171631 6365 consensus_queue.cc:237] T 4e7ceb1453804246a41bd1dda0e39d02 P 4127249ba42644388d225d614c935a25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: true } }
I20240503 01:59:38.173311 6625 raft_consensus.cc:1261] T 4e7ceb1453804246a41bd1dda0e39d02 P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Refusing update from remote peer 4127249ba42644388d225d614c935a25: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 01:59:38.173702 6298 consensus_peers.cc:485] T 4e7ceb1453804246a41bd1dda0e39d02 P 4127249ba42644388d225d614c935a25 -> Peer 65d4632bcdd54ba596acdbbe1b5ef8af (127.0.99.134:35795): Couldn't send request to peer 65d4632bcdd54ba596acdbbe1b5ef8af. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4e7ceb1453804246a41bd1dda0e39d02. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:38.173800 8790 consensus_queue.cc:1035] T 4e7ceb1453804246a41bd1dda0e39d02 P 4127249ba42644388d225d614c935a25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:38.174243 6495 raft_consensus.cc:1261] T 4e7ceb1453804246a41bd1dda0e39d02 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Refusing update from remote peer 4127249ba42644388d225d614c935a25: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:38.174623 8790 consensus_queue.cc:1035] T 4e7ceb1453804246a41bd1dda0e39d02 P 4127249ba42644388d225d614c935a25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:38.176360 8917 raft_consensus.cc:2932] T 4e7ceb1453804246a41bd1dda0e39d02 P 4127249ba42644388d225d614c935a25 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 65d4632bcdd54ba596acdbbe1b5ef8af (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: true } } }
I20240503 01:59:38.176823 6495 raft_consensus.cc:2932] T 4e7ceb1453804246a41bd1dda0e39d02 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 65d4632bcdd54ba596acdbbe1b5ef8af (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: true } } }
I20240503 01:59:38.177134 6625 raft_consensus.cc:2932] T 4e7ceb1453804246a41bd1dda0e39d02 P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 65d4632bcdd54ba596acdbbe1b5ef8af (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: true } } }
I20240503 01:59:38.179070 6216 catalog_manager.cc:5484] T 4e7ceb1453804246a41bd1dda0e39d02 P 0c4821d9898d42799c5789d4d2da3f63 reported cstate change: config changed from index -1 to 4, NON_VOTER 65d4632bcdd54ba596acdbbe1b5ef8af (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "4127249ba42644388d225d614c935a25" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: true } } }
I20240503 01:59:38.186630 6757 consensus_queue.cc:237] T e5a00d4ce3a64ca5a7c5bd65c556f429 P b6b1d5677b8d47788b7968f64b3dac3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: true } }
I20240503 01:59:38.188323 6495 raft_consensus.cc:1261] T e5a00d4ce3a64ca5a7c5bd65c556f429 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Refusing update from remote peer b6b1d5677b8d47788b7968f64b3dac3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:38.188719 6625 raft_consensus.cc:1261] T e5a00d4ce3a64ca5a7c5bd65c556f429 P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Refusing update from remote peer b6b1d5677b8d47788b7968f64b3dac3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:38.188880 8828 consensus_queue.cc:1035] T e5a00d4ce3a64ca5a7c5bd65c556f429 P b6b1d5677b8d47788b7968f64b3dac3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:38.189136 8832 consensus_queue.cc:1035] T e5a00d4ce3a64ca5a7c5bd65c556f429 P b6b1d5677b8d47788b7968f64b3dac3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:38.190950 8832 raft_consensus.cc:2932] T e5a00d4ce3a64ca5a7c5bd65c556f429 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER de2f076398ee44008622fb5db2a75952 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: true } } }
W20240503 01:59:38.191893 6694 consensus_peers.cc:485] T e5a00d4ce3a64ca5a7c5bd65c556f429 P b6b1d5677b8d47788b7968f64b3dac3c -> Peer de2f076398ee44008622fb5db2a75952 (127.0.99.135:34035): Couldn't send request to peer de2f076398ee44008622fb5db2a75952. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e5a00d4ce3a64ca5a7c5bd65c556f429. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:38.191776 6625 raft_consensus.cc:2932] T e5a00d4ce3a64ca5a7c5bd65c556f429 P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER de2f076398ee44008622fb5db2a75952 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: true } } }
I20240503 01:59:38.192828 6495 raft_consensus.cc:2932] T e5a00d4ce3a64ca5a7c5bd65c556f429 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER de2f076398ee44008622fb5db2a75952 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: true } } }
I20240503 01:59:38.195139 6220 catalog_manager.cc:5484] T e5a00d4ce3a64ca5a7c5bd65c556f429 P 0c4821d9898d42799c5789d4d2da3f63 reported cstate change: config changed from index -1 to 4, NON_VOTER de2f076398ee44008622fb5db2a75952 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: true } } }
I20240503 01:59:38.204850 6625 consensus_queue.cc:237] T a655d914858447b59fe5763e43c0807d P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: true } }
I20240503 01:59:38.206353 6365 raft_consensus.cc:1261] T a655d914858447b59fe5763e43c0807d P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:38.206343 6757 raft_consensus.cc:1261] T a655d914858447b59fe5763e43c0807d P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 01:59:38.206741 6561 consensus_peers.cc:485] T a655d914858447b59fe5763e43c0807d P 79163c69f6f04a9abd0208d74e1c61be -> Peer 7a1404fe1c1b4b388b821c8261b53104 (127.0.99.136:46759): Couldn't send request to peer 7a1404fe1c1b4b388b821c8261b53104. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a655d914858447b59fe5763e43c0807d. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:38.206780 7647 consensus_queue.cc:1035] T a655d914858447b59fe5763e43c0807d P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:38.206951 8929 consensus_queue.cc:1035] T a655d914858447b59fe5763e43c0807d P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:38.208750 7647 raft_consensus.cc:2932] T a655d914858447b59fe5763e43c0807d P 79163c69f6f04a9abd0208d74e1c61be [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 7a1404fe1c1b4b388b821c8261b53104 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: true } } }
I20240503 01:59:38.210355 6757 raft_consensus.cc:2932] T a655d914858447b59fe5763e43c0807d P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 7a1404fe1c1b4b388b821c8261b53104 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: true } } }
I20240503 01:59:38.211324 6216 catalog_manager.cc:5484] T a655d914858447b59fe5763e43c0807d P 79163c69f6f04a9abd0208d74e1c61be reported cstate change: config changed from index -1 to 4, NON_VOTER 7a1404fe1c1b4b388b821c8261b53104 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "79163c69f6f04a9abd0208d74e1c61be" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:38.212391 6365 raft_consensus.cc:2932] T a655d914858447b59fe5763e43c0807d P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 7a1404fe1c1b4b388b821c8261b53104 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: true } } }
I20240503 01:59:38.245774 9020 ts_tablet_manager.cc:902] T 470b553976064bdd8c321799ac718274 P 65d4632bcdd54ba596acdbbe1b5ef8af: Initiating tablet copy from peer 4127249ba42644388d225d614c935a25 (127.0.99.129:35613)
I20240503 01:59:38.246776 9020 tablet_copy_client.cc:308] T 470b553976064bdd8c321799ac718274 P 65d4632bcdd54ba596acdbbe1b5ef8af: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:35613
I20240503 01:59:38.251737 6385 tablet_copy_service.cc:140] P 4127249ba42644388d225d614c935a25: Received BeginTabletCopySession request for tablet 470b553976064bdd8c321799ac718274 from peer 65d4632bcdd54ba596acdbbe1b5ef8af ({username='slave'} at 127.0.99.134:49429)
I20240503 01:59:38.251899 6385 tablet_copy_service.cc:161] P 4127249ba42644388d225d614c935a25: Beginning new tablet copy session on tablet 470b553976064bdd8c321799ac718274 from peer 65d4632bcdd54ba596acdbbe1b5ef8af at {username='slave'} at 127.0.99.134:49429: session id = 65d4632bcdd54ba596acdbbe1b5ef8af-470b553976064bdd8c321799ac718274
I20240503 01:59:38.253207 6385 tablet_copy_source_session.cc:205] T 470b553976064bdd8c321799ac718274 P 4127249ba42644388d225d614c935a25: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:38.256469 9020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 470b553976064bdd8c321799ac718274. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:38.259250 9033 ts_tablet_manager.cc:902] T e5ed3e729bc64ebeac216ce5076b9fd9 P de2f076398ee44008622fb5db2a75952: Initiating tablet copy from peer 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130:34189)
I20240503 01:59:38.259948 9033 tablet_copy_client.cc:308] T e5ed3e729bc64ebeac216ce5076b9fd9 P de2f076398ee44008622fb5db2a75952: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:34189
I20240503 01:59:38.262919 9020 tablet_copy_client.cc:783] T 470b553976064bdd8c321799ac718274 P 65d4632bcdd54ba596acdbbe1b5ef8af: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:38.263130 9020 tablet_copy_client.cc:647] T 470b553976064bdd8c321799ac718274 P 65d4632bcdd54ba596acdbbe1b5ef8af: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:38.264576 9023 tablet_copy_client.cc:1002] T 470b553976064bdd8c321799ac718274 P 65d4632bcdd54ba596acdbbe1b5ef8af: tablet copy: Injecting latency into file download: 1500
I20240503 01:59:38.265058 6516 tablet_copy_service.cc:140] P 0c4821d9898d42799c5789d4d2da3f63: Received BeginTabletCopySession request for tablet e5ed3e729bc64ebeac216ce5076b9fd9 from peer de2f076398ee44008622fb5db2a75952 ({username='slave'} at 127.0.99.135:55869)
I20240503 01:59:38.265198 6516 tablet_copy_service.cc:161] P 0c4821d9898d42799c5789d4d2da3f63: Beginning new tablet copy session on tablet e5ed3e729bc64ebeac216ce5076b9fd9 from peer de2f076398ee44008622fb5db2a75952 at {username='slave'} at 127.0.99.135:55869: session id = de2f076398ee44008622fb5db2a75952-e5ed3e729bc64ebeac216ce5076b9fd9
I20240503 01:59:38.266494 6516 tablet_copy_source_session.cc:205] T e5ed3e729bc64ebeac216ce5076b9fd9 P 0c4821d9898d42799c5789d4d2da3f63: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:38.269701 9033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5ed3e729bc64ebeac216ce5076b9fd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:38.275219 9033 tablet_copy_client.cc:783] T e5ed3e729bc64ebeac216ce5076b9fd9 P de2f076398ee44008622fb5db2a75952: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:38.275460 9033 tablet_copy_client.cc:647] T e5ed3e729bc64ebeac216ce5076b9fd9 P de2f076398ee44008622fb5db2a75952: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:38.277323 9035 tablet_copy_client.cc:1002] T e5ed3e729bc64ebeac216ce5076b9fd9 P de2f076398ee44008622fb5db2a75952: tablet copy: Injecting latency into file download: 1500
I20240503 01:59:38.284447 9059 ts_tablet_manager.cc:902] T d6ee8175874c4063986033b0a80ccb82 P 7a1404fe1c1b4b388b821c8261b53104: Initiating tablet copy from peer 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131:44005)
I20240503 01:59:38.285326 9059 tablet_copy_client.cc:308] T d6ee8175874c4063986033b0a80ccb82 P 7a1404fe1c1b4b388b821c8261b53104: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:44005
I20240503 01:59:38.290386 6647 tablet_copy_service.cc:140] P 79163c69f6f04a9abd0208d74e1c61be: Received BeginTabletCopySession request for tablet d6ee8175874c4063986033b0a80ccb82 from peer 7a1404fe1c1b4b388b821c8261b53104 ({username='slave'} at 127.0.99.136:51745)
I20240503 01:59:38.290531 6647 tablet_copy_service.cc:161] P 79163c69f6f04a9abd0208d74e1c61be: Beginning new tablet copy session on tablet d6ee8175874c4063986033b0a80ccb82 from peer 7a1404fe1c1b4b388b821c8261b53104 at {username='slave'} at 127.0.99.136:51745: session id = 7a1404fe1c1b4b388b821c8261b53104-d6ee8175874c4063986033b0a80ccb82
I20240503 01:59:38.291842 6647 tablet_copy_source_session.cc:205] T d6ee8175874c4063986033b0a80ccb82 P 79163c69f6f04a9abd0208d74e1c61be: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:38.295055 9059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6ee8175874c4063986033b0a80ccb82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:38.300220 9059 tablet_copy_client.cc:783] T d6ee8175874c4063986033b0a80ccb82 P 7a1404fe1c1b4b388b821c8261b53104: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:38.300505 9059 tablet_copy_client.cc:647] T d6ee8175874c4063986033b0a80ccb82 P 7a1404fe1c1b4b388b821c8261b53104: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:38.302484 9060 tablet_copy_client.cc:1002] T d6ee8175874c4063986033b0a80ccb82 P 7a1404fe1c1b4b388b821c8261b53104: tablet copy: Injecting latency into file download: 1500
I20240503 01:59:38.318508 9083 ts_tablet_manager.cc:902] T ff8f06c5ad8442ee942bf900212fe858 P 65d4632bcdd54ba596acdbbe1b5ef8af: Initiating tablet copy from peer 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130:34189)
I20240503 01:59:38.319108 9083 tablet_copy_client.cc:308] T ff8f06c5ad8442ee942bf900212fe858 P 65d4632bcdd54ba596acdbbe1b5ef8af: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:34189
W20240503 01:59:38.322518 8980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:38.322784 8980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240503 01:59:38.323321 6516 tablet_copy_service.cc:140] P 0c4821d9898d42799c5789d4d2da3f63: Received BeginTabletCopySession request for tablet ff8f06c5ad8442ee942bf900212fe858 from peer 65d4632bcdd54ba596acdbbe1b5ef8af ({username='slave'} at 127.0.99.134:55903)
I20240503 01:59:38.323413 6516 tablet_copy_service.cc:161] P 0c4821d9898d42799c5789d4d2da3f63: Beginning new tablet copy session on tablet ff8f06c5ad8442ee942bf900212fe858 from peer 65d4632bcdd54ba596acdbbe1b5ef8af at {username='slave'} at 127.0.99.134:55903: session id = 65d4632bcdd54ba596acdbbe1b5ef8af-ff8f06c5ad8442ee942bf900212fe858
I20240503 01:59:38.324165 6516 tablet_copy_source_session.cc:205] T ff8f06c5ad8442ee942bf900212fe858 P 0c4821d9898d42799c5789d4d2da3f63: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:38.325136 9083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff8f06c5ad8442ee942bf900212fe858. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 01:59:38.326696 8980 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 01:59:38.327044 9083 tablet_copy_client.cc:783] T ff8f06c5ad8442ee942bf900212fe858 P 65d4632bcdd54ba596acdbbe1b5ef8af: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:38.327185 9083 tablet_copy_client.cc:647] T ff8f06c5ad8442ee942bf900212fe858 P 65d4632bcdd54ba596acdbbe1b5ef8af: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:38.328164 9085 tablet_copy_client.cc:1002] T ff8f06c5ad8442ee942bf900212fe858 P 65d4632bcdd54ba596acdbbe1b5ef8af: tablet copy: Injecting latency into file download: 1500
I20240503 01:59:38.347908 9119 ts_tablet_manager.cc:902] T 2ac8004211e64c3481e75377092fa93b P 9264f24d028347d98f7e4e2d1aedeb79: Initiating tablet copy from peer 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130:34189)
I20240503 01:59:38.348897 9119 tablet_copy_client.cc:308] T 2ac8004211e64c3481e75377092fa93b P 9264f24d028347d98f7e4e2d1aedeb79: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:34189
I20240503 01:59:38.357798 6516 tablet_copy_service.cc:140] P 0c4821d9898d42799c5789d4d2da3f63: Received BeginTabletCopySession request for tablet 2ac8004211e64c3481e75377092fa93b from peer 9264f24d028347d98f7e4e2d1aedeb79 ({username='slave'} at 127.0.99.133:35409)
I20240503 01:59:38.357906 6516 tablet_copy_service.cc:161] P 0c4821d9898d42799c5789d4d2da3f63: Beginning new tablet copy session on tablet 2ac8004211e64c3481e75377092fa93b from peer 9264f24d028347d98f7e4e2d1aedeb79 at {username='slave'} at 127.0.99.133:35409: session id = 9264f24d028347d98f7e4e2d1aedeb79-2ac8004211e64c3481e75377092fa93b
I20240503 01:59:38.358228 9126 ts_tablet_manager.cc:902] T 1d392e08c15a4d4098a350287b3163ef P de2f076398ee44008622fb5db2a75952: Initiating tablet copy from peer 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130:34189)
I20240503 01:59:38.358907 6516 tablet_copy_source_session.cc:205] T 2ac8004211e64c3481e75377092fa93b P 0c4821d9898d42799c5789d4d2da3f63: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:38.358911 9126 tablet_copy_client.cc:308] T 1d392e08c15a4d4098a350287b3163ef P de2f076398ee44008622fb5db2a75952: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:34189
I20240503 01:59:38.362401 9119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ac8004211e64c3481e75377092fa93b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:38.364449 6516 tablet_copy_service.cc:140] P 0c4821d9898d42799c5789d4d2da3f63: Received BeginTabletCopySession request for tablet 1d392e08c15a4d4098a350287b3163ef from peer de2f076398ee44008622fb5db2a75952 ({username='slave'} at 127.0.99.135:55869)
I20240503 01:59:38.364539 6516 tablet_copy_service.cc:161] P 0c4821d9898d42799c5789d4d2da3f63: Beginning new tablet copy session on tablet 1d392e08c15a4d4098a350287b3163ef from peer de2f076398ee44008622fb5db2a75952 at {username='slave'} at 127.0.99.135:55869: session id = de2f076398ee44008622fb5db2a75952-1d392e08c15a4d4098a350287b3163ef
I20240503 01:59:38.365271 6516 tablet_copy_source_session.cc:205] T 1d392e08c15a4d4098a350287b3163ef P 0c4821d9898d42799c5789d4d2da3f63: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:38.367818 9126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d392e08c15a4d4098a350287b3163ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:38.370220 9141 ts_tablet_manager.cc:902] T 0bdcea4e59ec4e2da63b9316f9d718be P de2f076398ee44008622fb5db2a75952: Initiating tablet copy from peer 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131:44005)
I20240503 01:59:38.370355 9126 tablet_copy_client.cc:783] T 1d392e08c15a4d4098a350287b3163ef P de2f076398ee44008622fb5db2a75952: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:38.370517 9126 tablet_copy_client.cc:647] T 1d392e08c15a4d4098a350287b3163ef P de2f076398ee44008622fb5db2a75952: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:38.370816 9141 tablet_copy_client.cc:308] T 0bdcea4e59ec4e2da63b9316f9d718be P de2f076398ee44008622fb5db2a75952: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:44005
I20240503 01:59:38.370880 9136 ts_tablet_manager.cc:902] T f11a8034585a4d57be0e9b1e5e8bcd85 P 9264f24d028347d98f7e4e2d1aedeb79: Initiating tablet copy from peer b6b1d5677b8d47788b7968f64b3dac3c (127.0.99.132:46067)
I20240503 01:59:38.371508 9136 tablet_copy_client.cc:308] T f11a8034585a4d57be0e9b1e5e8bcd85 P 9264f24d028347d98f7e4e2d1aedeb79: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:46067
I20240503 01:59:38.373237 9130 tablet_copy_client.cc:1002] T 1d392e08c15a4d4098a350287b3163ef P de2f076398ee44008622fb5db2a75952: tablet copy: Injecting latency into file download: 1500
I20240503 01:59:38.374408 9119 tablet_copy_client.cc:783] T 2ac8004211e64c3481e75377092fa93b P 9264f24d028347d98f7e4e2d1aedeb79: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:38.374595 9119 tablet_copy_client.cc:647] T 2ac8004211e64c3481e75377092fa93b P 9264f24d028347d98f7e4e2d1aedeb79: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:38.375694 6647 tablet_copy_service.cc:140] P 79163c69f6f04a9abd0208d74e1c61be: Received BeginTabletCopySession request for tablet 0bdcea4e59ec4e2da63b9316f9d718be from peer de2f076398ee44008622fb5db2a75952 ({username='slave'} at 127.0.99.135:58241)
I20240503 01:59:38.375741 9123 tablet_copy_client.cc:1002] T 2ac8004211e64c3481e75377092fa93b P 9264f24d028347d98f7e4e2d1aedeb79: tablet copy: Injecting latency into file download: 1500
I20240503 01:59:38.375785 6647 tablet_copy_service.cc:161] P 79163c69f6f04a9abd0208d74e1c61be: Beginning new tablet copy session on tablet 0bdcea4e59ec4e2da63b9316f9d718be from peer de2f076398ee44008622fb5db2a75952 at {username='slave'} at 127.0.99.135:58241: session id = de2f076398ee44008622fb5db2a75952-0bdcea4e59ec4e2da63b9316f9d718be
I20240503 01:59:38.375981 6778 tablet_copy_service.cc:140] P b6b1d5677b8d47788b7968f64b3dac3c: Received BeginTabletCopySession request for tablet f11a8034585a4d57be0e9b1e5e8bcd85 from peer 9264f24d028347d98f7e4e2d1aedeb79 ({username='slave'} at 127.0.99.133:42687)
I20240503 01:59:38.376094 6778 tablet_copy_service.cc:161] P b6b1d5677b8d47788b7968f64b3dac3c: Beginning new tablet copy session on tablet f11a8034585a4d57be0e9b1e5e8bcd85 from peer 9264f24d028347d98f7e4e2d1aedeb79 at {username='slave'} at 127.0.99.133:42687: session id = 9264f24d028347d98f7e4e2d1aedeb79-f11a8034585a4d57be0e9b1e5e8bcd85
I20240503 01:59:38.376641 6647 tablet_copy_source_session.cc:205] T 0bdcea4e59ec4e2da63b9316f9d718be P 79163c69f6f04a9abd0208d74e1c61be: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:38.377535 6778 tablet_copy_source_session.cc:205] T f11a8034585a4d57be0e9b1e5e8bcd85 P b6b1d5677b8d47788b7968f64b3dac3c: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:38.377584 9141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bdcea4e59ec4e2da63b9316f9d718be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:38.378643 9136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f11a8034585a4d57be0e9b1e5e8bcd85. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:38.379973 9141 tablet_copy_client.cc:783] T 0bdcea4e59ec4e2da63b9316f9d718be P de2f076398ee44008622fb5db2a75952: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:38.380110 9141 tablet_copy_client.cc:647] T 0bdcea4e59ec4e2da63b9316f9d718be P de2f076398ee44008622fb5db2a75952: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:38.380539 9136 tablet_copy_client.cc:783] T f11a8034585a4d57be0e9b1e5e8bcd85 P 9264f24d028347d98f7e4e2d1aedeb79: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:38.380955 9136 tablet_copy_client.cc:647] T f11a8034585a4d57be0e9b1e5e8bcd85 P 9264f24d028347d98f7e4e2d1aedeb79: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:38.382524 9149 tablet_copy_client.cc:1002] T f11a8034585a4d57be0e9b1e5e8bcd85 P 9264f24d028347d98f7e4e2d1aedeb79: tablet copy: Injecting latency into file download: 1500
I20240503 01:59:38.382560 9143 tablet_copy_client.cc:1002] T 0bdcea4e59ec4e2da63b9316f9d718be P de2f076398ee44008622fb5db2a75952: tablet copy: Injecting latency into file download: 1500
I20240503 01:59:38.397584 9166 ts_tablet_manager.cc:902] T 67150a2f5cb147b8b57a8c8040b567e1 P 65d4632bcdd54ba596acdbbe1b5ef8af: Initiating tablet copy from peer 4127249ba42644388d225d614c935a25 (127.0.99.129:35613)
I20240503 01:59:38.398180 9166 tablet_copy_client.cc:308] T 67150a2f5cb147b8b57a8c8040b567e1 P 65d4632bcdd54ba596acdbbe1b5ef8af: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:35613
I20240503 01:59:38.398689 6385 tablet_copy_service.cc:140] P 4127249ba42644388d225d614c935a25: Received BeginTabletCopySession request for tablet 67150a2f5cb147b8b57a8c8040b567e1 from peer 65d4632bcdd54ba596acdbbe1b5ef8af ({username='slave'} at 127.0.99.134:49429)
I20240503 01:59:38.398805 6385 tablet_copy_service.cc:161] P 4127249ba42644388d225d614c935a25: Beginning new tablet copy session on tablet 67150a2f5cb147b8b57a8c8040b567e1 from peer 65d4632bcdd54ba596acdbbe1b5ef8af at {username='slave'} at 127.0.99.134:49429: session id = 65d4632bcdd54ba596acdbbe1b5ef8af-67150a2f5cb147b8b57a8c8040b567e1
I20240503 01:59:38.399844 6385 tablet_copy_source_session.cc:205] T 67150a2f5cb147b8b57a8c8040b567e1 P 4127249ba42644388d225d614c935a25: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:38.400859 9166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67150a2f5cb147b8b57a8c8040b567e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:38.401930 9171 ts_tablet_manager.cc:902] T a9a74e04ce3745b2b479e96a3f3dc86e P 7a1404fe1c1b4b388b821c8261b53104: Initiating tablet copy from peer 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131:44005)
I20240503 01:59:38.402521 9171 tablet_copy_client.cc:308] T a9a74e04ce3745b2b479e96a3f3dc86e P 7a1404fe1c1b4b388b821c8261b53104: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:44005
I20240503 01:59:38.403646 6647 tablet_copy_service.cc:140] P 79163c69f6f04a9abd0208d74e1c61be: Received BeginTabletCopySession request for tablet a9a74e04ce3745b2b479e96a3f3dc86e from peer 7a1404fe1c1b4b388b821c8261b53104 ({username='slave'} at 127.0.99.136:51745)
I20240503 01:59:38.403745 6647 tablet_copy_service.cc:161] P 79163c69f6f04a9abd0208d74e1c61be: Beginning new tablet copy session on tablet a9a74e04ce3745b2b479e96a3f3dc86e from peer 7a1404fe1c1b4b388b821c8261b53104 at {username='slave'} at 127.0.99.136:51745: session id = 7a1404fe1c1b4b388b821c8261b53104-a9a74e04ce3745b2b479e96a3f3dc86e
I20240503 01:59:38.404798 6647 tablet_copy_source_session.cc:205] T a9a74e04ce3745b2b479e96a3f3dc86e P 79163c69f6f04a9abd0208d74e1c61be: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:38.405647 9171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9a74e04ce3745b2b479e96a3f3dc86e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:38.408025 9171 tablet_copy_client.cc:783] T a9a74e04ce3745b2b479e96a3f3dc86e P 7a1404fe1c1b4b388b821c8261b53104: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:38.408241 9171 tablet_copy_client.cc:647] T a9a74e04ce3745b2b479e96a3f3dc86e P 7a1404fe1c1b4b388b821c8261b53104: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:38.410493 9173 tablet_copy_client.cc:1002] T a9a74e04ce3745b2b479e96a3f3dc86e P 7a1404fe1c1b4b388b821c8261b53104: tablet copy: Injecting latency into file download: 1500
I20240503 01:59:38.412729 9176 ts_tablet_manager.cc:902] T 23b3b3d386544e5d9bfae8041cdf01a0 P 9264f24d028347d98f7e4e2d1aedeb79: Initiating tablet copy from peer 4127249ba42644388d225d614c935a25 (127.0.99.129:35613)
I20240503 01:59:38.413321 9176 tablet_copy_client.cc:308] T 23b3b3d386544e5d9bfae8041cdf01a0 P 9264f24d028347d98f7e4e2d1aedeb79: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:35613
I20240503 01:59:38.415055 9166 tablet_copy_client.cc:783] T 67150a2f5cb147b8b57a8c8040b567e1 P 65d4632bcdd54ba596acdbbe1b5ef8af: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:38.415207 9166 tablet_copy_client.cc:647] T 67150a2f5cb147b8b57a8c8040b567e1 P 65d4632bcdd54ba596acdbbe1b5ef8af: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:38.419546 6385 tablet_copy_service.cc:140] P 4127249ba42644388d225d614c935a25: Received BeginTabletCopySession request for tablet 23b3b3d386544e5d9bfae8041cdf01a0 from peer 9264f24d028347d98f7e4e2d1aedeb79 ({username='slave'} at 127.0.99.133:57445)
I20240503 01:59:38.419639 6385 tablet_copy_service.cc:161] P 4127249ba42644388d225d614c935a25: Beginning new tablet copy session on tablet 23b3b3d386544e5d9bfae8041cdf01a0 from peer 9264f24d028347d98f7e4e2d1aedeb79 at {username='slave'} at 127.0.99.133:57445: session id = 9264f24d028347d98f7e4e2d1aedeb79-23b3b3d386544e5d9bfae8041cdf01a0
I20240503 01:59:38.420524 6385 tablet_copy_source_session.cc:205] T 23b3b3d386544e5d9bfae8041cdf01a0 P 4127249ba42644388d225d614c935a25: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:38.421213 9170 tablet_copy_client.cc:1002] T 67150a2f5cb147b8b57a8c8040b567e1 P 65d4632bcdd54ba596acdbbe1b5ef8af: tablet copy: Injecting latency into file download: 1500
I20240503 01:59:38.421362 9176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23b3b3d386544e5d9bfae8041cdf01a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:38.423501 9176 tablet_copy_client.cc:783] T 23b3b3d386544e5d9bfae8041cdf01a0 P 9264f24d028347d98f7e4e2d1aedeb79: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:38.423643 9176 tablet_copy_client.cc:647] T 23b3b3d386544e5d9bfae8041cdf01a0 P 9264f24d028347d98f7e4e2d1aedeb79: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:38.424984 9177 tablet_copy_client.cc:1002] T 23b3b3d386544e5d9bfae8041cdf01a0 P 9264f24d028347d98f7e4e2d1aedeb79: tablet copy: Injecting latency into file download: 1500
I20240503 01:59:38.439105 6476 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 01:59:38.440310 9192 ts_tablet_manager.cc:902] T 21d27cca81e8455e9934e1cc604fc2b2 P 7a1404fe1c1b4b388b821c8261b53104: Initiating tablet copy from peer 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131:44005)
I20240503 01:59:38.440806 6607 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 01:59:38.440817 6344 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 01:59:38.441818 8073 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:38.443284 8466 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:38.443540 9192 tablet_copy_client.cc:308] T 21d27cca81e8455e9934e1cc604fc2b2 P 7a1404fe1c1b4b388b821c8261b53104: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:44005
I20240503 01:59:38.443970 6647 tablet_copy_service.cc:140] P 79163c69f6f04a9abd0208d74e1c61be: Received BeginTabletCopySession request for tablet 21d27cca81e8455e9934e1cc604fc2b2 from peer 7a1404fe1c1b4b388b821c8261b53104 ({username='slave'} at 127.0.99.136:51745)
I20240503 01:59:38.444061 6647 tablet_copy_service.cc:161] P 79163c69f6f04a9abd0208d74e1c61be: Beginning new tablet copy session on tablet 21d27cca81e8455e9934e1cc604fc2b2 from peer 7a1404fe1c1b4b388b821c8261b53104 at {username='slave'} at 127.0.99.136:51745: session id = 7a1404fe1c1b4b388b821c8261b53104-21d27cca81e8455e9934e1cc604fc2b2
I20240503 01:59:38.444989 6647 tablet_copy_source_session.cc:205] T 21d27cca81e8455e9934e1cc604fc2b2 P 79163c69f6f04a9abd0208d74e1c61be: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:38.449060 9192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21d27cca81e8455e9934e1cc604fc2b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:38.452466 9192 tablet_copy_client.cc:783] T 21d27cca81e8455e9934e1cc604fc2b2 P 7a1404fe1c1b4b388b821c8261b53104: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:38.452723 9192 tablet_copy_client.cc:647] T 21d27cca81e8455e9934e1cc604fc2b2 P 7a1404fe1c1b4b388b821c8261b53104: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:38.465358 9196 tablet_copy_client.cc:1002] T 21d27cca81e8455e9934e1cc604fc2b2 P 7a1404fe1c1b4b388b821c8261b53104: tablet copy: Injecting latency into file download: 1500
I20240503 01:59:38.468741 8204 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:38.479678 8335 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:38.479990 6738 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:38.490656 9197 ts_tablet_manager.cc:902] T 120d5dd4f26c42399702141094ce7982 P 7a1404fe1c1b4b388b821c8261b53104: Initiating tablet copy from peer 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130:34189)
I20240503 01:59:38.491238 9197 tablet_copy_client.cc:308] T 120d5dd4f26c42399702141094ce7982 P 7a1404fe1c1b4b388b821c8261b53104: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:34189
I20240503 01:59:38.496511 6516 tablet_copy_service.cc:140] P 0c4821d9898d42799c5789d4d2da3f63: Received BeginTabletCopySession request for tablet 120d5dd4f26c42399702141094ce7982 from peer 7a1404fe1c1b4b388b821c8261b53104 ({username='slave'} at 127.0.99.136:32823)
I20240503 01:59:38.496639 6516 tablet_copy_service.cc:161] P 0c4821d9898d42799c5789d4d2da3f63: Beginning new tablet copy session on tablet 120d5dd4f26c42399702141094ce7982 from peer 7a1404fe1c1b4b388b821c8261b53104 at {username='slave'} at 127.0.99.136:32823: session id = 7a1404fe1c1b4b388b821c8261b53104-120d5dd4f26c42399702141094ce7982
I20240503 01:59:38.497543 6516 tablet_copy_source_session.cc:205] T 120d5dd4f26c42399702141094ce7982 P 0c4821d9898d42799c5789d4d2da3f63: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:38.498378 9197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 120d5dd4f26c42399702141094ce7982. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:38.500602 9197 tablet_copy_client.cc:783] T 120d5dd4f26c42399702141094ce7982 P 7a1404fe1c1b4b388b821c8261b53104: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:38.500770 9197 tablet_copy_client.cc:647] T 120d5dd4f26c42399702141094ce7982 P 7a1404fe1c1b4b388b821c8261b53104: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:38.502009 9198 tablet_copy_client.cc:1002] T 120d5dd4f26c42399702141094ce7982 P 7a1404fe1c1b4b388b821c8261b53104: tablet copy: Injecting latency into file download: 1500
Runtime error: ksck discovered errors
I20240503 01:59:38.518488 9202 ts_tablet_manager.cc:902] T a5926dae08184952bc693342c2c61ebf P de2f076398ee44008622fb5db2a75952: Initiating tablet copy from peer 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131:44005)
I20240503 01:59:38.519165 9202 tablet_copy_client.cc:308] T a5926dae08184952bc693342c2c61ebf P de2f076398ee44008622fb5db2a75952: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:44005
I20240503 01:59:38.519599 6647 tablet_copy_service.cc:140] P 79163c69f6f04a9abd0208d74e1c61be: Received BeginTabletCopySession request for tablet a5926dae08184952bc693342c2c61ebf from peer de2f076398ee44008622fb5db2a75952 ({username='slave'} at 127.0.99.135:58241)
I20240503 01:59:38.519708 6647 tablet_copy_service.cc:161] P 79163c69f6f04a9abd0208d74e1c61be: Beginning new tablet copy session on tablet a5926dae08184952bc693342c2c61ebf from peer de2f076398ee44008622fb5db2a75952 at {username='slave'} at 127.0.99.135:58241: session id = de2f076398ee44008622fb5db2a75952-a5926dae08184952bc693342c2c61ebf
I20240503 01:59:38.520489 6647 tablet_copy_source_session.cc:205] T a5926dae08184952bc693342c2c61ebf P 79163c69f6f04a9abd0208d74e1c61be: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:38.521327 9202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5926dae08184952bc693342c2c61ebf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:38.521910 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task42rE7p/build/debug/bin/kudu
/tmp/dist-test-task42rE7p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-8/wal
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-8/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-8/logs
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-8/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.137:0
--local_ip_for_outbound_sockets=127.0.99.137
--webserver_interface=127.0.99.137
--webserver_port=0
--tserver_master_addrs=127.0.99.190:38597
--builtin_ntp_servers=127.0.99.148:34293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240503 01:59:38.522962 9202 tablet_copy_client.cc:783] T a5926dae08184952bc693342c2c61ebf P de2f076398ee44008622fb5db2a75952: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:38.523152 9202 tablet_copy_client.cc:647] T a5926dae08184952bc693342c2c61ebf P de2f076398ee44008622fb5db2a75952: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:38.524220 9206 tablet_copy_client.cc:1002] T a5926dae08184952bc693342c2c61ebf P de2f076398ee44008622fb5db2a75952: tablet copy: Injecting latency into file download: 1500
I20240503 01:59:38.592491 9223 ts_tablet_manager.cc:902] T 2c8c328220e64e168c74fea7619f0c3b P 65d4632bcdd54ba596acdbbe1b5ef8af: Initiating tablet copy from peer b6b1d5677b8d47788b7968f64b3dac3c (127.0.99.132:46067)
I20240503 01:59:38.593206 9223 tablet_copy_client.cc:308] T 2c8c328220e64e168c74fea7619f0c3b P 65d4632bcdd54ba596acdbbe1b5ef8af: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:46067
I20240503 01:59:38.593691 9228 ts_tablet_manager.cc:902] T e5a00d4ce3a64ca5a7c5bd65c556f429 P de2f076398ee44008622fb5db2a75952: Initiating tablet copy from peer b6b1d5677b8d47788b7968f64b3dac3c (127.0.99.132:46067)
I20240503 01:59:38.594664 9228 tablet_copy_client.cc:308] T e5a00d4ce3a64ca5a7c5bd65c556f429 P de2f076398ee44008622fb5db2a75952: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:46067
I20240503 01:59:38.597949 6778 tablet_copy_service.cc:140] P b6b1d5677b8d47788b7968f64b3dac3c: Received BeginTabletCopySession request for tablet 2c8c328220e64e168c74fea7619f0c3b from peer 65d4632bcdd54ba596acdbbe1b5ef8af ({username='slave'} at 127.0.99.134:47629)
I20240503 01:59:38.598053 6778 tablet_copy_service.cc:161] P b6b1d5677b8d47788b7968f64b3dac3c: Beginning new tablet copy session on tablet 2c8c328220e64e168c74fea7619f0c3b from peer 65d4632bcdd54ba596acdbbe1b5ef8af at {username='slave'} at 127.0.99.134:47629: session id = 65d4632bcdd54ba596acdbbe1b5ef8af-2c8c328220e64e168c74fea7619f0c3b
I20240503 01:59:38.598989 6778 tablet_copy_source_session.cc:205] T 2c8c328220e64e168c74fea7619f0c3b P b6b1d5677b8d47788b7968f64b3dac3c: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:38.599218 6777 tablet_copy_service.cc:140] P b6b1d5677b8d47788b7968f64b3dac3c: Received BeginTabletCopySession request for tablet e5a00d4ce3a64ca5a7c5bd65c556f429 from peer de2f076398ee44008622fb5db2a75952 ({username='slave'} at 127.0.99.135:32853)
I20240503 01:59:38.599318 6777 tablet_copy_service.cc:161] P b6b1d5677b8d47788b7968f64b3dac3c: Beginning new tablet copy session on tablet e5a00d4ce3a64ca5a7c5bd65c556f429 from peer de2f076398ee44008622fb5db2a75952 at {username='slave'} at 127.0.99.135:32853: session id = de2f076398ee44008622fb5db2a75952-e5a00d4ce3a64ca5a7c5bd65c556f429
I20240503 01:59:38.599928 9223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c8c328220e64e168c74fea7619f0c3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:38.600010 6777 tablet_copy_source_session.cc:205] T e5a00d4ce3a64ca5a7c5bd65c556f429 P b6b1d5677b8d47788b7968f64b3dac3c: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:38.600780 9228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5a00d4ce3a64ca5a7c5bd65c556f429. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:38.602878 9223 tablet_copy_client.cc:783] T 2c8c328220e64e168c74fea7619f0c3b P 65d4632bcdd54ba596acdbbe1b5ef8af: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:38.603013 9228 tablet_copy_client.cc:783] T e5a00d4ce3a64ca5a7c5bd65c556f429 P de2f076398ee44008622fb5db2a75952: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:38.603241 9228 tablet_copy_client.cc:647] T e5a00d4ce3a64ca5a7c5bd65c556f429 P de2f076398ee44008622fb5db2a75952: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:38.603307 9223 tablet_copy_client.cc:647] T 2c8c328220e64e168c74fea7619f0c3b P 65d4632bcdd54ba596acdbbe1b5ef8af: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:38.604447 9224 tablet_copy_client.cc:1002] T 2c8c328220e64e168c74fea7619f0c3b P 65d4632bcdd54ba596acdbbe1b5ef8af: tablet copy: Injecting latency into file download: 1500
I20240503 01:59:38.604460 9236 tablet_copy_client.cc:1002] T e5a00d4ce3a64ca5a7c5bd65c556f429 P de2f076398ee44008622fb5db2a75952: tablet copy: Injecting latency into file download: 1500
I20240503 01:59:38.623343 9254 ts_tablet_manager.cc:902] T a388d59f6ba94e91958f55b340343fa6 P 9264f24d028347d98f7e4e2d1aedeb79: Initiating tablet copy from peer 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131:44005)
I20240503 01:59:38.624286 9254 tablet_copy_client.cc:308] T a388d59f6ba94e91958f55b340343fa6 P 9264f24d028347d98f7e4e2d1aedeb79: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:44005
I20240503 01:59:38.630044 6647 tablet_copy_service.cc:140] P 79163c69f6f04a9abd0208d74e1c61be: Received BeginTabletCopySession request for tablet a388d59f6ba94e91958f55b340343fa6 from peer 9264f24d028347d98f7e4e2d1aedeb79 ({username='slave'} at 127.0.99.133:50661)
I20240503 01:59:38.630146 6647 tablet_copy_service.cc:161] P 79163c69f6f04a9abd0208d74e1c61be: Beginning new tablet copy session on tablet a388d59f6ba94e91958f55b340343fa6 from peer 9264f24d028347d98f7e4e2d1aedeb79 at {username='slave'} at 127.0.99.133:50661: session id = 9264f24d028347d98f7e4e2d1aedeb79-a388d59f6ba94e91958f55b340343fa6
I20240503 01:59:38.631027 6647 tablet_copy_source_session.cc:205] T a388d59f6ba94e91958f55b340343fa6 P 79163c69f6f04a9abd0208d74e1c61be: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:38.631959 9254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a388d59f6ba94e91958f55b340343fa6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:38.634626 9254 tablet_copy_client.cc:783] T a388d59f6ba94e91958f55b340343fa6 P 9264f24d028347d98f7e4e2d1aedeb79: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:38.634769 9254 tablet_copy_client.cc:647] T a388d59f6ba94e91958f55b340343fa6 P 9264f24d028347d98f7e4e2d1aedeb79: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:38.636751 9258 tablet_copy_client.cc:1002] T a388d59f6ba94e91958f55b340343fa6 P 9264f24d028347d98f7e4e2d1aedeb79: tablet copy: Injecting latency into file download: 1500
I20240503 01:59:38.638041 9269 ts_tablet_manager.cc:902] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 9264f24d028347d98f7e4e2d1aedeb79: Initiating tablet copy from peer 4127249ba42644388d225d614c935a25 (127.0.99.129:35613)
I20240503 01:59:38.638576 9269 tablet_copy_client.cc:308] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 9264f24d028347d98f7e4e2d1aedeb79: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:35613
I20240503 01:59:38.639286 6385 tablet_copy_service.cc:140] P 4127249ba42644388d225d614c935a25: Received BeginTabletCopySession request for tablet 2088c84e4eef4ccfbc70d7b668b8f2cf from peer 9264f24d028347d98f7e4e2d1aedeb79 ({username='slave'} at 127.0.99.133:57445)
I20240503 01:59:38.639407 6385 tablet_copy_service.cc:161] P 4127249ba42644388d225d614c935a25: Beginning new tablet copy session on tablet 2088c84e4eef4ccfbc70d7b668b8f2cf from peer 9264f24d028347d98f7e4e2d1aedeb79 at {username='slave'} at 127.0.99.133:57445: session id = 9264f24d028347d98f7e4e2d1aedeb79-2088c84e4eef4ccfbc70d7b668b8f2cf
I20240503 01:59:38.640372 6385 tablet_copy_source_session.cc:205] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 4127249ba42644388d225d614c935a25: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:38.641448 9269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2088c84e4eef4ccfbc70d7b668b8f2cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:38.643432 9269 tablet_copy_client.cc:783] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 9264f24d028347d98f7e4e2d1aedeb79: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:38.643641 9269 tablet_copy_client.cc:647] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 9264f24d028347d98f7e4e2d1aedeb79: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:38.644634 9273 tablet_copy_client.cc:1002] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 9264f24d028347d98f7e4e2d1aedeb79: tablet copy: Injecting latency into file download: 1500
W20240503 01:59:38.648113 9207 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:38.648346 9207 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:59:38.648434 9207 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:38.651584 9207 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:59:38.651655 9207 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:59:38.651744 9207 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.137
I20240503 01:59:38.656121 9207 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:34293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-8/data
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-8/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.137:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-8/data/info.pb
--webserver_interface=127.0.99.137
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.0.99.190:38597
--never_fsync=true
--heap_profile_path=/tmp/kudu.9207
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.137
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-8/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:59:38.657184 9207 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:59:38.658447 9207 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:38.668737 9298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:38.669862 9301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:38.670763 9299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:38.671633 9207 server_base.cc:1034] running on GCE node
I20240503 01:59:38.672011 9207 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:59:38.672560 9207 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:59:38.675801 9207 hybrid_clock.cc:648] HybridClock initialized: now 1714701578675777 us; error 33 us; skew 500 ppm
I20240503 01:59:38.677788 9207 webserver.cc:460] Webserver started at http://127.0.99.137:38023/ using document root <none> and password file <none>
I20240503 01:59:38.678483 9207 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:38.678562 9207 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:38.678764 9207 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:59:38.681478 9207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-8/data/instance:
uuid: "2d0fd64aaa6548e09e1ebde581c54658"
format_stamp: "Formatted at 2024-05-03 01:59:38 on dist-test-slave-dtz3"
I20240503 01:59:38.682119 9207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-8/wal/instance:
uuid: "2d0fd64aaa6548e09e1ebde581c54658"
format_stamp: "Formatted at 2024-05-03 01:59:38 on dist-test-slave-dtz3"
I20240503 01:59:38.685961 9207 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240503 01:59:38.688398 9327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:38.689735 9207 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 01:59:38.689848 9207 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-8/data,/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-8/wal
uuid: "2d0fd64aaa6548e09e1ebde581c54658"
format_stamp: "Formatted at 2024-05-03 01:59:38 on dist-test-slave-dtz3"
I20240503 01:59:38.689921 9207 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-8/wal
metadata directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-8/wal
1 data directories: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-8/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:38.705488 9207 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:38.705981 9207 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:59:38.706120 9207 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:38.706643 9207 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:38.707422 9207 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:59:38.707473 9207 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:38.707528 9207 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:59:38.707563 9207 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:38.713297 9411 ts_tablet_manager.cc:902] T 4e7ceb1453804246a41bd1dda0e39d02 P 65d4632bcdd54ba596acdbbe1b5ef8af: Initiating tablet copy from peer 4127249ba42644388d225d614c935a25 (127.0.99.129:35613)
I20240503 01:59:38.713932 9411 tablet_copy_client.cc:308] T 4e7ceb1453804246a41bd1dda0e39d02 P 65d4632bcdd54ba596acdbbe1b5ef8af: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:35613
I20240503 01:59:38.714591 6385 tablet_copy_service.cc:140] P 4127249ba42644388d225d614c935a25: Received BeginTabletCopySession request for tablet 4e7ceb1453804246a41bd1dda0e39d02 from peer 65d4632bcdd54ba596acdbbe1b5ef8af ({username='slave'} at 127.0.99.134:49429)
I20240503 01:59:38.714713 6385 tablet_copy_service.cc:161] P 4127249ba42644388d225d614c935a25: Beginning new tablet copy session on tablet 4e7ceb1453804246a41bd1dda0e39d02 from peer 65d4632bcdd54ba596acdbbe1b5ef8af at {username='slave'} at 127.0.99.134:49429: session id = 65d4632bcdd54ba596acdbbe1b5ef8af-4e7ceb1453804246a41bd1dda0e39d02
I20240503 01:59:38.715956 6385 tablet_copy_source_session.cc:205] T 4e7ceb1453804246a41bd1dda0e39d02 P 4127249ba42644388d225d614c935a25: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:38.716909 9411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e7ceb1453804246a41bd1dda0e39d02. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:38.717123 9207 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.137:40613
I20240503 01:59:38.717183 9460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.137:40613 every 8 connection(s)
I20240503 01:59:38.717937 9207 server_base.cc:1166] Dumped server information to /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-8/data/info.pb
I20240503 01:59:38.718973 9411 tablet_copy_client.cc:783] T 4e7ceb1453804246a41bd1dda0e39d02 P 65d4632bcdd54ba596acdbbe1b5ef8af: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:38.719127 9411 tablet_copy_client.cc:647] T 4e7ceb1453804246a41bd1dda0e39d02 P 65d4632bcdd54ba596acdbbe1b5ef8af: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:38.720247 9425 tablet_copy_client.cc:1002] T 4e7ceb1453804246a41bd1dda0e39d02 P 65d4632bcdd54ba596acdbbe1b5ef8af: tablet copy: Injecting latency into file download: 1500
I20240503 01:59:38.726312 9461 heartbeater.cc:347] Connected to a master server at 127.0.99.190:38597
I20240503 01:59:38.726498 9461 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:38.726936 9461 heartbeater.cc:511] Master 127.0.99.190:38597 requested a full tablet report, sending...
I20240503 01:59:38.727790 6223 ts_manager.cc:194] Registered new tserver with Master: 2d0fd64aaa6548e09e1ebde581c54658 (127.0.99.137:40613)
I20240503 01:59:38.727994 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task42rE7p/build/debug/bin/kudu as pid 9207
I20240503 01:59:38.728133 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-8/wal/instance
I20240503 01:59:38.728515 6223 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.137:45695
I20240503 01:59:38.768313 9466 ts_tablet_manager.cc:902] T a655d914858447b59fe5763e43c0807d P 7a1404fe1c1b4b388b821c8261b53104: Initiating tablet copy from peer 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131:44005)
I20240503 01:59:38.768883 9466 tablet_copy_client.cc:308] T a655d914858447b59fe5763e43c0807d P 7a1404fe1c1b4b388b821c8261b53104: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:44005
I20240503 01:59:38.769485 6647 tablet_copy_service.cc:140] P 79163c69f6f04a9abd0208d74e1c61be: Received BeginTabletCopySession request for tablet a655d914858447b59fe5763e43c0807d from peer 7a1404fe1c1b4b388b821c8261b53104 ({username='slave'} at 127.0.99.136:51745)
I20240503 01:59:38.769588 6647 tablet_copy_service.cc:161] P 79163c69f6f04a9abd0208d74e1c61be: Beginning new tablet copy session on tablet a655d914858447b59fe5763e43c0807d from peer 7a1404fe1c1b4b388b821c8261b53104 at {username='slave'} at 127.0.99.136:51745: session id = 7a1404fe1c1b4b388b821c8261b53104-a655d914858447b59fe5763e43c0807d
I20240503 01:59:38.770295 6647 tablet_copy_source_session.cc:205] T a655d914858447b59fe5763e43c0807d P 79163c69f6f04a9abd0208d74e1c61be: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:38.771109 9466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a655d914858447b59fe5763e43c0807d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:38.772776 9466 tablet_copy_client.cc:783] T a655d914858447b59fe5763e43c0807d P 7a1404fe1c1b4b388b821c8261b53104: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:38.772946 9466 tablet_copy_client.cc:647] T a655d914858447b59fe5763e43c0807d P 7a1404fe1c1b4b388b821c8261b53104: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:38.773958 9470 tablet_copy_client.cc:1002] T a655d914858447b59fe5763e43c0807d P 7a1404fe1c1b4b388b821c8261b53104: tablet copy: Injecting latency into file download: 1500
I20240503 01:59:39.730325 9461 heartbeater.cc:503] Master 127.0.99.190:38597 was elected leader, sending a full tablet report...
I20240503 01:59:39.764994 9020 tablet_copy_client.cc:523] T 470b553976064bdd8c321799ac718274 P 65d4632bcdd54ba596acdbbe1b5ef8af: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:39.766875 9020 tablet_bootstrap.cc:492] T 470b553976064bdd8c321799ac718274 P 65d4632bcdd54ba596acdbbe1b5ef8af: Bootstrap starting.
I20240503 01:59:39.772079 9020 log.cc:824] T 470b553976064bdd8c321799ac718274 P 65d4632bcdd54ba596acdbbe1b5ef8af: Log is configured to *not* fsync() on all Append() calls
I20240503 01:59:39.777729 9033 tablet_copy_client.cc:523] T e5ed3e729bc64ebeac216ce5076b9fd9 P de2f076398ee44008622fb5db2a75952: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:39.779330 9033 tablet_bootstrap.cc:492] T e5ed3e729bc64ebeac216ce5076b9fd9 P de2f076398ee44008622fb5db2a75952: Bootstrap starting.
I20240503 01:59:39.784483 9033 log.cc:824] T e5ed3e729bc64ebeac216ce5076b9fd9 P de2f076398ee44008622fb5db2a75952: Log is configured to *not* fsync() on all Append() calls
I20240503 01:59:39.794059 9020 tablet_bootstrap.cc:492] T 470b553976064bdd8c321799ac718274 P 65d4632bcdd54ba596acdbbe1b5ef8af: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:39.794386 9020 tablet_bootstrap.cc:492] T 470b553976064bdd8c321799ac718274 P 65d4632bcdd54ba596acdbbe1b5ef8af: Bootstrap complete.
I20240503 01:59:39.794714 9020 ts_tablet_manager.cc:1367] T 470b553976064bdd8c321799ac718274 P 65d4632bcdd54ba596acdbbe1b5ef8af: Time spent bootstrapping tablet: real 0.028s user 0.029s sys 0.000s
I20240503 01:59:39.797251 9020 raft_consensus.cc:348] T 470b553976064bdd8c321799ac718274 P 65d4632bcdd54ba596acdbbe1b5ef8af [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: true } }
I20240503 01:59:39.797484 9020 raft_consensus.cc:729] T 470b553976064bdd8c321799ac718274 P 65d4632bcdd54ba596acdbbe1b5ef8af [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 65d4632bcdd54ba596acdbbe1b5ef8af, State: Initialized, Role: LEARNER
I20240503 01:59:39.797811 9020 consensus_queue.cc:260] T 470b553976064bdd8c321799ac718274 P 65d4632bcdd54ba596acdbbe1b5ef8af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: true } }
I20240503 01:59:39.798590 9020 ts_tablet_manager.cc:1398] T 470b553976064bdd8c321799ac718274 P 65d4632bcdd54ba596acdbbe1b5ef8af: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240503 01:59:39.800357 6385 tablet_copy_service.cc:342] P 4127249ba42644388d225d614c935a25: Request end of tablet copy session 65d4632bcdd54ba596acdbbe1b5ef8af-470b553976064bdd8c321799ac718274 received from {username='slave'} at 127.0.99.134:49429
I20240503 01:59:39.800457 6385 tablet_copy_service.cc:433] P 4127249ba42644388d225d614c935a25: ending tablet copy session 65d4632bcdd54ba596acdbbe1b5ef8af-470b553976064bdd8c321799ac718274 on tablet 470b553976064bdd8c321799ac718274 with peer 65d4632bcdd54ba596acdbbe1b5ef8af
I20240503 01:59:39.803210 9059 tablet_copy_client.cc:523] T d6ee8175874c4063986033b0a80ccb82 P 7a1404fe1c1b4b388b821c8261b53104: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:39.805156 9059 tablet_bootstrap.cc:492] T d6ee8175874c4063986033b0a80ccb82 P 7a1404fe1c1b4b388b821c8261b53104: Bootstrap starting.
I20240503 01:59:39.809237 9033 tablet_bootstrap.cc:492] T e5ed3e729bc64ebeac216ce5076b9fd9 P de2f076398ee44008622fb5db2a75952: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:39.809616 9033 tablet_bootstrap.cc:492] T e5ed3e729bc64ebeac216ce5076b9fd9 P de2f076398ee44008622fb5db2a75952: Bootstrap complete.
I20240503 01:59:39.809906 9033 ts_tablet_manager.cc:1367] T e5ed3e729bc64ebeac216ce5076b9fd9 P de2f076398ee44008622fb5db2a75952: Time spent bootstrapping tablet: real 0.031s user 0.030s sys 0.000s
I20240503 01:59:39.811720 9059 log.cc:824] T d6ee8175874c4063986033b0a80ccb82 P 7a1404fe1c1b4b388b821c8261b53104: Log is configured to *not* fsync() on all Append() calls
I20240503 01:59:39.812376 9033 raft_consensus.cc:348] T e5ed3e729bc64ebeac216ce5076b9fd9 P de2f076398ee44008622fb5db2a75952 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: true } }
I20240503 01:59:39.812620 9033 raft_consensus.cc:729] T e5ed3e729bc64ebeac216ce5076b9fd9 P de2f076398ee44008622fb5db2a75952 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: de2f076398ee44008622fb5db2a75952, State: Initialized, Role: LEARNER
I20240503 01:59:39.812983 9033 consensus_queue.cc:260] T e5ed3e729bc64ebeac216ce5076b9fd9 P de2f076398ee44008622fb5db2a75952 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: true } }
I20240503 01:59:39.813871 9033 ts_tablet_manager.cc:1398] T e5ed3e729bc64ebeac216ce5076b9fd9 P de2f076398ee44008622fb5db2a75952: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20240503 01:59:39.814778 6516 tablet_copy_service.cc:342] P 0c4821d9898d42799c5789d4d2da3f63: Request end of tablet copy session de2f076398ee44008622fb5db2a75952-e5ed3e729bc64ebeac216ce5076b9fd9 received from {username='slave'} at 127.0.99.135:55869
I20240503 01:59:39.814900 6516 tablet_copy_service.cc:433] P 0c4821d9898d42799c5789d4d2da3f63: ending tablet copy session de2f076398ee44008622fb5db2a75952-e5ed3e729bc64ebeac216ce5076b9fd9 on tablet e5ed3e729bc64ebeac216ce5076b9fd9 with peer de2f076398ee44008622fb5db2a75952
I20240503 01:59:39.828471 9083 tablet_copy_client.cc:523] T ff8f06c5ad8442ee942bf900212fe858 P 65d4632bcdd54ba596acdbbe1b5ef8af: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:39.830015 9083 tablet_bootstrap.cc:492] T ff8f06c5ad8442ee942bf900212fe858 P 65d4632bcdd54ba596acdbbe1b5ef8af: Bootstrap starting.
I20240503 01:59:39.833868 9059 tablet_bootstrap.cc:492] T d6ee8175874c4063986033b0a80ccb82 P 7a1404fe1c1b4b388b821c8261b53104: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:39.834209 9059 tablet_bootstrap.cc:492] T d6ee8175874c4063986033b0a80ccb82 P 7a1404fe1c1b4b388b821c8261b53104: Bootstrap complete.
I20240503 01:59:39.834544 9059 ts_tablet_manager.cc:1367] T d6ee8175874c4063986033b0a80ccb82 P 7a1404fe1c1b4b388b821c8261b53104: Time spent bootstrapping tablet: real 0.030s user 0.027s sys 0.000s
I20240503 01:59:39.837157 9059 raft_consensus.cc:348] T d6ee8175874c4063986033b0a80ccb82 P 7a1404fe1c1b4b388b821c8261b53104 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: true } }
I20240503 01:59:39.837426 9059 raft_consensus.cc:729] T d6ee8175874c4063986033b0a80ccb82 P 7a1404fe1c1b4b388b821c8261b53104 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7a1404fe1c1b4b388b821c8261b53104, State: Initialized, Role: LEARNER
I20240503 01:59:39.837764 9059 consensus_queue.cc:260] T d6ee8175874c4063986033b0a80ccb82 P 7a1404fe1c1b4b388b821c8261b53104 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: true } }
I20240503 01:59:39.838611 9059 ts_tablet_manager.cc:1398] T d6ee8175874c4063986033b0a80ccb82 P 7a1404fe1c1b4b388b821c8261b53104: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240503 01:59:39.839329 6647 tablet_copy_service.cc:342] P 79163c69f6f04a9abd0208d74e1c61be: Request end of tablet copy session 7a1404fe1c1b4b388b821c8261b53104-d6ee8175874c4063986033b0a80ccb82 received from {username='slave'} at 127.0.99.136:51745
I20240503 01:59:39.839434 6647 tablet_copy_service.cc:433] P 79163c69f6f04a9abd0208d74e1c61be: ending tablet copy session 7a1404fe1c1b4b388b821c8261b53104-d6ee8175874c4063986033b0a80ccb82 on tablet d6ee8175874c4063986033b0a80ccb82 with peer 7a1404fe1c1b4b388b821c8261b53104
I20240503 01:59:39.852874 9083 tablet_bootstrap.cc:492] T ff8f06c5ad8442ee942bf900212fe858 P 65d4632bcdd54ba596acdbbe1b5ef8af: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:39.853197 9083 tablet_bootstrap.cc:492] T ff8f06c5ad8442ee942bf900212fe858 P 65d4632bcdd54ba596acdbbe1b5ef8af: Bootstrap complete.
I20240503 01:59:39.853401 9083 ts_tablet_manager.cc:1367] T ff8f06c5ad8442ee942bf900212fe858 P 65d4632bcdd54ba596acdbbe1b5ef8af: Time spent bootstrapping tablet: real 0.023s user 0.022s sys 0.002s
I20240503 01:59:39.853960 9083 raft_consensus.cc:348] T ff8f06c5ad8442ee942bf900212fe858 P 65d4632bcdd54ba596acdbbe1b5ef8af [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: true } }
I20240503 01:59:39.854068 9083 raft_consensus.cc:729] T ff8f06c5ad8442ee942bf900212fe858 P 65d4632bcdd54ba596acdbbe1b5ef8af [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 65d4632bcdd54ba596acdbbe1b5ef8af, State: Initialized, Role: LEARNER
I20240503 01:59:39.854175 9083 consensus_queue.cc:260] T ff8f06c5ad8442ee942bf900212fe858 P 65d4632bcdd54ba596acdbbe1b5ef8af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: true } }
I20240503 01:59:39.854423 9083 ts_tablet_manager.cc:1398] T ff8f06c5ad8442ee942bf900212fe858 P 65d4632bcdd54ba596acdbbe1b5ef8af: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:39.854946 6516 tablet_copy_service.cc:342] P 0c4821d9898d42799c5789d4d2da3f63: Request end of tablet copy session 65d4632bcdd54ba596acdbbe1b5ef8af-ff8f06c5ad8442ee942bf900212fe858 received from {username='slave'} at 127.0.99.134:55903
I20240503 01:59:39.855028 6516 tablet_copy_service.cc:433] P 0c4821d9898d42799c5789d4d2da3f63: ending tablet copy session 65d4632bcdd54ba596acdbbe1b5ef8af-ff8f06c5ad8442ee942bf900212fe858 on tablet ff8f06c5ad8442ee942bf900212fe858 with peer 65d4632bcdd54ba596acdbbe1b5ef8af
I20240503 01:59:39.863131 8355 raft_consensus.cc:1203] T e5ed3e729bc64ebeac216ce5076b9fd9 P de2f076398ee44008622fb5db2a75952 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 01:59:39.873703 9126 tablet_copy_client.cc:523] T 1d392e08c15a4d4098a350287b3163ef P de2f076398ee44008622fb5db2a75952: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:39.875458 9126 tablet_bootstrap.cc:492] T 1d392e08c15a4d4098a350287b3163ef P de2f076398ee44008622fb5db2a75952: Bootstrap starting.
I20240503 01:59:39.876806 9119 tablet_copy_client.cc:523] T 2ac8004211e64c3481e75377092fa93b P 9264f24d028347d98f7e4e2d1aedeb79: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:39.878726 9119 tablet_bootstrap.cc:492] T 2ac8004211e64c3481e75377092fa93b P 9264f24d028347d98f7e4e2d1aedeb79: Bootstrap starting.
I20240503 01:59:39.882788 9141 tablet_copy_client.cc:523] T 0bdcea4e59ec4e2da63b9316f9d718be P de2f076398ee44008622fb5db2a75952: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:39.882822 9136 tablet_copy_client.cc:523] T f11a8034585a4d57be0e9b1e5e8bcd85 P 9264f24d028347d98f7e4e2d1aedeb79: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:39.884023 9141 tablet_bootstrap.cc:492] T 0bdcea4e59ec4e2da63b9316f9d718be P de2f076398ee44008622fb5db2a75952: Bootstrap starting.
I20240503 01:59:39.884159 9136 tablet_bootstrap.cc:492] T f11a8034585a4d57be0e9b1e5e8bcd85 P 9264f24d028347d98f7e4e2d1aedeb79: Bootstrap starting.
I20240503 01:59:39.888278 9136 log.cc:824] T f11a8034585a4d57be0e9b1e5e8bcd85 P 9264f24d028347d98f7e4e2d1aedeb79: Log is configured to *not* fsync() on all Append() calls
I20240503 01:59:39.895793 8486 raft_consensus.cc:1203] T d6ee8175874c4063986033b0a80ccb82 P 7a1404fe1c1b4b388b821c8261b53104 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 01:59:39.902334 9126 tablet_bootstrap.cc:492] T 1d392e08c15a4d4098a350287b3163ef P de2f076398ee44008622fb5db2a75952: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:39.902640 9126 tablet_bootstrap.cc:492] T 1d392e08c15a4d4098a350287b3163ef P de2f076398ee44008622fb5db2a75952: Bootstrap complete.
I20240503 01:59:39.902801 9126 ts_tablet_manager.cc:1367] T 1d392e08c15a4d4098a350287b3163ef P de2f076398ee44008622fb5db2a75952: Time spent bootstrapping tablet: real 0.027s user 0.026s sys 0.000s
I20240503 01:59:39.903370 9126 raft_consensus.cc:348] T 1d392e08c15a4d4098a350287b3163ef P de2f076398ee44008622fb5db2a75952 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: true } }
I20240503 01:59:39.903497 9126 raft_consensus.cc:729] T 1d392e08c15a4d4098a350287b3163ef P de2f076398ee44008622fb5db2a75952 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: de2f076398ee44008622fb5db2a75952, State: Initialized, Role: LEARNER
I20240503 01:59:39.903592 9126 consensus_queue.cc:260] T 1d392e08c15a4d4098a350287b3163ef P de2f076398ee44008622fb5db2a75952 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: true } }
I20240503 01:59:39.903918 9126 ts_tablet_manager.cc:1398] T 1d392e08c15a4d4098a350287b3163ef P de2f076398ee44008622fb5db2a75952: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 01:59:39.904417 6516 tablet_copy_service.cc:342] P 0c4821d9898d42799c5789d4d2da3f63: Request end of tablet copy session de2f076398ee44008622fb5db2a75952-1d392e08c15a4d4098a350287b3163ef received from {username='slave'} at 127.0.99.135:55869
I20240503 01:59:39.904542 6516 tablet_copy_service.cc:433] P 0c4821d9898d42799c5789d4d2da3f63: ending tablet copy session de2f076398ee44008622fb5db2a75952-1d392e08c15a4d4098a350287b3163ef on tablet 1d392e08c15a4d4098a350287b3163ef with peer de2f076398ee44008622fb5db2a75952
I20240503 01:59:39.910852 9171 tablet_copy_client.cc:523] T a9a74e04ce3745b2b479e96a3f3dc86e P 7a1404fe1c1b4b388b821c8261b53104: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:39.912179 9171 tablet_bootstrap.cc:492] T a9a74e04ce3745b2b479e96a3f3dc86e P 7a1404fe1c1b4b388b821c8261b53104: Bootstrap starting.
I20240503 01:59:39.921649 9166 tablet_copy_client.cc:523] T 67150a2f5cb147b8b57a8c8040b567e1 P 65d4632bcdd54ba596acdbbe1b5ef8af: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:39.921893 9136 tablet_bootstrap.cc:492] T f11a8034585a4d57be0e9b1e5e8bcd85 P 9264f24d028347d98f7e4e2d1aedeb79: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:39.924532 9166 tablet_bootstrap.cc:492] T 67150a2f5cb147b8b57a8c8040b567e1 P 65d4632bcdd54ba596acdbbe1b5ef8af: Bootstrap starting.
I20240503 01:59:39.925215 9141 tablet_bootstrap.cc:492] T 0bdcea4e59ec4e2da63b9316f9d718be P de2f076398ee44008622fb5db2a75952: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:39.925504 9141 tablet_bootstrap.cc:492] T 0bdcea4e59ec4e2da63b9316f9d718be P de2f076398ee44008622fb5db2a75952: Bootstrap complete.
I20240503 01:59:39.925665 9141 ts_tablet_manager.cc:1367] T 0bdcea4e59ec4e2da63b9316f9d718be P de2f076398ee44008622fb5db2a75952: Time spent bootstrapping tablet: real 0.042s user 0.024s sys 0.003s
I20240503 01:59:39.925932 9119 tablet_bootstrap.cc:492] T 2ac8004211e64c3481e75377092fa93b P 9264f24d028347d98f7e4e2d1aedeb79: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:39.926108 9141 raft_consensus.cc:348] T 0bdcea4e59ec4e2da63b9316f9d718be P de2f076398ee44008622fb5db2a75952 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: true } }
I20240503 01:59:39.926232 9119 tablet_bootstrap.cc:492] T 2ac8004211e64c3481e75377092fa93b P 9264f24d028347d98f7e4e2d1aedeb79: Bootstrap complete.
I20240503 01:59:39.926244 9141 raft_consensus.cc:729] T 0bdcea4e59ec4e2da63b9316f9d718be P de2f076398ee44008622fb5db2a75952 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: de2f076398ee44008622fb5db2a75952, State: Initialized, Role: LEARNER
I20240503 01:59:39.926337 9141 consensus_queue.cc:260] T 0bdcea4e59ec4e2da63b9316f9d718be P de2f076398ee44008622fb5db2a75952 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: true } }
I20240503 01:59:39.926613 9119 ts_tablet_manager.cc:1367] T 2ac8004211e64c3481e75377092fa93b P 9264f24d028347d98f7e4e2d1aedeb79: Time spent bootstrapping tablet: real 0.048s user 0.027s sys 0.002s
I20240503 01:59:39.927271 9141 ts_tablet_manager.cc:1398] T 0bdcea4e59ec4e2da63b9316f9d718be P de2f076398ee44008622fb5db2a75952: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:39.927682 6647 tablet_copy_service.cc:342] P 79163c69f6f04a9abd0208d74e1c61be: Request end of tablet copy session de2f076398ee44008622fb5db2a75952-0bdcea4e59ec4e2da63b9316f9d718be received from {username='slave'} at 127.0.99.135:58241
I20240503 01:59:39.927768 6647 tablet_copy_service.cc:433] P 79163c69f6f04a9abd0208d74e1c61be: ending tablet copy session de2f076398ee44008622fb5db2a75952-0bdcea4e59ec4e2da63b9316f9d718be on tablet 0bdcea4e59ec4e2da63b9316f9d718be with peer de2f076398ee44008622fb5db2a75952
I20240503 01:59:39.928094 9176 tablet_copy_client.cc:523] T 23b3b3d386544e5d9bfae8041cdf01a0 P 9264f24d028347d98f7e4e2d1aedeb79: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:39.929505 9176 tablet_bootstrap.cc:492] T 23b3b3d386544e5d9bfae8041cdf01a0 P 9264f24d028347d98f7e4e2d1aedeb79: Bootstrap starting.
I20240503 01:59:39.930563 9119 raft_consensus.cc:348] T 2ac8004211e64c3481e75377092fa93b P 9264f24d028347d98f7e4e2d1aedeb79 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: true } }
I20240503 01:59:39.930750 9119 raft_consensus.cc:729] T 2ac8004211e64c3481e75377092fa93b P 9264f24d028347d98f7e4e2d1aedeb79 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9264f24d028347d98f7e4e2d1aedeb79, State: Initialized, Role: LEARNER
I20240503 01:59:39.931005 9119 consensus_queue.cc:260] T 2ac8004211e64c3481e75377092fa93b P 9264f24d028347d98f7e4e2d1aedeb79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: true } }
I20240503 01:59:39.931788 9119 ts_tablet_manager.cc:1398] T 2ac8004211e64c3481e75377092fa93b P 9264f24d028347d98f7e4e2d1aedeb79: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240503 01:59:39.932317 6516 tablet_copy_service.cc:342] P 0c4821d9898d42799c5789d4d2da3f63: Request end of tablet copy session 9264f24d028347d98f7e4e2d1aedeb79-2ac8004211e64c3481e75377092fa93b received from {username='slave'} at 127.0.99.133:35409
I20240503 01:59:39.932404 6516 tablet_copy_service.cc:433] P 0c4821d9898d42799c5789d4d2da3f63: ending tablet copy session 9264f24d028347d98f7e4e2d1aedeb79-2ac8004211e64c3481e75377092fa93b on tablet 2ac8004211e64c3481e75377092fa93b with peer 9264f24d028347d98f7e4e2d1aedeb79
I20240503 01:59:39.937474 9136 tablet_bootstrap.cc:492] T f11a8034585a4d57be0e9b1e5e8bcd85 P 9264f24d028347d98f7e4e2d1aedeb79: Bootstrap complete.
I20240503 01:59:39.937654 9136 ts_tablet_manager.cc:1367] T f11a8034585a4d57be0e9b1e5e8bcd85 P 9264f24d028347d98f7e4e2d1aedeb79: Time spent bootstrapping tablet: real 0.054s user 0.024s sys 0.004s
I20240503 01:59:39.938189 9136 raft_consensus.cc:348] T f11a8034585a4d57be0e9b1e5e8bcd85 P 9264f24d028347d98f7e4e2d1aedeb79 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: true } }
I20240503 01:59:39.938306 9136 raft_consensus.cc:729] T f11a8034585a4d57be0e9b1e5e8bcd85 P 9264f24d028347d98f7e4e2d1aedeb79 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9264f24d028347d98f7e4e2d1aedeb79, State: Initialized, Role: LEARNER
I20240503 01:59:39.938403 9136 consensus_queue.cc:260] T f11a8034585a4d57be0e9b1e5e8bcd85 P 9264f24d028347d98f7e4e2d1aedeb79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: true } }
I20240503 01:59:39.940016 9136 ts_tablet_manager.cc:1398] T f11a8034585a4d57be0e9b1e5e8bcd85 P 9264f24d028347d98f7e4e2d1aedeb79: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:39.941718 6778 tablet_copy_service.cc:342] P b6b1d5677b8d47788b7968f64b3dac3c: Request end of tablet copy session 9264f24d028347d98f7e4e2d1aedeb79-f11a8034585a4d57be0e9b1e5e8bcd85 received from {username='slave'} at 127.0.99.133:42687
I20240503 01:59:39.941838 6778 tablet_copy_service.cc:433] P b6b1d5677b8d47788b7968f64b3dac3c: ending tablet copy session 9264f24d028347d98f7e4e2d1aedeb79-f11a8034585a4d57be0e9b1e5e8bcd85 on tablet f11a8034585a4d57be0e9b1e5e8bcd85 with peer 9264f24d028347d98f7e4e2d1aedeb79
I20240503 01:59:39.952750 9171 tablet_bootstrap.cc:492] T a9a74e04ce3745b2b479e96a3f3dc86e P 7a1404fe1c1b4b388b821c8261b53104: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:39.953147 9171 tablet_bootstrap.cc:492] T a9a74e04ce3745b2b479e96a3f3dc86e P 7a1404fe1c1b4b388b821c8261b53104: Bootstrap complete.
I20240503 01:59:39.953367 9171 ts_tablet_manager.cc:1367] T a9a74e04ce3745b2b479e96a3f3dc86e P 7a1404fe1c1b4b388b821c8261b53104: Time spent bootstrapping tablet: real 0.041s user 0.022s sys 0.005s
I20240503 01:59:39.953953 9171 raft_consensus.cc:348] T a9a74e04ce3745b2b479e96a3f3dc86e P 7a1404fe1c1b4b388b821c8261b53104 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: true } }
I20240503 01:59:39.954135 9171 raft_consensus.cc:729] T a9a74e04ce3745b2b479e96a3f3dc86e P 7a1404fe1c1b4b388b821c8261b53104 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7a1404fe1c1b4b388b821c8261b53104, State: Initialized, Role: LEARNER
I20240503 01:59:39.954267 9171 consensus_queue.cc:260] T a9a74e04ce3745b2b479e96a3f3dc86e P 7a1404fe1c1b4b388b821c8261b53104 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: true } }
I20240503 01:59:39.954572 9171 ts_tablet_manager.cc:1398] T a9a74e04ce3745b2b479e96a3f3dc86e P 7a1404fe1c1b4b388b821c8261b53104: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:39.955200 6647 tablet_copy_service.cc:342] P 79163c69f6f04a9abd0208d74e1c61be: Request end of tablet copy session 7a1404fe1c1b4b388b821c8261b53104-a9a74e04ce3745b2b479e96a3f3dc86e received from {username='slave'} at 127.0.99.136:51745
I20240503 01:59:39.955277 6647 tablet_copy_service.cc:433] P 79163c69f6f04a9abd0208d74e1c61be: ending tablet copy session 7a1404fe1c1b4b388b821c8261b53104-a9a74e04ce3745b2b479e96a3f3dc86e on tablet a9a74e04ce3745b2b479e96a3f3dc86e with peer 7a1404fe1c1b4b388b821c8261b53104
I20240503 01:59:39.958662 9176 tablet_bootstrap.cc:492] T 23b3b3d386544e5d9bfae8041cdf01a0 P 9264f24d028347d98f7e4e2d1aedeb79: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:39.958956 9176 tablet_bootstrap.cc:492] T 23b3b3d386544e5d9bfae8041cdf01a0 P 9264f24d028347d98f7e4e2d1aedeb79: Bootstrap complete.
I20240503 01:59:39.959124 9176 ts_tablet_manager.cc:1367] T 23b3b3d386544e5d9bfae8041cdf01a0 P 9264f24d028347d98f7e4e2d1aedeb79: Time spent bootstrapping tablet: real 0.030s user 0.023s sys 0.000s
I20240503 01:59:39.959481 9176 raft_consensus.cc:348] T 23b3b3d386544e5d9bfae8041cdf01a0 P 9264f24d028347d98f7e4e2d1aedeb79 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: true } }
I20240503 01:59:39.959583 9176 raft_consensus.cc:729] T 23b3b3d386544e5d9bfae8041cdf01a0 P 9264f24d028347d98f7e4e2d1aedeb79 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9264f24d028347d98f7e4e2d1aedeb79, State: Initialized, Role: LEARNER
I20240503 01:59:39.959707 9176 consensus_queue.cc:260] T 23b3b3d386544e5d9bfae8041cdf01a0 P 9264f24d028347d98f7e4e2d1aedeb79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: true } }
I20240503 01:59:39.960044 9176 ts_tablet_manager.cc:1398] T 23b3b3d386544e5d9bfae8041cdf01a0 P 9264f24d028347d98f7e4e2d1aedeb79: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240503 01:59:39.960500 6385 tablet_copy_service.cc:342] P 4127249ba42644388d225d614c935a25: Request end of tablet copy session 9264f24d028347d98f7e4e2d1aedeb79-23b3b3d386544e5d9bfae8041cdf01a0 received from {username='slave'} at 127.0.99.133:57445
I20240503 01:59:39.960593 6385 tablet_copy_service.cc:433] P 4127249ba42644388d225d614c935a25: ending tablet copy session 9264f24d028347d98f7e4e2d1aedeb79-23b3b3d386544e5d9bfae8041cdf01a0 on tablet 23b3b3d386544e5d9bfae8041cdf01a0 with peer 9264f24d028347d98f7e4e2d1aedeb79
I20240503 01:59:39.960683 9166 tablet_bootstrap.cc:492] T 67150a2f5cb147b8b57a8c8040b567e1 P 65d4632bcdd54ba596acdbbe1b5ef8af: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:39.960958 9166 tablet_bootstrap.cc:492] T 67150a2f5cb147b8b57a8c8040b567e1 P 65d4632bcdd54ba596acdbbe1b5ef8af: Bootstrap complete.
I20240503 01:59:39.961093 9166 ts_tablet_manager.cc:1367] T 67150a2f5cb147b8b57a8c8040b567e1 P 65d4632bcdd54ba596acdbbe1b5ef8af: Time spent bootstrapping tablet: real 0.037s user 0.028s sys 0.000s
I20240503 01:59:39.961566 9166 raft_consensus.cc:348] T 67150a2f5cb147b8b57a8c8040b567e1 P 65d4632bcdd54ba596acdbbe1b5ef8af [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: true } }
I20240503 01:59:39.961681 9166 raft_consensus.cc:729] T 67150a2f5cb147b8b57a8c8040b567e1 P 65d4632bcdd54ba596acdbbe1b5ef8af [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 65d4632bcdd54ba596acdbbe1b5ef8af, State: Initialized, Role: LEARNER
I20240503 01:59:39.961782 9166 consensus_queue.cc:260] T 67150a2f5cb147b8b57a8c8040b567e1 P 65d4632bcdd54ba596acdbbe1b5ef8af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: true } }
I20240503 01:59:39.962024 9166 ts_tablet_manager.cc:1398] T 67150a2f5cb147b8b57a8c8040b567e1 P 65d4632bcdd54ba596acdbbe1b5ef8af: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:39.962889 6385 tablet_copy_service.cc:342] P 4127249ba42644388d225d614c935a25: Request end of tablet copy session 65d4632bcdd54ba596acdbbe1b5ef8af-67150a2f5cb147b8b57a8c8040b567e1 received from {username='slave'} at 127.0.99.134:49429
I20240503 01:59:39.962965 6385 tablet_copy_service.cc:433] P 4127249ba42644388d225d614c935a25: ending tablet copy session 65d4632bcdd54ba596acdbbe1b5ef8af-67150a2f5cb147b8b57a8c8040b567e1 on tablet 67150a2f5cb147b8b57a8c8040b567e1 with peer 65d4632bcdd54ba596acdbbe1b5ef8af
I20240503 01:59:39.965673 9192 tablet_copy_client.cc:523] T 21d27cca81e8455e9934e1cc604fc2b2 P 7a1404fe1c1b4b388b821c8261b53104: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:39.967356 9192 tablet_bootstrap.cc:492] T 21d27cca81e8455e9934e1cc604fc2b2 P 7a1404fe1c1b4b388b821c8261b53104: Bootstrap starting.
I20240503 01:59:39.991247 9192 tablet_bootstrap.cc:492] T 21d27cca81e8455e9934e1cc604fc2b2 P 7a1404fe1c1b4b388b821c8261b53104: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:39.991546 9192 tablet_bootstrap.cc:492] T 21d27cca81e8455e9934e1cc604fc2b2 P 7a1404fe1c1b4b388b821c8261b53104: Bootstrap complete.
I20240503 01:59:39.991734 9192 ts_tablet_manager.cc:1367] T 21d27cca81e8455e9934e1cc604fc2b2 P 7a1404fe1c1b4b388b821c8261b53104: Time spent bootstrapping tablet: real 0.024s user 0.019s sys 0.002s
I20240503 01:59:39.992126 9192 raft_consensus.cc:348] T 21d27cca81e8455e9934e1cc604fc2b2 P 7a1404fe1c1b4b388b821c8261b53104 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: true } }
I20240503 01:59:39.992231 9192 raft_consensus.cc:729] T 21d27cca81e8455e9934e1cc604fc2b2 P 7a1404fe1c1b4b388b821c8261b53104 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7a1404fe1c1b4b388b821c8261b53104, State: Initialized, Role: LEARNER
I20240503 01:59:39.992336 9192 consensus_queue.cc:260] T 21d27cca81e8455e9934e1cc604fc2b2 P 7a1404fe1c1b4b388b821c8261b53104 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: true } }
I20240503 01:59:39.992628 9192 ts_tablet_manager.cc:1398] T 21d27cca81e8455e9934e1cc604fc2b2 P 7a1404fe1c1b4b388b821c8261b53104: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:39.993234 6647 tablet_copy_service.cc:342] P 79163c69f6f04a9abd0208d74e1c61be: Request end of tablet copy session 7a1404fe1c1b4b388b821c8261b53104-21d27cca81e8455e9934e1cc604fc2b2 received from {username='slave'} at 127.0.99.136:51745
I20240503 01:59:39.993307 6647 tablet_copy_service.cc:433] P 79163c69f6f04a9abd0208d74e1c61be: ending tablet copy session 7a1404fe1c1b4b388b821c8261b53104-21d27cca81e8455e9934e1cc604fc2b2 on tablet 21d27cca81e8455e9934e1cc604fc2b2 with peer 7a1404fe1c1b4b388b821c8261b53104
I20240503 01:59:39.994582 8486 raft_consensus.cc:1203] T 21d27cca81e8455e9934e1cc604fc2b2 P 7a1404fe1c1b4b388b821c8261b53104 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 01:59:40.002331 9197 tablet_copy_client.cc:523] T 120d5dd4f26c42399702141094ce7982 P 7a1404fe1c1b4b388b821c8261b53104: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:40.003762 9197 tablet_bootstrap.cc:492] T 120d5dd4f26c42399702141094ce7982 P 7a1404fe1c1b4b388b821c8261b53104: Bootstrap starting.
I20240503 01:59:40.024547 9202 tablet_copy_client.cc:523] T a5926dae08184952bc693342c2c61ebf P de2f076398ee44008622fb5db2a75952: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:40.026031 9202 tablet_bootstrap.cc:492] T a5926dae08184952bc693342c2c61ebf P de2f076398ee44008622fb5db2a75952: Bootstrap starting.
I20240503 01:59:40.026484 9197 tablet_bootstrap.cc:492] T 120d5dd4f26c42399702141094ce7982 P 7a1404fe1c1b4b388b821c8261b53104: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:40.026746 9197 tablet_bootstrap.cc:492] T 120d5dd4f26c42399702141094ce7982 P 7a1404fe1c1b4b388b821c8261b53104: Bootstrap complete.
I20240503 01:59:40.026903 9197 ts_tablet_manager.cc:1367] T 120d5dd4f26c42399702141094ce7982 P 7a1404fe1c1b4b388b821c8261b53104: Time spent bootstrapping tablet: real 0.023s user 0.021s sys 0.000s
I20240503 01:59:40.027649 9197 raft_consensus.cc:348] T 120d5dd4f26c42399702141094ce7982 P 7a1404fe1c1b4b388b821c8261b53104 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: true } }
I20240503 01:59:40.027777 9197 raft_consensus.cc:729] T 120d5dd4f26c42399702141094ce7982 P 7a1404fe1c1b4b388b821c8261b53104 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7a1404fe1c1b4b388b821c8261b53104, State: Initialized, Role: LEARNER
I20240503 01:59:40.027868 9197 consensus_queue.cc:260] T 120d5dd4f26c42399702141094ce7982 P 7a1404fe1c1b4b388b821c8261b53104 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: true } }
I20240503 01:59:40.028208 9197 ts_tablet_manager.cc:1398] T 120d5dd4f26c42399702141094ce7982 P 7a1404fe1c1b4b388b821c8261b53104: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240503 01:59:40.029358 6516 tablet_copy_service.cc:342] P 0c4821d9898d42799c5789d4d2da3f63: Request end of tablet copy session 7a1404fe1c1b4b388b821c8261b53104-120d5dd4f26c42399702141094ce7982 received from {username='slave'} at 127.0.99.136:32823
I20240503 01:59:40.029443 6516 tablet_copy_service.cc:433] P 0c4821d9898d42799c5789d4d2da3f63: ending tablet copy session 7a1404fe1c1b4b388b821c8261b53104-120d5dd4f26c42399702141094ce7982 on tablet 120d5dd4f26c42399702141094ce7982 with peer 7a1404fe1c1b4b388b821c8261b53104
I20240503 01:59:40.047423 9202 tablet_bootstrap.cc:492] T a5926dae08184952bc693342c2c61ebf P de2f076398ee44008622fb5db2a75952: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:40.047715 9202 tablet_bootstrap.cc:492] T a5926dae08184952bc693342c2c61ebf P de2f076398ee44008622fb5db2a75952: Bootstrap complete.
I20240503 01:59:40.047881 9202 ts_tablet_manager.cc:1367] T a5926dae08184952bc693342c2c61ebf P de2f076398ee44008622fb5db2a75952: Time spent bootstrapping tablet: real 0.022s user 0.018s sys 0.000s
I20240503 01:59:40.048274 9202 raft_consensus.cc:348] T a5926dae08184952bc693342c2c61ebf P de2f076398ee44008622fb5db2a75952 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: true } }
I20240503 01:59:40.048383 9202 raft_consensus.cc:729] T a5926dae08184952bc693342c2c61ebf P de2f076398ee44008622fb5db2a75952 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: de2f076398ee44008622fb5db2a75952, State: Initialized, Role: LEARNER
I20240503 01:59:40.048460 9202 consensus_queue.cc:260] T a5926dae08184952bc693342c2c61ebf P de2f076398ee44008622fb5db2a75952 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: true } }
I20240503 01:59:40.048755 9202 ts_tablet_manager.cc:1398] T a5926dae08184952bc693342c2c61ebf P de2f076398ee44008622fb5db2a75952: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240503 01:59:40.049254 6647 tablet_copy_service.cc:342] P 79163c69f6f04a9abd0208d74e1c61be: Request end of tablet copy session de2f076398ee44008622fb5db2a75952-a5926dae08184952bc693342c2c61ebf received from {username='slave'} at 127.0.99.135:58241
I20240503 01:59:40.049340 6647 tablet_copy_service.cc:433] P 79163c69f6f04a9abd0208d74e1c61be: ending tablet copy session de2f076398ee44008622fb5db2a75952-a5926dae08184952bc693342c2c61ebf on tablet a5926dae08184952bc693342c2c61ebf with peer de2f076398ee44008622fb5db2a75952
I20240503 01:59:40.104795 9228 tablet_copy_client.cc:523] T e5a00d4ce3a64ca5a7c5bd65c556f429 P de2f076398ee44008622fb5db2a75952: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:40.104826 9223 tablet_copy_client.cc:523] T 2c8c328220e64e168c74fea7619f0c3b P 65d4632bcdd54ba596acdbbe1b5ef8af: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:40.106150 9223 tablet_bootstrap.cc:492] T 2c8c328220e64e168c74fea7619f0c3b P 65d4632bcdd54ba596acdbbe1b5ef8af: Bootstrap starting.
I20240503 01:59:40.106155 9228 tablet_bootstrap.cc:492] T e5a00d4ce3a64ca5a7c5bd65c556f429 P de2f076398ee44008622fb5db2a75952: Bootstrap starting.
I20240503 01:59:40.118846 8080 raft_consensus.cc:1203] T f11a8034585a4d57be0e9b1e5e8bcd85 P 9264f24d028347d98f7e4e2d1aedeb79 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 01:59:40.130679 9228 tablet_bootstrap.cc:492] T e5a00d4ce3a64ca5a7c5bd65c556f429 P de2f076398ee44008622fb5db2a75952: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:40.130995 9228 tablet_bootstrap.cc:492] T e5a00d4ce3a64ca5a7c5bd65c556f429 P de2f076398ee44008622fb5db2a75952: Bootstrap complete.
I20240503 01:59:40.131163 9228 ts_tablet_manager.cc:1367] T e5a00d4ce3a64ca5a7c5bd65c556f429 P de2f076398ee44008622fb5db2a75952: Time spent bootstrapping tablet: real 0.025s user 0.024s sys 0.000s
I20240503 01:59:40.131614 9228 raft_consensus.cc:348] T e5a00d4ce3a64ca5a7c5bd65c556f429 P de2f076398ee44008622fb5db2a75952 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: true } }
I20240503 01:59:40.131767 9228 raft_consensus.cc:729] T e5a00d4ce3a64ca5a7c5bd65c556f429 P de2f076398ee44008622fb5db2a75952 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: de2f076398ee44008622fb5db2a75952, State: Initialized, Role: LEARNER
I20240503 01:59:40.131852 9223 tablet_bootstrap.cc:492] T 2c8c328220e64e168c74fea7619f0c3b P 65d4632bcdd54ba596acdbbe1b5ef8af: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:40.131878 9228 consensus_queue.cc:260] T e5a00d4ce3a64ca5a7c5bd65c556f429 P de2f076398ee44008622fb5db2a75952 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: true } }
I20240503 01:59:40.132151 9223 tablet_bootstrap.cc:492] T 2c8c328220e64e168c74fea7619f0c3b P 65d4632bcdd54ba596acdbbe1b5ef8af: Bootstrap complete.
I20240503 01:59:40.132215 9228 ts_tablet_manager.cc:1398] T e5a00d4ce3a64ca5a7c5bd65c556f429 P de2f076398ee44008622fb5db2a75952: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:40.132323 9223 ts_tablet_manager.cc:1367] T 2c8c328220e64e168c74fea7619f0c3b P 65d4632bcdd54ba596acdbbe1b5ef8af: Time spent bootstrapping tablet: real 0.026s user 0.024s sys 0.000s
I20240503 01:59:40.132733 9223 raft_consensus.cc:348] T 2c8c328220e64e168c74fea7619f0c3b P 65d4632bcdd54ba596acdbbe1b5ef8af [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: true } }
I20240503 01:59:40.132853 9223 raft_consensus.cc:729] T 2c8c328220e64e168c74fea7619f0c3b P 65d4632bcdd54ba596acdbbe1b5ef8af [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 65d4632bcdd54ba596acdbbe1b5ef8af, State: Initialized, Role: LEARNER
I20240503 01:59:40.132834 6778 tablet_copy_service.cc:342] P b6b1d5677b8d47788b7968f64b3dac3c: Request end of tablet copy session de2f076398ee44008622fb5db2a75952-e5a00d4ce3a64ca5a7c5bd65c556f429 received from {username='slave'} at 127.0.99.135:32853
I20240503 01:59:40.132918 6778 tablet_copy_service.cc:433] P b6b1d5677b8d47788b7968f64b3dac3c: ending tablet copy session de2f076398ee44008622fb5db2a75952-e5a00d4ce3a64ca5a7c5bd65c556f429 on tablet e5a00d4ce3a64ca5a7c5bd65c556f429 with peer de2f076398ee44008622fb5db2a75952
I20240503 01:59:40.133039 9223 consensus_queue.cc:260] T 2c8c328220e64e168c74fea7619f0c3b P 65d4632bcdd54ba596acdbbe1b5ef8af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: true } }
I20240503 01:59:40.133325 9223 ts_tablet_manager.cc:1398] T 2c8c328220e64e168c74fea7619f0c3b P 65d4632bcdd54ba596acdbbe1b5ef8af: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240503 01:59:40.133842 6778 tablet_copy_service.cc:342] P b6b1d5677b8d47788b7968f64b3dac3c: Request end of tablet copy session 65d4632bcdd54ba596acdbbe1b5ef8af-2c8c328220e64e168c74fea7619f0c3b received from {username='slave'} at 127.0.99.134:47629
I20240503 01:59:40.133913 6778 tablet_copy_service.cc:433] P b6b1d5677b8d47788b7968f64b3dac3c: ending tablet copy session 65d4632bcdd54ba596acdbbe1b5ef8af-2c8c328220e64e168c74fea7619f0c3b on tablet 2c8c328220e64e168c74fea7619f0c3b with peer 65d4632bcdd54ba596acdbbe1b5ef8af
I20240503 01:59:40.137027 9254 tablet_copy_client.cc:523] T a388d59f6ba94e91958f55b340343fa6 P 9264f24d028347d98f7e4e2d1aedeb79: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:40.138167 9254 tablet_bootstrap.cc:492] T a388d59f6ba94e91958f55b340343fa6 P 9264f24d028347d98f7e4e2d1aedeb79: Bootstrap starting.
I20240503 01:59:40.144918 9269 tablet_copy_client.cc:523] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 9264f24d028347d98f7e4e2d1aedeb79: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:40.146188 9269 tablet_bootstrap.cc:492] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 9264f24d028347d98f7e4e2d1aedeb79: Bootstrap starting.
I20240503 01:59:40.156297 9254 tablet_bootstrap.cc:492] T a388d59f6ba94e91958f55b340343fa6 P 9264f24d028347d98f7e4e2d1aedeb79: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:40.156569 9254 tablet_bootstrap.cc:492] T a388d59f6ba94e91958f55b340343fa6 P 9264f24d028347d98f7e4e2d1aedeb79: Bootstrap complete.
I20240503 01:59:40.156757 9254 ts_tablet_manager.cc:1367] T a388d59f6ba94e91958f55b340343fa6 P 9264f24d028347d98f7e4e2d1aedeb79: Time spent bootstrapping tablet: real 0.019s user 0.019s sys 0.000s
I20240503 01:59:40.157233 9254 raft_consensus.cc:348] T a388d59f6ba94e91958f55b340343fa6 P 9264f24d028347d98f7e4e2d1aedeb79 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: true } }
I20240503 01:59:40.157460 9254 raft_consensus.cc:729] T a388d59f6ba94e91958f55b340343fa6 P 9264f24d028347d98f7e4e2d1aedeb79 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9264f24d028347d98f7e4e2d1aedeb79, State: Initialized, Role: LEARNER
I20240503 01:59:40.157586 9254 consensus_queue.cc:260] T a388d59f6ba94e91958f55b340343fa6 P 9264f24d028347d98f7e4e2d1aedeb79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: true } }
I20240503 01:59:40.158106 9254 ts_tablet_manager.cc:1398] T a388d59f6ba94e91958f55b340343fa6 P 9264f24d028347d98f7e4e2d1aedeb79: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:40.158520 6647 tablet_copy_service.cc:342] P 79163c69f6f04a9abd0208d74e1c61be: Request end of tablet copy session 9264f24d028347d98f7e4e2d1aedeb79-a388d59f6ba94e91958f55b340343fa6 received from {username='slave'} at 127.0.99.133:50661
I20240503 01:59:40.158605 6647 tablet_copy_service.cc:433] P 79163c69f6f04a9abd0208d74e1c61be: ending tablet copy session 9264f24d028347d98f7e4e2d1aedeb79-a388d59f6ba94e91958f55b340343fa6 on tablet a388d59f6ba94e91958f55b340343fa6 with peer 9264f24d028347d98f7e4e2d1aedeb79
I20240503 01:59:40.167358 9269 tablet_bootstrap.cc:492] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 9264f24d028347d98f7e4e2d1aedeb79: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:40.168525 9269 tablet_bootstrap.cc:492] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 9264f24d028347d98f7e4e2d1aedeb79: Bootstrap complete.
I20240503 01:59:40.168731 9269 ts_tablet_manager.cc:1367] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 9264f24d028347d98f7e4e2d1aedeb79: Time spent bootstrapping tablet: real 0.023s user 0.020s sys 0.001s
I20240503 01:59:40.169142 9269 raft_consensus.cc:348] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 9264f24d028347d98f7e4e2d1aedeb79 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: true } }
I20240503 01:59:40.169238 9269 raft_consensus.cc:729] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 9264f24d028347d98f7e4e2d1aedeb79 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9264f24d028347d98f7e4e2d1aedeb79, State: Initialized, Role: LEARNER
I20240503 01:59:40.169345 9269 consensus_queue.cc:260] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 9264f24d028347d98f7e4e2d1aedeb79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: true } }
I20240503 01:59:40.169584 9269 ts_tablet_manager.cc:1398] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 9264f24d028347d98f7e4e2d1aedeb79: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:40.170073 6385 tablet_copy_service.cc:342] P 4127249ba42644388d225d614c935a25: Request end of tablet copy session 9264f24d028347d98f7e4e2d1aedeb79-2088c84e4eef4ccfbc70d7b668b8f2cf received from {username='slave'} at 127.0.99.133:57445
I20240503 01:59:40.170158 6385 tablet_copy_service.cc:433] P 4127249ba42644388d225d614c935a25: ending tablet copy session 9264f24d028347d98f7e4e2d1aedeb79-2088c84e4eef4ccfbc70d7b668b8f2cf on tablet 2088c84e4eef4ccfbc70d7b668b8f2cf with peer 9264f24d028347d98f7e4e2d1aedeb79
I20240503 01:59:40.201042 8223 raft_consensus.cc:1203] T 2c8c328220e64e168c74fea7619f0c3b P 65d4632bcdd54ba596acdbbe1b5ef8af [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 01:59:40.203810 8080 raft_consensus.cc:1203] T 23b3b3d386544e5d9bfae8041cdf01a0 P 9264f24d028347d98f7e4e2d1aedeb79 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 01:59:40.212026 8223 raft_consensus.cc:1203] T 470b553976064bdd8c321799ac718274 P 65d4632bcdd54ba596acdbbe1b5ef8af [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 01:59:40.212114 8224 raft_consensus.cc:1203] T ff8f06c5ad8442ee942bf900212fe858 P 65d4632bcdd54ba596acdbbe1b5ef8af [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 01:59:40.220520 9411 tablet_copy_client.cc:523] T 4e7ceb1453804246a41bd1dda0e39d02 P 65d4632bcdd54ba596acdbbe1b5ef8af: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:40.221801 9411 tablet_bootstrap.cc:492] T 4e7ceb1453804246a41bd1dda0e39d02 P 65d4632bcdd54ba596acdbbe1b5ef8af: Bootstrap starting.
I20240503 01:59:40.240657 9411 tablet_bootstrap.cc:492] T 4e7ceb1453804246a41bd1dda0e39d02 P 65d4632bcdd54ba596acdbbe1b5ef8af: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:40.241019 9411 tablet_bootstrap.cc:492] T 4e7ceb1453804246a41bd1dda0e39d02 P 65d4632bcdd54ba596acdbbe1b5ef8af: Bootstrap complete.
I20240503 01:59:40.241230 9411 ts_tablet_manager.cc:1367] T 4e7ceb1453804246a41bd1dda0e39d02 P 65d4632bcdd54ba596acdbbe1b5ef8af: Time spent bootstrapping tablet: real 0.019s user 0.020s sys 0.000s
I20240503 01:59:40.241760 9411 raft_consensus.cc:348] T 4e7ceb1453804246a41bd1dda0e39d02 P 65d4632bcdd54ba596acdbbe1b5ef8af [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: true } }
I20240503 01:59:40.241895 9411 raft_consensus.cc:729] T 4e7ceb1453804246a41bd1dda0e39d02 P 65d4632bcdd54ba596acdbbe1b5ef8af [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 65d4632bcdd54ba596acdbbe1b5ef8af, State: Initialized, Role: LEARNER
I20240503 01:59:40.241993 9411 consensus_queue.cc:260] T 4e7ceb1453804246a41bd1dda0e39d02 P 65d4632bcdd54ba596acdbbe1b5ef8af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: true } }
I20240503 01:59:40.242775 9411 ts_tablet_manager.cc:1398] T 4e7ceb1453804246a41bd1dda0e39d02 P 65d4632bcdd54ba596acdbbe1b5ef8af: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:40.243288 6385 tablet_copy_service.cc:342] P 4127249ba42644388d225d614c935a25: Request end of tablet copy session 65d4632bcdd54ba596acdbbe1b5ef8af-4e7ceb1453804246a41bd1dda0e39d02 received from {username='slave'} at 127.0.99.134:49429
I20240503 01:59:40.243377 6385 tablet_copy_service.cc:433] P 4127249ba42644388d225d614c935a25: ending tablet copy session 65d4632bcdd54ba596acdbbe1b5ef8af-4e7ceb1453804246a41bd1dda0e39d02 on tablet 4e7ceb1453804246a41bd1dda0e39d02 with peer 65d4632bcdd54ba596acdbbe1b5ef8af
I20240503 01:59:40.268925 9923 raft_consensus.cc:1050] T e5ed3e729bc64ebeac216ce5076b9fd9 P 0c4821d9898d42799c5789d4d2da3f63: attempting to promote NON_VOTER de2f076398ee44008622fb5db2a75952 to VOTER
I20240503 01:59:40.269421 9923 consensus_queue.cc:237] T e5ed3e729bc64ebeac216ce5076b9fd9 P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } }
I20240503 01:59:40.271219 6365 raft_consensus.cc:1261] T e5ed3e729bc64ebeac216ce5076b9fd9 P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Refusing update from remote peer 0c4821d9898d42799c5789d4d2da3f63: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:40.271353 8355 raft_consensus.cc:1261] T e5ed3e729bc64ebeac216ce5076b9fd9 P de2f076398ee44008622fb5db2a75952 [term 1 LEARNER]: Refusing update from remote peer 0c4821d9898d42799c5789d4d2da3f63: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:40.271701 9931 consensus_queue.cc:1035] T e5ed3e729bc64ebeac216ce5076b9fd9 P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:40.271908 6758 raft_consensus.cc:1261] T e5ed3e729bc64ebeac216ce5076b9fd9 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Refusing update from remote peer 0c4821d9898d42799c5789d4d2da3f63: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:40.272298 9931 consensus_queue.cc:1035] T e5ed3e729bc64ebeac216ce5076b9fd9 P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:40.272716 9924 consensus_queue.cc:1035] T e5ed3e729bc64ebeac216ce5076b9fd9 P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:40.275341 9466 tablet_copy_client.cc:523] T a655d914858447b59fe5763e43c0807d P 7a1404fe1c1b4b388b821c8261b53104: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:40.276679 9466 tablet_bootstrap.cc:492] T a655d914858447b59fe5763e43c0807d P 7a1404fe1c1b4b388b821c8261b53104: Bootstrap starting.
I20240503 01:59:40.281122 9931 raft_consensus.cc:2932] T e5ed3e729bc64ebeac216ce5076b9fd9 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, de2f076398ee44008622fb5db2a75952 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } }
I20240503 01:59:40.281553 6758 raft_consensus.cc:2932] T e5ed3e729bc64ebeac216ce5076b9fd9 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, de2f076398ee44008622fb5db2a75952 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } }
I20240503 01:59:40.282315 6365 raft_consensus.cc:2932] T e5ed3e729bc64ebeac216ce5076b9fd9 P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, de2f076398ee44008622fb5db2a75952 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } }
I20240503 01:59:40.284413 6214 catalog_manager.cc:5484] T e5ed3e729bc64ebeac216ce5076b9fd9 P b6b1d5677b8d47788b7968f64b3dac3c reported cstate change: config changed from index 4 to 5, de2f076398ee44008622fb5db2a75952 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0c4821d9898d42799c5789d4d2da3f63" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } }
I20240503 01:59:40.284916 8355 raft_consensus.cc:2932] T e5ed3e729bc64ebeac216ce5076b9fd9 P de2f076398ee44008622fb5db2a75952 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, de2f076398ee44008622fb5db2a75952 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } }
I20240503 01:59:40.290889 6495 consensus_queue.cc:237] T e5ed3e729bc64ebeac216ce5076b9fd9 P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } }
I20240503 01:59:40.291886 6758 raft_consensus.cc:1261] T e5ed3e729bc64ebeac216ce5076b9fd9 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Refusing update from remote peer 0c4821d9898d42799c5789d4d2da3f63: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:40.292179 8355 raft_consensus.cc:1261] T e5ed3e729bc64ebeac216ce5076b9fd9 P de2f076398ee44008622fb5db2a75952 [term 1 FOLLOWER]: Refusing update from remote peer 0c4821d9898d42799c5789d4d2da3f63: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:40.292232 9924 consensus_queue.cc:1035] T e5ed3e729bc64ebeac216ce5076b9fd9 P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:40.292794 7917 consensus_queue.cc:1035] T e5ed3e729bc64ebeac216ce5076b9fd9 P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:40.293284 8080 raft_consensus.cc:1203] T 2ac8004211e64c3481e75377092fa93b P 9264f24d028347d98f7e4e2d1aedeb79 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 01:59:40.294348 9923 raft_consensus.cc:2932] T e5ed3e729bc64ebeac216ce5076b9fd9 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4127249ba42644388d225d614c935a25 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } }
I20240503 01:59:40.294885 6758 raft_consensus.cc:2932] T e5ed3e729bc64ebeac216ce5076b9fd9 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4127249ba42644388d225d614c935a25 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } }
I20240503 01:59:40.297359 6223 catalog_manager.cc:5484] T e5ed3e729bc64ebeac216ce5076b9fd9 P 0c4821d9898d42799c5789d4d2da3f63 reported cstate change: config changed from index 5 to 6, VOTER 4127249ba42644388d225d614c935a25 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "0c4821d9898d42799c5789d4d2da3f63" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:40.298154 6210 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e5ed3e729bc64ebeac216ce5076b9fd9 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:40.298700 8355 raft_consensus.cc:2932] T e5ed3e729bc64ebeac216ce5076b9fd9 P de2f076398ee44008622fb5db2a75952 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4127249ba42644388d225d614c935a25 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } }
I20240503 01:59:40.299969 6344 tablet_service.cc:1514] Processing DeleteTablet for tablet e5ed3e729bc64ebeac216ce5076b9fd9 with delete_type TABLET_DATA_TOMBSTONED (TS 4127249ba42644388d225d614c935a25 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:57398
I20240503 01:59:40.301209 9950 tablet_replica.cc:330] T e5ed3e729bc64ebeac216ce5076b9fd9 P 4127249ba42644388d225d614c935a25: stopping tablet replica
I20240503 01:59:40.302510 9950 raft_consensus.cc:2229] T e5ed3e729bc64ebeac216ce5076b9fd9 P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:40.302824 9950 raft_consensus.cc:2258] T e5ed3e729bc64ebeac216ce5076b9fd9 P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:40.304354 9950 ts_tablet_manager.cc:1875] T e5ed3e729bc64ebeac216ce5076b9fd9 P 4127249ba42644388d225d614c935a25: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:40.306886 9950 ts_tablet_manager.cc:1888] T e5ed3e729bc64ebeac216ce5076b9fd9 P 4127249ba42644388d225d614c935a25: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 01:59:40.306983 9950 log.cc:1196] T e5ed3e729bc64ebeac216ce5076b9fd9 P 4127249ba42644388d225d614c935a25: Deleting WAL directory at /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/e5ed3e729bc64ebeac216ce5076b9fd9
I20240503 01:59:40.307552 6208 catalog_manager.cc:4830] TS 4127249ba42644388d225d614c935a25 (127.0.99.129:35613): tablet e5ed3e729bc64ebeac216ce5076b9fd9 (table rebalance_test_table_0 [id=2e2540d8e10e43848a2e486f767d1576]) successfully deleted
I20240503 01:59:40.307646 9466 tablet_bootstrap.cc:492] T a655d914858447b59fe5763e43c0807d P 7a1404fe1c1b4b388b821c8261b53104: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:40.307891 9466 tablet_bootstrap.cc:492] T a655d914858447b59fe5763e43c0807d P 7a1404fe1c1b4b388b821c8261b53104: Bootstrap complete.
I20240503 01:59:40.308045 9466 ts_tablet_manager.cc:1367] T a655d914858447b59fe5763e43c0807d P 7a1404fe1c1b4b388b821c8261b53104: Time spent bootstrapping tablet: real 0.031s user 0.025s sys 0.000s
I20240503 01:59:40.308498 9466 raft_consensus.cc:348] T a655d914858447b59fe5763e43c0807d P 7a1404fe1c1b4b388b821c8261b53104 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: true } }
I20240503 01:59:40.308635 9466 raft_consensus.cc:729] T a655d914858447b59fe5763e43c0807d P 7a1404fe1c1b4b388b821c8261b53104 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7a1404fe1c1b4b388b821c8261b53104, State: Initialized, Role: LEARNER
I20240503 01:59:40.308763 9466 consensus_queue.cc:260] T a655d914858447b59fe5763e43c0807d P 7a1404fe1c1b4b388b821c8261b53104 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: true } }
I20240503 01:59:40.309010 9466 ts_tablet_manager.cc:1398] T a655d914858447b59fe5763e43c0807d P 7a1404fe1c1b4b388b821c8261b53104: Time spent starting tablet: real 0.001s user 0.000s sys 0.003s
I20240503 01:59:40.309530 6647 tablet_copy_service.cc:342] P 79163c69f6f04a9abd0208d74e1c61be: Request end of tablet copy session 7a1404fe1c1b4b388b821c8261b53104-a655d914858447b59fe5763e43c0807d received from {username='slave'} at 127.0.99.136:51745
I20240503 01:59:40.309633 6647 tablet_copy_service.cc:433] P 79163c69f6f04a9abd0208d74e1c61be: ending tablet copy session 7a1404fe1c1b4b388b821c8261b53104-a655d914858447b59fe5763e43c0807d on tablet a655d914858447b59fe5763e43c0807d with peer 7a1404fe1c1b4b388b821c8261b53104
I20240503 01:59:40.313342 8355 raft_consensus.cc:1203] T 1d392e08c15a4d4098a350287b3163ef P de2f076398ee44008622fb5db2a75952 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 01:59:40.329108 8355 raft_consensus.cc:1203] T 0bdcea4e59ec4e2da63b9316f9d718be P de2f076398ee44008622fb5db2a75952 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 01:59:40.346143 8224 raft_consensus.cc:1203] T 67150a2f5cb147b8b57a8c8040b567e1 P 65d4632bcdd54ba596acdbbe1b5ef8af [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 01:59:40.424067 8355 raft_consensus.cc:1203] T e5a00d4ce3a64ca5a7c5bd65c556f429 P de2f076398ee44008622fb5db2a75952 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 01:59:40.472167 8355 raft_consensus.cc:1203] T a5926dae08184952bc693342c2c61ebf P de2f076398ee44008622fb5db2a75952 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 01:59:40.475966 8486 raft_consensus.cc:1203] T a9a74e04ce3745b2b479e96a3f3dc86e P 7a1404fe1c1b4b388b821c8261b53104 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 01:59:40.475966 8485 raft_consensus.cc:1203] T 120d5dd4f26c42399702141094ce7982 P 7a1404fe1c1b4b388b821c8261b53104 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 01:59:40.480194 10101 raft_consensus.cc:1050] T d6ee8175874c4063986033b0a80ccb82 P 79163c69f6f04a9abd0208d74e1c61be: attempting to promote NON_VOTER 7a1404fe1c1b4b388b821c8261b53104 to VOTER
I20240503 01:59:40.480827 10101 consensus_queue.cc:237] T d6ee8175874c4063986033b0a80ccb82 P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } }
I20240503 01:59:40.482208 6495 raft_consensus.cc:1261] T d6ee8175874c4063986033b0a80ccb82 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:40.482260 8485 raft_consensus.cc:1261] T d6ee8175874c4063986033b0a80ccb82 P 7a1404fe1c1b4b388b821c8261b53104 [term 1 LEARNER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:40.482354 6758 raft_consensus.cc:1261] T d6ee8175874c4063986033b0a80ccb82 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:40.482641 10108 consensus_queue.cc:1035] T d6ee8175874c4063986033b0a80ccb82 P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:40.482818 10101 consensus_queue.cc:1035] T d6ee8175874c4063986033b0a80ccb82 P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:40.482976 7647 consensus_queue.cc:1035] T d6ee8175874c4063986033b0a80ccb82 P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:40.485724 10101 raft_consensus.cc:2932] T d6ee8175874c4063986033b0a80ccb82 P 79163c69f6f04a9abd0208d74e1c61be [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7a1404fe1c1b4b388b821c8261b53104 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } }
I20240503 01:59:40.486255 6495 raft_consensus.cc:2932] T d6ee8175874c4063986033b0a80ccb82 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7a1404fe1c1b4b388b821c8261b53104 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } }
I20240503 01:59:40.486239 8485 raft_consensus.cc:2932] T d6ee8175874c4063986033b0a80ccb82 P 7a1404fe1c1b4b388b821c8261b53104 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7a1404fe1c1b4b388b821c8261b53104 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } }
I20240503 01:59:40.488827 6221 catalog_manager.cc:5484] T d6ee8175874c4063986033b0a80ccb82 P 7a1404fe1c1b4b388b821c8261b53104 reported cstate change: config changed from index 4 to 5, 7a1404fe1c1b4b388b821c8261b53104 (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "79163c69f6f04a9abd0208d74e1c61be" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } }
I20240503 01:59:40.490288 6758 raft_consensus.cc:2932] T d6ee8175874c4063986033b0a80ccb82 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7a1404fe1c1b4b388b821c8261b53104 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } }
I20240503 01:59:40.492277 6625 consensus_queue.cc:237] T d6ee8175874c4063986033b0a80ccb82 P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } }
I20240503 01:59:40.493448 6758 raft_consensus.cc:1261] T d6ee8175874c4063986033b0a80ccb82 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:40.493816 10101 consensus_queue.cc:1035] T d6ee8175874c4063986033b0a80ccb82 P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:40.493979 8485 raft_consensus.cc:1261] T d6ee8175874c4063986033b0a80ccb82 P 7a1404fe1c1b4b388b821c8261b53104 [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:40.494315 10101 consensus_queue.cc:1035] T d6ee8175874c4063986033b0a80ccb82 P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:40.495471 7647 raft_consensus.cc:2932] T d6ee8175874c4063986033b0a80ccb82 P 79163c69f6f04a9abd0208d74e1c61be [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } }
I20240503 01:59:40.496018 6758 raft_consensus.cc:2932] T d6ee8175874c4063986033b0a80ccb82 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } }
I20240503 01:59:40.496413 8485 raft_consensus.cc:2932] T d6ee8175874c4063986033b0a80ccb82 P 7a1404fe1c1b4b388b821c8261b53104 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } }
I20240503 01:59:40.497094 6208 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d6ee8175874c4063986033b0a80ccb82 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:40.498216 6223 catalog_manager.cc:5484] T d6ee8175874c4063986033b0a80ccb82 P b6b1d5677b8d47788b7968f64b3dac3c reported cstate change: config changed from index 5 to 6, VOTER 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "79163c69f6f04a9abd0208d74e1c61be" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } }
I20240503 01:59:40.500564 6476 tablet_service.cc:1514] Processing DeleteTablet for tablet d6ee8175874c4063986033b0a80ccb82 with delete_type TABLET_DATA_TOMBSTONED (TS 0c4821d9898d42799c5789d4d2da3f63 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:39850
I20240503 01:59:40.501212 10112 tablet_replica.cc:330] T d6ee8175874c4063986033b0a80ccb82 P 0c4821d9898d42799c5789d4d2da3f63: stopping tablet replica
I20240503 01:59:40.501533 10112 raft_consensus.cc:2229] T d6ee8175874c4063986033b0a80ccb82 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:40.501722 10112 raft_consensus.cc:2258] T d6ee8175874c4063986033b0a80ccb82 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:40.502722 10112 ts_tablet_manager.cc:1875] T d6ee8175874c4063986033b0a80ccb82 P 0c4821d9898d42799c5789d4d2da3f63: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:40.504601 10112 ts_tablet_manager.cc:1888] T d6ee8175874c4063986033b0a80ccb82 P 0c4821d9898d42799c5789d4d2da3f63: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 01:59:40.504720 10112 log.cc:1196] T d6ee8175874c4063986033b0a80ccb82 P 0c4821d9898d42799c5789d4d2da3f63: Deleting WAL directory at /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/d6ee8175874c4063986033b0a80ccb82
I20240503 01:59:40.505236 6210 catalog_manager.cc:4830] TS 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130:34189): tablet d6ee8175874c4063986033b0a80ccb82 (table rebalance_test_table_1 [id=91bdb83bc45b4b61b094a0180eacf8e3]) successfully deleted
I20240503 01:59:40.521298 8080 raft_consensus.cc:1203] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 9264f24d028347d98f7e4e2d1aedeb79 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 01:59:40.526232 7647 raft_consensus.cc:1050] T 21d27cca81e8455e9934e1cc604fc2b2 P 79163c69f6f04a9abd0208d74e1c61be: attempting to promote NON_VOTER 7a1404fe1c1b4b388b821c8261b53104 to VOTER
I20240503 01:59:40.526674 7647 consensus_queue.cc:237] T 21d27cca81e8455e9934e1cc604fc2b2 P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } }
I20240503 01:59:40.527998 6365 raft_consensus.cc:1261] T 21d27cca81e8455e9934e1cc604fc2b2 P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:40.528043 8485 raft_consensus.cc:1261] T 21d27cca81e8455e9934e1cc604fc2b2 P 7a1404fe1c1b4b388b821c8261b53104 [term 1 LEARNER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:40.528090 6495 raft_consensus.cc:1261] T 21d27cca81e8455e9934e1cc604fc2b2 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:40.528456 7647 consensus_queue.cc:1035] T 21d27cca81e8455e9934e1cc604fc2b2 P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:40.528630 10108 consensus_queue.cc:1035] T 21d27cca81e8455e9934e1cc604fc2b2 P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:40.528869 10101 consensus_queue.cc:1035] T 21d27cca81e8455e9934e1cc604fc2b2 P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:40.530427 7647 raft_consensus.cc:2932] T 21d27cca81e8455e9934e1cc604fc2b2 P 79163c69f6f04a9abd0208d74e1c61be [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7a1404fe1c1b4b388b821c8261b53104 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } }
I20240503 01:59:40.530826 6495 raft_consensus.cc:2932] T 21d27cca81e8455e9934e1cc604fc2b2 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7a1404fe1c1b4b388b821c8261b53104 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } }
I20240503 01:59:40.531366 8485 raft_consensus.cc:2932] T 21d27cca81e8455e9934e1cc604fc2b2 P 7a1404fe1c1b4b388b821c8261b53104 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7a1404fe1c1b4b388b821c8261b53104 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } }
I20240503 01:59:40.532347 6365 raft_consensus.cc:2932] T 21d27cca81e8455e9934e1cc604fc2b2 P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7a1404fe1c1b4b388b821c8261b53104 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } }
I20240503 01:59:40.533329 6219 catalog_manager.cc:5484] T 21d27cca81e8455e9934e1cc604fc2b2 P 0c4821d9898d42799c5789d4d2da3f63 reported cstate change: config changed from index 4 to 5, 7a1404fe1c1b4b388b821c8261b53104 (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "79163c69f6f04a9abd0208d74e1c61be" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } }
I20240503 01:59:40.537053 6625 consensus_queue.cc:237] T 21d27cca81e8455e9934e1cc604fc2b2 P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } }
I20240503 01:59:40.538241 6365 raft_consensus.cc:1261] T 21d27cca81e8455e9934e1cc604fc2b2 P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:40.538362 8485 raft_consensus.cc:1261] T 21d27cca81e8455e9934e1cc604fc2b2 P 7a1404fe1c1b4b388b821c8261b53104 [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:40.538619 10101 consensus_queue.cc:1035] T 21d27cca81e8455e9934e1cc604fc2b2 P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:40.538785 9472 consensus_queue.cc:1035] T 21d27cca81e8455e9934e1cc604fc2b2 P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:40.540194 10108 raft_consensus.cc:2932] T 21d27cca81e8455e9934e1cc604fc2b2 P 79163c69f6f04a9abd0208d74e1c61be [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } }
I20240503 01:59:40.540736 6365 raft_consensus.cc:2932] T 21d27cca81e8455e9934e1cc604fc2b2 P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } }
I20240503 01:59:40.540736 8485 raft_consensus.cc:2932] T 21d27cca81e8455e9934e1cc604fc2b2 P 7a1404fe1c1b4b388b821c8261b53104 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } }
I20240503 01:59:40.542201 6208 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 21d27cca81e8455e9934e1cc604fc2b2 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:40.543218 6223 catalog_manager.cc:5484] T 21d27cca81e8455e9934e1cc604fc2b2 P 79163c69f6f04a9abd0208d74e1c61be reported cstate change: config changed from index 5 to 6, VOTER 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "79163c69f6f04a9abd0208d74e1c61be" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:40.545539 6476 tablet_service.cc:1514] Processing DeleteTablet for tablet 21d27cca81e8455e9934e1cc604fc2b2 with delete_type TABLET_DATA_TOMBSTONED (TS 0c4821d9898d42799c5789d4d2da3f63 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:39850
I20240503 01:59:40.545760 10112 tablet_replica.cc:330] T 21d27cca81e8455e9934e1cc604fc2b2 P 0c4821d9898d42799c5789d4d2da3f63: stopping tablet replica
I20240503 01:59:40.545915 10112 raft_consensus.cc:2229] T 21d27cca81e8455e9934e1cc604fc2b2 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:40.546089 10112 raft_consensus.cc:2258] T 21d27cca81e8455e9934e1cc604fc2b2 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:40.546944 10112 ts_tablet_manager.cc:1875] T 21d27cca81e8455e9934e1cc604fc2b2 P 0c4821d9898d42799c5789d4d2da3f63: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:40.548596 10112 ts_tablet_manager.cc:1888] T 21d27cca81e8455e9934e1cc604fc2b2 P 0c4821d9898d42799c5789d4d2da3f63: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 01:59:40.548686 10112 log.cc:1196] T 21d27cca81e8455e9934e1cc604fc2b2 P 0c4821d9898d42799c5789d4d2da3f63: Deleting WAL directory at /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/21d27cca81e8455e9934e1cc604fc2b2
I20240503 01:59:40.549129 6210 catalog_manager.cc:4830] TS 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130:34189): tablet 21d27cca81e8455e9934e1cc604fc2b2 (table rebalance_test_table_3 [id=2c07d364168a409a98aed9a7c3c1015c]) successfully deleted
I20240503 01:59:40.552950 8224 raft_consensus.cc:1203] T 4e7ceb1453804246a41bd1dda0e39d02 P 65d4632bcdd54ba596acdbbe1b5ef8af [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 01:59:40.636417 7917 raft_consensus.cc:1050] T ff8f06c5ad8442ee942bf900212fe858 P 0c4821d9898d42799c5789d4d2da3f63: attempting to promote NON_VOTER 65d4632bcdd54ba596acdbbe1b5ef8af to VOTER
I20240503 01:59:40.636937 7917 consensus_queue.cc:237] T ff8f06c5ad8442ee942bf900212fe858 P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } }
I20240503 01:59:40.637403 9773 raft_consensus.cc:1050] T 23b3b3d386544e5d9bfae8041cdf01a0 P 4127249ba42644388d225d614c935a25: attempting to promote NON_VOTER 9264f24d028347d98f7e4e2d1aedeb79 to VOTER
I20240503 01:59:40.637938 9773 consensus_queue.cc:237] T 23b3b3d386544e5d9bfae8041cdf01a0 P 4127249ba42644388d225d614c935a25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } }
I20240503 01:59:40.638401 6627 raft_consensus.cc:1261] T ff8f06c5ad8442ee942bf900212fe858 P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Refusing update from remote peer 0c4821d9898d42799c5789d4d2da3f63: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:40.638448 8224 raft_consensus.cc:1261] T ff8f06c5ad8442ee942bf900212fe858 P 65d4632bcdd54ba596acdbbe1b5ef8af [term 1 LEARNER]: Refusing update from remote peer 0c4821d9898d42799c5789d4d2da3f63: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:40.638734 9924 consensus_queue.cc:1035] T ff8f06c5ad8442ee942bf900212fe858 P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:40.639309 8080 raft_consensus.cc:1261] T 23b3b3d386544e5d9bfae8041cdf01a0 P 9264f24d028347d98f7e4e2d1aedeb79 [term 1 LEARNER]: Refusing update from remote peer 4127249ba42644388d225d614c935a25: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:40.639539 6758 raft_consensus.cc:1261] T 23b3b3d386544e5d9bfae8041cdf01a0 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Refusing update from remote peer 4127249ba42644388d225d614c935a25: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:40.639739 8701 consensus_queue.cc:1035] T 23b3b3d386544e5d9bfae8041cdf01a0 P 4127249ba42644388d225d614c935a25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:40.640121 9924 consensus_queue.cc:1035] T ff8f06c5ad8442ee942bf900212fe858 P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:40.640424 6625 raft_consensus.cc:1261] T 23b3b3d386544e5d9bfae8041cdf01a0 P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Refusing update from remote peer 4127249ba42644388d225d614c935a25: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:40.640435 8701 consensus_queue.cc:1035] T 23b3b3d386544e5d9bfae8041cdf01a0 P 4127249ba42644388d225d614c935a25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:40.640821 10121 consensus_queue.cc:1035] T 23b3b3d386544e5d9bfae8041cdf01a0 P 4127249ba42644388d225d614c935a25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:40.641899 6365 raft_consensus.cc:1261] T ff8f06c5ad8442ee942bf900212fe858 P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Refusing update from remote peer 0c4821d9898d42799c5789d4d2da3f63: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:40.642426 9924 consensus_queue.cc:1035] T ff8f06c5ad8442ee942bf900212fe858 P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:40.643082 8485 raft_consensus.cc:1203] T a655d914858447b59fe5763e43c0807d P 7a1404fe1c1b4b388b821c8261b53104 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 01:59:40.644333 9923 raft_consensus.cc:2932] T ff8f06c5ad8442ee942bf900212fe858 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 65d4632bcdd54ba596acdbbe1b5ef8af (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } }
I20240503 01:59:40.644402 10121 raft_consensus.cc:2932] T 23b3b3d386544e5d9bfae8041cdf01a0 P 4127249ba42644388d225d614c935a25 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9264f24d028347d98f7e4e2d1aedeb79 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } }
I20240503 01:59:40.645180 6365 raft_consensus.cc:2932] T ff8f06c5ad8442ee942bf900212fe858 P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 65d4632bcdd54ba596acdbbe1b5ef8af (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } }
I20240503 01:59:40.645277 8224 raft_consensus.cc:2932] T ff8f06c5ad8442ee942bf900212fe858 P 65d4632bcdd54ba596acdbbe1b5ef8af [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 65d4632bcdd54ba596acdbbe1b5ef8af (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } }
I20240503 01:59:40.646871 6758 raft_consensus.cc:2932] T 23b3b3d386544e5d9bfae8041cdf01a0 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9264f24d028347d98f7e4e2d1aedeb79 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } }
I20240503 01:59:40.647274 6625 raft_consensus.cc:2932] T ff8f06c5ad8442ee942bf900212fe858 P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 65d4632bcdd54ba596acdbbe1b5ef8af (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } }
I20240503 01:59:40.648144 6219 catalog_manager.cc:5484] T 23b3b3d386544e5d9bfae8041cdf01a0 P 4127249ba42644388d225d614c935a25 reported cstate change: config changed from index 4 to 5, 9264f24d028347d98f7e4e2d1aedeb79 (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4127249ba42644388d225d614c935a25" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:40.649628 6223 catalog_manager.cc:5484] T ff8f06c5ad8442ee942bf900212fe858 P 65d4632bcdd54ba596acdbbe1b5ef8af reported cstate change: config changed from index 4 to 5, 65d4632bcdd54ba596acdbbe1b5ef8af (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0c4821d9898d42799c5789d4d2da3f63" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } }
I20240503 01:59:40.651167 8080 raft_consensus.cc:1203] T a388d59f6ba94e91958f55b340343fa6 P 9264f24d028347d98f7e4e2d1aedeb79 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 01:59:40.651495 6625 raft_consensus.cc:2932] T 23b3b3d386544e5d9bfae8041cdf01a0 P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9264f24d028347d98f7e4e2d1aedeb79 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } }
I20240503 01:59:40.651614 8093 raft_consensus.cc:2932] T 23b3b3d386544e5d9bfae8041cdf01a0 P 9264f24d028347d98f7e4e2d1aedeb79 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9264f24d028347d98f7e4e2d1aedeb79 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } }
I20240503 01:59:40.654390 6495 consensus_queue.cc:237] T ff8f06c5ad8442ee942bf900212fe858 P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } }
I20240503 01:59:40.655768 8224 raft_consensus.cc:1261] T ff8f06c5ad8442ee942bf900212fe858 P 65d4632bcdd54ba596acdbbe1b5ef8af [term 1 FOLLOWER]: Refusing update from remote peer 0c4821d9898d42799c5789d4d2da3f63: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:40.656123 6625 raft_consensus.cc:1261] T ff8f06c5ad8442ee942bf900212fe858 P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Refusing update from remote peer 0c4821d9898d42799c5789d4d2da3f63: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:40.656196 7917 consensus_queue.cc:1035] T ff8f06c5ad8442ee942bf900212fe858 P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:40.656494 9931 consensus_queue.cc:1035] T ff8f06c5ad8442ee942bf900212fe858 P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:40.656821 6365 consensus_queue.cc:237] T 23b3b3d386544e5d9bfae8041cdf01a0 P 4127249ba42644388d225d614c935a25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } }
I20240503 01:59:40.657964 8093 raft_consensus.cc:1261] T 23b3b3d386544e5d9bfae8041cdf01a0 P 9264f24d028347d98f7e4e2d1aedeb79 [term 1 FOLLOWER]: Refusing update from remote peer 4127249ba42644388d225d614c935a25: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:40.658036 6758 raft_consensus.cc:1261] T 23b3b3d386544e5d9bfae8041cdf01a0 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Refusing update from remote peer 4127249ba42644388d225d614c935a25: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:40.658404 10120 consensus_queue.cc:1035] T 23b3b3d386544e5d9bfae8041cdf01a0 P 4127249ba42644388d225d614c935a25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:40.658586 9773 consensus_queue.cc:1035] T 23b3b3d386544e5d9bfae8041cdf01a0 P 4127249ba42644388d225d614c935a25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:40.658756 7917 raft_consensus.cc:2932] T ff8f06c5ad8442ee942bf900212fe858 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4127249ba42644388d225d614c935a25 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } }
I20240503 01:59:40.660198 6210 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ff8f06c5ad8442ee942bf900212fe858 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:40.660192 9773 raft_consensus.cc:2932] T 23b3b3d386544e5d9bfae8041cdf01a0 P 4127249ba42644388d225d614c935a25 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } }
I20240503 01:59:40.660513 8224 raft_consensus.cc:2932] T ff8f06c5ad8442ee942bf900212fe858 P 65d4632bcdd54ba596acdbbe1b5ef8af [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4127249ba42644388d225d614c935a25 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } }
I20240503 01:59:40.661396 6758 raft_consensus.cc:2932] T 23b3b3d386544e5d9bfae8041cdf01a0 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } }
I20240503 01:59:40.661454 8093 raft_consensus.cc:2932] T 23b3b3d386544e5d9bfae8041cdf01a0 P 9264f24d028347d98f7e4e2d1aedeb79 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } }
I20240503 01:59:40.661808 6208 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 23b3b3d386544e5d9bfae8041cdf01a0 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:40.661731 6221 catalog_manager.cc:5484] T ff8f06c5ad8442ee942bf900212fe858 P 0c4821d9898d42799c5789d4d2da3f63 reported cstate change: config changed from index 5 to 6, VOTER 4127249ba42644388d225d614c935a25 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "0c4821d9898d42799c5789d4d2da3f63" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:40.663719 6214 catalog_manager.cc:5484] T 23b3b3d386544e5d9bfae8041cdf01a0 P 9264f24d028347d98f7e4e2d1aedeb79 reported cstate change: config changed from index 5 to 6, VOTER 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "4127249ba42644388d225d614c935a25" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } }
I20240503 01:59:40.659229 6625 raft_consensus.cc:2932] T ff8f06c5ad8442ee942bf900212fe858 P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4127249ba42644388d225d614c935a25 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } }
I20240503 01:59:40.665073 6344 tablet_service.cc:1514] Processing DeleteTablet for tablet ff8f06c5ad8442ee942bf900212fe858 with delete_type TABLET_DATA_TOMBSTONED (TS 4127249ba42644388d225d614c935a25 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:57398
I20240503 01:59:40.665257 9950 tablet_replica.cc:330] T ff8f06c5ad8442ee942bf900212fe858 P 4127249ba42644388d225d614c935a25: stopping tablet replica
I20240503 01:59:40.665385 9950 raft_consensus.cc:2229] T ff8f06c5ad8442ee942bf900212fe858 P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:40.665544 9950 raft_consensus.cc:2258] T ff8f06c5ad8442ee942bf900212fe858 P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:40.666334 9950 ts_tablet_manager.cc:1875] T ff8f06c5ad8442ee942bf900212fe858 P 4127249ba42644388d225d614c935a25: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:40.668552 6607 tablet_service.cc:1514] Processing DeleteTablet for tablet 23b3b3d386544e5d9bfae8041cdf01a0 with delete_type TABLET_DATA_TOMBSTONED (TS 79163c69f6f04a9abd0208d74e1c61be not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:47560
I20240503 01:59:40.669023 9950 ts_tablet_manager.cc:1888] T ff8f06c5ad8442ee942bf900212fe858 P 4127249ba42644388d225d614c935a25: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 01:59:40.669057 10128 tablet_replica.cc:330] T 23b3b3d386544e5d9bfae8041cdf01a0 P 79163c69f6f04a9abd0208d74e1c61be: stopping tablet replica
I20240503 01:59:40.669096 9950 log.cc:1196] T ff8f06c5ad8442ee942bf900212fe858 P 4127249ba42644388d225d614c935a25: Deleting WAL directory at /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/ff8f06c5ad8442ee942bf900212fe858
I20240503 01:59:40.669298 10128 raft_consensus.cc:2229] T 23b3b3d386544e5d9bfae8041cdf01a0 P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:40.669525 6208 catalog_manager.cc:4830] TS 4127249ba42644388d225d614c935a25 (127.0.99.129:35613): tablet ff8f06c5ad8442ee942bf900212fe858 (table rebalance_test_table_0 [id=2e2540d8e10e43848a2e486f767d1576]) successfully deleted
I20240503 01:59:40.669620 10128 raft_consensus.cc:2258] T 23b3b3d386544e5d9bfae8041cdf01a0 P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:40.670552 10128 ts_tablet_manager.cc:1875] T 23b3b3d386544e5d9bfae8041cdf01a0 P 79163c69f6f04a9abd0208d74e1c61be: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:40.672292 10128 ts_tablet_manager.cc:1888] T 23b3b3d386544e5d9bfae8041cdf01a0 P 79163c69f6f04a9abd0208d74e1c61be: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 01:59:40.672359 10128 log.cc:1196] T 23b3b3d386544e5d9bfae8041cdf01a0 P 79163c69f6f04a9abd0208d74e1c61be: Deleting WAL directory at /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/23b3b3d386544e5d9bfae8041cdf01a0
I20240503 01:59:40.672829 6208 catalog_manager.cc:4830] TS 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131:44005): tablet 23b3b3d386544e5d9bfae8041cdf01a0 (table rebalance_test_table_7 [id=dbd6f15c28634e3dac01249aa7cc2e25]) successfully deleted
I20240503 01:59:40.675843 10130 raft_consensus.cc:1050] T 2c8c328220e64e168c74fea7619f0c3b P b6b1d5677b8d47788b7968f64b3dac3c: attempting to promote NON_VOTER 65d4632bcdd54ba596acdbbe1b5ef8af to VOTER
I20240503 01:59:40.676338 10130 consensus_queue.cc:237] T 2c8c328220e64e168c74fea7619f0c3b P b6b1d5677b8d47788b7968f64b3dac3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } }
I20240503 01:59:40.677631 6495 raft_consensus.cc:1261] T 2c8c328220e64e168c74fea7619f0c3b P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Refusing update from remote peer b6b1d5677b8d47788b7968f64b3dac3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:40.677688 6625 raft_consensus.cc:1261] T 2c8c328220e64e168c74fea7619f0c3b P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Refusing update from remote peer b6b1d5677b8d47788b7968f64b3dac3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:40.677857 8224 raft_consensus.cc:1261] T 2c8c328220e64e168c74fea7619f0c3b P 65d4632bcdd54ba596acdbbe1b5ef8af [term 1 LEARNER]: Refusing update from remote peer b6b1d5677b8d47788b7968f64b3dac3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:40.678025 10132 consensus_queue.cc:1035] T 2c8c328220e64e168c74fea7619f0c3b P b6b1d5677b8d47788b7968f64b3dac3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:40.678243 9464 consensus_queue.cc:1035] T 2c8c328220e64e168c74fea7619f0c3b P b6b1d5677b8d47788b7968f64b3dac3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:40.678417 10130 consensus_queue.cc:1035] T 2c8c328220e64e168c74fea7619f0c3b P b6b1d5677b8d47788b7968f64b3dac3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:40.680212 10130 raft_consensus.cc:2932] T 2c8c328220e64e168c74fea7619f0c3b P b6b1d5677b8d47788b7968f64b3dac3c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 65d4632bcdd54ba596acdbbe1b5ef8af (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } }
I20240503 01:59:40.680724 6495 raft_consensus.cc:2932] T 2c8c328220e64e168c74fea7619f0c3b P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 65d4632bcdd54ba596acdbbe1b5ef8af (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } }
I20240503 01:59:40.681950 6625 raft_consensus.cc:2932] T 2c8c328220e64e168c74fea7619f0c3b P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 65d4632bcdd54ba596acdbbe1b5ef8af (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } }
I20240503 01:59:40.682300 8224 raft_consensus.cc:2932] T 2c8c328220e64e168c74fea7619f0c3b P 65d4632bcdd54ba596acdbbe1b5ef8af [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 65d4632bcdd54ba596acdbbe1b5ef8af (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } }
I20240503 01:59:40.682605 6222 catalog_manager.cc:5484] T 2c8c328220e64e168c74fea7619f0c3b P b6b1d5677b8d47788b7968f64b3dac3c reported cstate change: config changed from index 4 to 5, 65d4632bcdd54ba596acdbbe1b5ef8af (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:40.685564 6758 consensus_queue.cc:237] T 2c8c328220e64e168c74fea7619f0c3b P b6b1d5677b8d47788b7968f64b3dac3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } }
I20240503 01:59:40.686599 8224 raft_consensus.cc:1261] T 2c8c328220e64e168c74fea7619f0c3b P 65d4632bcdd54ba596acdbbe1b5ef8af [term 1 FOLLOWER]: Refusing update from remote peer b6b1d5677b8d47788b7968f64b3dac3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:40.686648 6495 raft_consensus.cc:1261] T 2c8c328220e64e168c74fea7619f0c3b P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Refusing update from remote peer b6b1d5677b8d47788b7968f64b3dac3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:40.686920 10132 consensus_queue.cc:1035] T 2c8c328220e64e168c74fea7619f0c3b P b6b1d5677b8d47788b7968f64b3dac3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:40.687155 9464 consensus_queue.cc:1035] T 2c8c328220e64e168c74fea7619f0c3b P b6b1d5677b8d47788b7968f64b3dac3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:40.688624 10130 raft_consensus.cc:2932] T 2c8c328220e64e168c74fea7619f0c3b P b6b1d5677b8d47788b7968f64b3dac3c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } }
I20240503 01:59:40.689136 8224 raft_consensus.cc:2932] T 2c8c328220e64e168c74fea7619f0c3b P 65d4632bcdd54ba596acdbbe1b5ef8af [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } }
I20240503 01:59:40.689945 6209 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2c8c328220e64e168c74fea7619f0c3b with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:40.690146 6495 raft_consensus.cc:2932] T 2c8c328220e64e168c74fea7619f0c3b P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } }
I20240503 01:59:40.691357 6219 catalog_manager.cc:5484] T 2c8c328220e64e168c74fea7619f0c3b P 65d4632bcdd54ba596acdbbe1b5ef8af reported cstate change: config changed from index 5 to 6, VOTER 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } }
I20240503 01:59:40.694201 6607 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c8c328220e64e168c74fea7619f0c3b with delete_type TABLET_DATA_TOMBSTONED (TS 79163c69f6f04a9abd0208d74e1c61be not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:47560
I20240503 01:59:40.694404 10128 tablet_replica.cc:330] T 2c8c328220e64e168c74fea7619f0c3b P 79163c69f6f04a9abd0208d74e1c61be: stopping tablet replica
I20240503 01:59:40.694561 10128 raft_consensus.cc:2229] T 2c8c328220e64e168c74fea7619f0c3b P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:40.694697 10128 raft_consensus.cc:2258] T 2c8c328220e64e168c74fea7619f0c3b P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:40.695360 10128 ts_tablet_manager.cc:1875] T 2c8c328220e64e168c74fea7619f0c3b P 79163c69f6f04a9abd0208d74e1c61be: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:40.697357 10128 ts_tablet_manager.cc:1888] T 2c8c328220e64e168c74fea7619f0c3b P 79163c69f6f04a9abd0208d74e1c61be: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 01:59:40.697432 10128 log.cc:1196] T 2c8c328220e64e168c74fea7619f0c3b P 79163c69f6f04a9abd0208d74e1c61be: Deleting WAL directory at /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/2c8c328220e64e168c74fea7619f0c3b
I20240503 01:59:40.697872 6208 catalog_manager.cc:4830] TS 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131:44005): tablet 2c8c328220e64e168c74fea7619f0c3b (table rebalance_test_table_3 [id=2c07d364168a409a98aed9a7c3c1015c]) successfully deleted
I20240503 01:59:40.704892 10132 raft_consensus.cc:1050] T f11a8034585a4d57be0e9b1e5e8bcd85 P b6b1d5677b8d47788b7968f64b3dac3c: attempting to promote NON_VOTER 9264f24d028347d98f7e4e2d1aedeb79 to VOTER
I20240503 01:59:40.705281 10132 consensus_queue.cc:237] T f11a8034585a4d57be0e9b1e5e8bcd85 P b6b1d5677b8d47788b7968f64b3dac3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } }
I20240503 01:59:40.706588 8093 raft_consensus.cc:1261] T f11a8034585a4d57be0e9b1e5e8bcd85 P 9264f24d028347d98f7e4e2d1aedeb79 [term 1 LEARNER]: Refusing update from remote peer b6b1d5677b8d47788b7968f64b3dac3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:40.706703 6495 raft_consensus.cc:1261] T f11a8034585a4d57be0e9b1e5e8bcd85 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Refusing update from remote peer b6b1d5677b8d47788b7968f64b3dac3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:40.706703 6625 raft_consensus.cc:1261] T f11a8034585a4d57be0e9b1e5e8bcd85 P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Refusing update from remote peer b6b1d5677b8d47788b7968f64b3dac3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:40.707057 10130 consensus_queue.cc:1035] T f11a8034585a4d57be0e9b1e5e8bcd85 P b6b1d5677b8d47788b7968f64b3dac3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:40.707631 10132 consensus_queue.cc:1035] T f11a8034585a4d57be0e9b1e5e8bcd85 P b6b1d5677b8d47788b7968f64b3dac3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:40.707921 10129 consensus_queue.cc:1035] T f11a8034585a4d57be0e9b1e5e8bcd85 P b6b1d5677b8d47788b7968f64b3dac3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:40.709750 10130 raft_consensus.cc:2932] T f11a8034585a4d57be0e9b1e5e8bcd85 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9264f24d028347d98f7e4e2d1aedeb79 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } }
I20240503 01:59:40.710086 6495 raft_consensus.cc:2932] T f11a8034585a4d57be0e9b1e5e8bcd85 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9264f24d028347d98f7e4e2d1aedeb79 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } }
I20240503 01:59:40.710098 8093 raft_consensus.cc:2932] T f11a8034585a4d57be0e9b1e5e8bcd85 P 9264f24d028347d98f7e4e2d1aedeb79 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9264f24d028347d98f7e4e2d1aedeb79 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } }
I20240503 01:59:40.711652 6625 raft_consensus.cc:2932] T f11a8034585a4d57be0e9b1e5e8bcd85 P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9264f24d028347d98f7e4e2d1aedeb79 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } }
I20240503 01:59:40.712445 6218 catalog_manager.cc:5484] T f11a8034585a4d57be0e9b1e5e8bcd85 P 0c4821d9898d42799c5789d4d2da3f63 reported cstate change: config changed from index 4 to 5, 9264f24d028347d98f7e4e2d1aedeb79 (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } }
I20240503 01:59:40.716176 6758 consensus_queue.cc:237] T f11a8034585a4d57be0e9b1e5e8bcd85 P b6b1d5677b8d47788b7968f64b3dac3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } }
I20240503 01:59:40.717556 6495 raft_consensus.cc:1261] T f11a8034585a4d57be0e9b1e5e8bcd85 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Refusing update from remote peer b6b1d5677b8d47788b7968f64b3dac3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:40.717940 10130 consensus_queue.cc:1035] T f11a8034585a4d57be0e9b1e5e8bcd85 P b6b1d5677b8d47788b7968f64b3dac3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:40.719292 8093 raft_consensus.cc:1261] T f11a8034585a4d57be0e9b1e5e8bcd85 P 9264f24d028347d98f7e4e2d1aedeb79 [term 1 FOLLOWER]: Refusing update from remote peer b6b1d5677b8d47788b7968f64b3dac3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:40.720042 10132 consensus_queue.cc:1035] T f11a8034585a4d57be0e9b1e5e8bcd85 P b6b1d5677b8d47788b7968f64b3dac3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:40.720196 10129 raft_consensus.cc:2932] T f11a8034585a4d57be0e9b1e5e8bcd85 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } }
I20240503 01:59:40.720836 6495 raft_consensus.cc:2932] T f11a8034585a4d57be0e9b1e5e8bcd85 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } }
I20240503 01:59:40.720969 8093 raft_consensus.cc:2932] T f11a8034585a4d57be0e9b1e5e8bcd85 P 9264f24d028347d98f7e4e2d1aedeb79 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } }
I20240503 01:59:40.722510 6209 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f11a8034585a4d57be0e9b1e5e8bcd85 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:40.723215 6222 catalog_manager.cc:5484] T f11a8034585a4d57be0e9b1e5e8bcd85 P 9264f24d028347d98f7e4e2d1aedeb79 reported cstate change: config changed from index 5 to 6, VOTER 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } }
I20240503 01:59:40.725553 6607 tablet_service.cc:1514] Processing DeleteTablet for tablet f11a8034585a4d57be0e9b1e5e8bcd85 with delete_type TABLET_DATA_TOMBSTONED (TS 79163c69f6f04a9abd0208d74e1c61be not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:47560
I20240503 01:59:40.725780 10128 tablet_replica.cc:330] T f11a8034585a4d57be0e9b1e5e8bcd85 P 79163c69f6f04a9abd0208d74e1c61be: stopping tablet replica
I20240503 01:59:40.725946 10128 raft_consensus.cc:2229] T f11a8034585a4d57be0e9b1e5e8bcd85 P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:40.726086 10128 raft_consensus.cc:2258] T f11a8034585a4d57be0e9b1e5e8bcd85 P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:40.727552 10128 ts_tablet_manager.cc:1875] T f11a8034585a4d57be0e9b1e5e8bcd85 P 79163c69f6f04a9abd0208d74e1c61be: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:40.729508 10128 ts_tablet_manager.cc:1888] T f11a8034585a4d57be0e9b1e5e8bcd85 P 79163c69f6f04a9abd0208d74e1c61be: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 01:59:40.729895 10128 log.cc:1196] T f11a8034585a4d57be0e9b1e5e8bcd85 P 79163c69f6f04a9abd0208d74e1c61be: Deleting WAL directory at /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/f11a8034585a4d57be0e9b1e5e8bcd85
I20240503 01:59:40.730362 6208 catalog_manager.cc:4830] TS 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131:44005): tablet f11a8034585a4d57be0e9b1e5e8bcd85 (table rebalance_test_table_9 [id=27f301d87e884061bddb2605833c29a6]) successfully deleted
I20240503 01:59:40.734265 10120 raft_consensus.cc:1050] T 67150a2f5cb147b8b57a8c8040b567e1 P 4127249ba42644388d225d614c935a25: attempting to promote NON_VOTER 65d4632bcdd54ba596acdbbe1b5ef8af to VOTER
I20240503 01:59:40.734737 10120 consensus_queue.cc:237] T 67150a2f5cb147b8b57a8c8040b567e1 P 4127249ba42644388d225d614c935a25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } }
I20240503 01:59:40.735958 6625 raft_consensus.cc:1261] T 67150a2f5cb147b8b57a8c8040b567e1 P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Refusing update from remote peer 4127249ba42644388d225d614c935a25: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:40.736353 6495 raft_consensus.cc:1261] T 67150a2f5cb147b8b57a8c8040b567e1 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Refusing update from remote peer 4127249ba42644388d225d614c935a25: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:40.736371 10120 consensus_queue.cc:1035] T 67150a2f5cb147b8b57a8c8040b567e1 P 4127249ba42644388d225d614c935a25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:40.736690 8224 raft_consensus.cc:1261] T 67150a2f5cb147b8b57a8c8040b567e1 P 65d4632bcdd54ba596acdbbe1b5ef8af [term 1 LEARNER]: Refusing update from remote peer 4127249ba42644388d225d614c935a25: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:40.736812 9773 consensus_queue.cc:1035] T 67150a2f5cb147b8b57a8c8040b567e1 P 4127249ba42644388d225d614c935a25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:40.737123 10120 consensus_queue.cc:1035] T 67150a2f5cb147b8b57a8c8040b567e1 P 4127249ba42644388d225d614c935a25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:40.739897 8224 raft_consensus.cc:2932] T 67150a2f5cb147b8b57a8c8040b567e1 P 65d4632bcdd54ba596acdbbe1b5ef8af [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 65d4632bcdd54ba596acdbbe1b5ef8af (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } }
I20240503 01:59:40.739899 10117 raft_consensus.cc:2932] T 67150a2f5cb147b8b57a8c8040b567e1 P 4127249ba42644388d225d614c935a25 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 65d4632bcdd54ba596acdbbe1b5ef8af (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } }
I20240503 01:59:40.741562 6625 raft_consensus.cc:2932] T 67150a2f5cb147b8b57a8c8040b567e1 P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 65d4632bcdd54ba596acdbbe1b5ef8af (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } }
I20240503 01:59:40.742035 6495 raft_consensus.cc:2932] T 67150a2f5cb147b8b57a8c8040b567e1 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 65d4632bcdd54ba596acdbbe1b5ef8af (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } }
I20240503 01:59:40.742280 6218 catalog_manager.cc:5484] T 67150a2f5cb147b8b57a8c8040b567e1 P 65d4632bcdd54ba596acdbbe1b5ef8af reported cstate change: config changed from index 4 to 5, 65d4632bcdd54ba596acdbbe1b5ef8af (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4127249ba42644388d225d614c935a25" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } }
I20240503 01:59:40.745580 6365 consensus_queue.cc:237] T 67150a2f5cb147b8b57a8c8040b567e1 P 4127249ba42644388d225d614c935a25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } }
I20240503 01:59:40.746606 8224 raft_consensus.cc:1261] T 67150a2f5cb147b8b57a8c8040b567e1 P 65d4632bcdd54ba596acdbbe1b5ef8af [term 1 FOLLOWER]: Refusing update from remote peer 4127249ba42644388d225d614c935a25: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:40.746970 10120 consensus_queue.cc:1035] T 67150a2f5cb147b8b57a8c8040b567e1 P 4127249ba42644388d225d614c935a25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:40.747447 6625 raft_consensus.cc:1261] T 67150a2f5cb147b8b57a8c8040b567e1 P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Refusing update from remote peer 4127249ba42644388d225d614c935a25: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:40.748005 10121 consensus_queue.cc:1035] T 67150a2f5cb147b8b57a8c8040b567e1 P 4127249ba42644388d225d614c935a25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:40.749286 10121 raft_consensus.cc:2932] T 67150a2f5cb147b8b57a8c8040b567e1 P 4127249ba42644388d225d614c935a25 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } }
I20240503 01:59:40.749606 8224 raft_consensus.cc:2932] T 67150a2f5cb147b8b57a8c8040b567e1 P 65d4632bcdd54ba596acdbbe1b5ef8af [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } }
I20240503 01:59:40.750721 6208 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 67150a2f5cb147b8b57a8c8040b567e1 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:40.751750 6214 catalog_manager.cc:5484] T 67150a2f5cb147b8b57a8c8040b567e1 P 4127249ba42644388d225d614c935a25 reported cstate change: config changed from index 5 to 6, VOTER 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "4127249ba42644388d225d614c935a25" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:40.751736 6625 raft_consensus.cc:2932] T 67150a2f5cb147b8b57a8c8040b567e1 P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } }
I20240503 01:59:40.754477 6476 tablet_service.cc:1514] Processing DeleteTablet for tablet 67150a2f5cb147b8b57a8c8040b567e1 with delete_type TABLET_DATA_TOMBSTONED (TS 0c4821d9898d42799c5789d4d2da3f63 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:39850
I20240503 01:59:40.754729 10112 tablet_replica.cc:330] T 67150a2f5cb147b8b57a8c8040b567e1 P 0c4821d9898d42799c5789d4d2da3f63: stopping tablet replica
I20240503 01:59:40.754884 10112 raft_consensus.cc:2229] T 67150a2f5cb147b8b57a8c8040b567e1 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:40.755038 10112 raft_consensus.cc:2258] T 67150a2f5cb147b8b57a8c8040b567e1 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:40.755729 10112 ts_tablet_manager.cc:1875] T 67150a2f5cb147b8b57a8c8040b567e1 P 0c4821d9898d42799c5789d4d2da3f63: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:40.757825 10112 ts_tablet_manager.cc:1888] T 67150a2f5cb147b8b57a8c8040b567e1 P 0c4821d9898d42799c5789d4d2da3f63: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 01:59:40.757910 10112 log.cc:1196] T 67150a2f5cb147b8b57a8c8040b567e1 P 0c4821d9898d42799c5789d4d2da3f63: Deleting WAL directory at /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/67150a2f5cb147b8b57a8c8040b567e1
I20240503 01:59:40.758498 6210 catalog_manager.cc:4830] TS 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130:34189): tablet 67150a2f5cb147b8b57a8c8040b567e1 (table rebalance_test_table_9 [id=27f301d87e884061bddb2605833c29a6]) successfully deleted
I20240503 01:59:40.784260 10120 raft_consensus.cc:1050] T 470b553976064bdd8c321799ac718274 P 4127249ba42644388d225d614c935a25: attempting to promote NON_VOTER 65d4632bcdd54ba596acdbbe1b5ef8af to VOTER
I20240503 01:59:40.784720 10120 consensus_queue.cc:237] T 470b553976064bdd8c321799ac718274 P 4127249ba42644388d225d614c935a25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } }
I20240503 01:59:40.786222 6495 raft_consensus.cc:1261] T 470b553976064bdd8c321799ac718274 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Refusing update from remote peer 4127249ba42644388d225d614c935a25: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:40.786350 6758 raft_consensus.cc:1261] T 470b553976064bdd8c321799ac718274 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Refusing update from remote peer 4127249ba42644388d225d614c935a25: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:40.786690 10120 consensus_queue.cc:1035] T 470b553976064bdd8c321799ac718274 P 4127249ba42644388d225d614c935a25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:40.787062 8224 raft_consensus.cc:1261] T 470b553976064bdd8c321799ac718274 P 65d4632bcdd54ba596acdbbe1b5ef8af [term 1 LEARNER]: Refusing update from remote peer 4127249ba42644388d225d614c935a25: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:40.787308 10117 consensus_queue.cc:1035] T 470b553976064bdd8c321799ac718274 P 4127249ba42644388d225d614c935a25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:40.787921 10117 consensus_queue.cc:1035] T 470b553976064bdd8c321799ac718274 P 4127249ba42644388d225d614c935a25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:40.789244 10121 raft_consensus.cc:2932] T 470b553976064bdd8c321799ac718274 P 4127249ba42644388d225d614c935a25 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 65d4632bcdd54ba596acdbbe1b5ef8af (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } }
I20240503 01:59:40.791069 6495 raft_consensus.cc:2932] T 470b553976064bdd8c321799ac718274 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 65d4632bcdd54ba596acdbbe1b5ef8af (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } }
I20240503 01:59:40.791746 6758 raft_consensus.cc:2932] T 470b553976064bdd8c321799ac718274 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 65d4632bcdd54ba596acdbbe1b5ef8af (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } }
I20240503 01:59:40.794023 8224 raft_consensus.cc:2932] T 470b553976064bdd8c321799ac718274 P 65d4632bcdd54ba596acdbbe1b5ef8af [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 65d4632bcdd54ba596acdbbe1b5ef8af (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } }
I20240503 01:59:40.794062 6222 catalog_manager.cc:5484] T 470b553976064bdd8c321799ac718274 P 0c4821d9898d42799c5789d4d2da3f63 reported cstate change: config changed from index 4 to 5, 65d4632bcdd54ba596acdbbe1b5ef8af (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4127249ba42644388d225d614c935a25" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } }
I20240503 01:59:40.798627 6365 consensus_queue.cc:237] T 470b553976064bdd8c321799ac718274 P 4127249ba42644388d225d614c935a25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } }
I20240503 01:59:40.800014 8224 raft_consensus.cc:1261] T 470b553976064bdd8c321799ac718274 P 65d4632bcdd54ba596acdbbe1b5ef8af [term 1 FOLLOWER]: Refusing update from remote peer 4127249ba42644388d225d614c935a25: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:40.800333 10120 consensus_queue.cc:1035] T 470b553976064bdd8c321799ac718274 P 4127249ba42644388d225d614c935a25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:40.801088 6495 raft_consensus.cc:1261] T 470b553976064bdd8c321799ac718274 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Refusing update from remote peer 4127249ba42644388d225d614c935a25: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:40.801596 10120 consensus_queue.cc:1035] T 470b553976064bdd8c321799ac718274 P 4127249ba42644388d225d614c935a25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:40.803521 10117 raft_consensus.cc:2932] T 470b553976064bdd8c321799ac718274 P 4127249ba42644388d225d614c935a25 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b6b1d5677b8d47788b7968f64b3dac3c (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } }
I20240503 01:59:40.803882 8224 raft_consensus.cc:2932] T 470b553976064bdd8c321799ac718274 P 65d4632bcdd54ba596acdbbe1b5ef8af [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b6b1d5677b8d47788b7968f64b3dac3c (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } }
I20240503 01:59:40.804893 6208 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 470b553976064bdd8c321799ac718274 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:40.806023 6495 raft_consensus.cc:2932] T 470b553976064bdd8c321799ac718274 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b6b1d5677b8d47788b7968f64b3dac3c (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } }
I20240503 01:59:40.806341 6219 catalog_manager.cc:5484] T 470b553976064bdd8c321799ac718274 P 4127249ba42644388d225d614c935a25 reported cstate change: config changed from index 5 to 6, VOTER b6b1d5677b8d47788b7968f64b3dac3c (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "4127249ba42644388d225d614c935a25" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:40.808933 6738 tablet_service.cc:1514] Processing DeleteTablet for tablet 470b553976064bdd8c321799ac718274 with delete_type TABLET_DATA_TOMBSTONED (TS b6b1d5677b8d47788b7968f64b3dac3c not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:51980
I20240503 01:59:40.809471 10222 tablet_replica.cc:330] T 470b553976064bdd8c321799ac718274 P b6b1d5677b8d47788b7968f64b3dac3c: stopping tablet replica
I20240503 01:59:40.809979 10222 raft_consensus.cc:2229] T 470b553976064bdd8c321799ac718274 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:40.810261 10222 raft_consensus.cc:2258] T 470b553976064bdd8c321799ac718274 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:40.811616 10222 ts_tablet_manager.cc:1875] T 470b553976064bdd8c321799ac718274 P b6b1d5677b8d47788b7968f64b3dac3c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:40.813465 10222 ts_tablet_manager.cc:1888] T 470b553976064bdd8c321799ac718274 P b6b1d5677b8d47788b7968f64b3dac3c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 01:59:40.813545 10222 log.cc:1196] T 470b553976064bdd8c321799ac718274 P b6b1d5677b8d47788b7968f64b3dac3c: Deleting WAL directory at /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/470b553976064bdd8c321799ac718274
I20240503 01:59:40.814213 6209 catalog_manager.cc:4830] TS b6b1d5677b8d47788b7968f64b3dac3c (127.0.99.132:46067): tablet 470b553976064bdd8c321799ac718274 (table rebalance_test_table_2 [id=4a248d80763944458620bf40775e2098]) successfully deleted
I20240503 01:59:40.847600 6365 consensus_queue.cc:237] T a4983426b2d843e7a9ed335c79178678 P 4127249ba42644388d225d614c935a25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: true } }
I20240503 01:59:40.848932 6495 raft_consensus.cc:1261] T a4983426b2d843e7a9ed335c79178678 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Refusing update from remote peer 4127249ba42644388d225d614c935a25: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:40.849113 6758 raft_consensus.cc:1261] T a4983426b2d843e7a9ed335c79178678 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Refusing update from remote peer 4127249ba42644388d225d614c935a25: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:40.849303 10117 consensus_queue.cc:1035] T a4983426b2d843e7a9ed335c79178678 P 4127249ba42644388d225d614c935a25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 01:59:40.849474 6298 consensus_peers.cc:485] T a4983426b2d843e7a9ed335c79178678 P 4127249ba42644388d225d614c935a25 -> Peer 65d4632bcdd54ba596acdbbe1b5ef8af (127.0.99.134:35795): Couldn't send request to peer 65d4632bcdd54ba596acdbbe1b5ef8af. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a4983426b2d843e7a9ed335c79178678. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:40.849575 9773 consensus_queue.cc:1035] T a4983426b2d843e7a9ed335c79178678 P 4127249ba42644388d225d614c935a25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:40.851230 8701 raft_consensus.cc:2932] T a4983426b2d843e7a9ed335c79178678 P 4127249ba42644388d225d614c935a25 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 65d4632bcdd54ba596acdbbe1b5ef8af (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: true } } }
I20240503 01:59:40.851581 6495 raft_consensus.cc:2932] T a4983426b2d843e7a9ed335c79178678 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 65d4632bcdd54ba596acdbbe1b5ef8af (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: true } } }
I20240503 01:59:40.852919 6758 raft_consensus.cc:2932] T a4983426b2d843e7a9ed335c79178678 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 65d4632bcdd54ba596acdbbe1b5ef8af (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: true } } }
I20240503 01:59:40.853915 6219 catalog_manager.cc:5484] T a4983426b2d843e7a9ed335c79178678 P 0c4821d9898d42799c5789d4d2da3f63 reported cstate change: config changed from index -1 to 4, NON_VOTER 65d4632bcdd54ba596acdbbe1b5ef8af (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "4127249ba42644388d225d614c935a25" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: true } } }
I20240503 01:59:40.863327 6495 consensus_queue.cc:237] T 897e88ac93484a11829189e027de8bae P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: true } }
I20240503 01:59:40.864759 6625 raft_consensus.cc:1261] T 897e88ac93484a11829189e027de8bae P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Refusing update from remote peer 0c4821d9898d42799c5789d4d2da3f63: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 01:59:40.864832 6432 consensus_peers.cc:485] T 897e88ac93484a11829189e027de8bae P 0c4821d9898d42799c5789d4d2da3f63 -> Peer de2f076398ee44008622fb5db2a75952 (127.0.99.135:34035): Couldn't send request to peer de2f076398ee44008622fb5db2a75952. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 897e88ac93484a11829189e027de8bae. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:40.864977 6758 raft_consensus.cc:1261] T 897e88ac93484a11829189e027de8bae P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Refusing update from remote peer 0c4821d9898d42799c5789d4d2da3f63: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:40.865149 9931 consensus_queue.cc:1035] T 897e88ac93484a11829189e027de8bae P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:40.865339 9923 consensus_queue.cc:1035] T 897e88ac93484a11829189e027de8bae P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:40.867204 9924 raft_consensus.cc:2932] T 897e88ac93484a11829189e027de8bae P 0c4821d9898d42799c5789d4d2da3f63 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER de2f076398ee44008622fb5db2a75952 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: true } } }
I20240503 01:59:40.867666 6758 raft_consensus.cc:2932] T 897e88ac93484a11829189e027de8bae P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER de2f076398ee44008622fb5db2a75952 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: true } } }
I20240503 01:59:40.868935 6625 raft_consensus.cc:2932] T 897e88ac93484a11829189e027de8bae P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER de2f076398ee44008622fb5db2a75952 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: true } } }
I20240503 01:59:40.870096 6222 catalog_manager.cc:5484] T 897e88ac93484a11829189e027de8bae P b6b1d5677b8d47788b7968f64b3dac3c reported cstate change: config changed from index -1 to 4, NON_VOTER de2f076398ee44008622fb5db2a75952 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "0c4821d9898d42799c5789d4d2da3f63" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: true } } }
I20240503 01:59:40.875025 9931 raft_consensus.cc:1050] T 2ac8004211e64c3481e75377092fa93b P 0c4821d9898d42799c5789d4d2da3f63: attempting to promote NON_VOTER 9264f24d028347d98f7e4e2d1aedeb79 to VOTER
I20240503 01:59:40.875378 9931 consensus_queue.cc:237] T 2ac8004211e64c3481e75377092fa93b P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } }
I20240503 01:59:40.876797 6758 raft_consensus.cc:1261] T 2ac8004211e64c3481e75377092fa93b P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Refusing update from remote peer 0c4821d9898d42799c5789d4d2da3f63: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:40.876839 6365 raft_consensus.cc:1261] T 2ac8004211e64c3481e75377092fa93b P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Refusing update from remote peer 0c4821d9898d42799c5789d4d2da3f63: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:40.877204 9923 consensus_queue.cc:1035] T 2ac8004211e64c3481e75377092fa93b P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:40.877496 8093 raft_consensus.cc:1261] T 2ac8004211e64c3481e75377092fa93b P 9264f24d028347d98f7e4e2d1aedeb79 [term 1 LEARNER]: Refusing update from remote peer 0c4821d9898d42799c5789d4d2da3f63: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:40.877554 7917 consensus_queue.cc:1035] T 2ac8004211e64c3481e75377092fa93b P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:40.878362 7917 consensus_queue.cc:1035] T 2ac8004211e64c3481e75377092fa93b P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:40.880518 9923 raft_consensus.cc:2932] T 2ac8004211e64c3481e75377092fa93b P 0c4821d9898d42799c5789d4d2da3f63 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9264f24d028347d98f7e4e2d1aedeb79 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } }
I20240503 01:59:40.881124 6758 raft_consensus.cc:2932] T 2ac8004211e64c3481e75377092fa93b P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9264f24d028347d98f7e4e2d1aedeb79 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } }
I20240503 01:59:40.881837 8093 raft_consensus.cc:2932] T 2ac8004211e64c3481e75377092fa93b P 9264f24d028347d98f7e4e2d1aedeb79 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9264f24d028347d98f7e4e2d1aedeb79 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } }
I20240503 01:59:40.882513 6365 raft_consensus.cc:2932] T 2ac8004211e64c3481e75377092fa93b P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9264f24d028347d98f7e4e2d1aedeb79 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } }
I20240503 01:59:40.885022 6625 consensus_queue.cc:237] T 5b599bd1fea6409fbe485c7a35c0891c P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: true } }
I20240503 01:59:40.885262 6219 catalog_manager.cc:5484] T 2ac8004211e64c3481e75377092fa93b P 0c4821d9898d42799c5789d4d2da3f63 reported cstate change: config changed from index 4 to 5, 9264f24d028347d98f7e4e2d1aedeb79 (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0c4821d9898d42799c5789d4d2da3f63" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:40.886497 6495 raft_consensus.cc:1261] T 5b599bd1fea6409fbe485c7a35c0891c P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:40.886847 6365 raft_consensus.cc:1261] T 5b599bd1fea6409fbe485c7a35c0891c P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:40.887223 10101 consensus_queue.cc:1035] T 5b599bd1fea6409fbe485c7a35c0891c P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:40.887387 10108 consensus_queue.cc:1035] T 5b599bd1fea6409fbe485c7a35c0891c P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 01:59:40.887615 6561 consensus_peers.cc:485] T 5b599bd1fea6409fbe485c7a35c0891c P 79163c69f6f04a9abd0208d74e1c61be -> Peer 7a1404fe1c1b4b388b821c8261b53104 (127.0.99.136:46759): Couldn't send request to peer 7a1404fe1c1b4b388b821c8261b53104. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5b599bd1fea6409fbe485c7a35c0891c. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:40.889159 6495 consensus_queue.cc:237] T 2ac8004211e64c3481e75377092fa93b P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } }
I20240503 01:59:40.890336 6363 raft_consensus.cc:1261] T 2ac8004211e64c3481e75377092fa93b P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Refusing update from remote peer 0c4821d9898d42799c5789d4d2da3f63: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:40.890427 8093 raft_consensus.cc:1261] T 2ac8004211e64c3481e75377092fa93b P 9264f24d028347d98f7e4e2d1aedeb79 [term 1 FOLLOWER]: Refusing update from remote peer 0c4821d9898d42799c5789d4d2da3f63: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:40.890522 10101 raft_consensus.cc:2932] T 5b599bd1fea6409fbe485c7a35c0891c P 79163c69f6f04a9abd0208d74e1c61be [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 7a1404fe1c1b4b388b821c8261b53104 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: true } } }
I20240503 01:59:40.890673 9923 consensus_queue.cc:1035] T 2ac8004211e64c3481e75377092fa93b P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:40.891556 9923 consensus_queue.cc:1035] T 2ac8004211e64c3481e75377092fa93b P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:40.892401 6363 raft_consensus.cc:2932] T 5b599bd1fea6409fbe485c7a35c0891c P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 7a1404fe1c1b4b388b821c8261b53104 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: true } } }
I20240503 01:59:40.892663 9923 raft_consensus.cc:2932] T 2ac8004211e64c3481e75377092fa93b P 0c4821d9898d42799c5789d4d2da3f63 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b6b1d5677b8d47788b7968f64b3dac3c (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } }
I20240503 01:59:40.892881 6494 raft_consensus.cc:2932] T 5b599bd1fea6409fbe485c7a35c0891c P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 7a1404fe1c1b4b388b821c8261b53104 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: true } } }
I20240503 01:59:40.893414 6218 catalog_manager.cc:5484] T 5b599bd1fea6409fbe485c7a35c0891c P 79163c69f6f04a9abd0208d74e1c61be reported cstate change: config changed from index -1 to 4, NON_VOTER 7a1404fe1c1b4b388b821c8261b53104 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "79163c69f6f04a9abd0208d74e1c61be" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:40.894412 6210 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2ac8004211e64c3481e75377092fa93b with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:40.895666 6363 raft_consensus.cc:2932] T 2ac8004211e64c3481e75377092fa93b P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b6b1d5677b8d47788b7968f64b3dac3c (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } }
I20240503 01:59:40.896401 8093 raft_consensus.cc:2932] T 2ac8004211e64c3481e75377092fa93b P 9264f24d028347d98f7e4e2d1aedeb79 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b6b1d5677b8d47788b7968f64b3dac3c (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } }
I20240503 01:59:40.898324 6222 catalog_manager.cc:5484] T 2ac8004211e64c3481e75377092fa93b P 0c4821d9898d42799c5789d4d2da3f63 reported cstate change: config changed from index 5 to 6, VOTER b6b1d5677b8d47788b7968f64b3dac3c (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "0c4821d9898d42799c5789d4d2da3f63" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:40.900740 6738 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ac8004211e64c3481e75377092fa93b with delete_type TABLET_DATA_TOMBSTONED (TS b6b1d5677b8d47788b7968f64b3dac3c not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:51980
I20240503 01:59:40.900980 10222 tablet_replica.cc:330] T 2ac8004211e64c3481e75377092fa93b P b6b1d5677b8d47788b7968f64b3dac3c: stopping tablet replica
I20240503 01:59:40.901114 10222 raft_consensus.cc:2229] T 2ac8004211e64c3481e75377092fa93b P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:40.901262 10222 raft_consensus.cc:2258] T 2ac8004211e64c3481e75377092fa93b P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:40.901897 10222 ts_tablet_manager.cc:1875] T 2ac8004211e64c3481e75377092fa93b P b6b1d5677b8d47788b7968f64b3dac3c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:40.903841 9923 raft_consensus.cc:1050] T 1d392e08c15a4d4098a350287b3163ef P 0c4821d9898d42799c5789d4d2da3f63: attempting to promote NON_VOTER de2f076398ee44008622fb5db2a75952 to VOTER
I20240503 01:59:40.904067 10222 ts_tablet_manager.cc:1888] T 2ac8004211e64c3481e75377092fa93b P b6b1d5677b8d47788b7968f64b3dac3c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 01:59:40.904137 10222 log.cc:1196] T 2ac8004211e64c3481e75377092fa93b P b6b1d5677b8d47788b7968f64b3dac3c: Deleting WAL directory at /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/2ac8004211e64c3481e75377092fa93b
I20240503 01:59:40.904297 9923 consensus_queue.cc:237] T 1d392e08c15a4d4098a350287b3163ef P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } }
I20240503 01:59:40.904860 6209 catalog_manager.cc:4830] TS b6b1d5677b8d47788b7968f64b3dac3c (127.0.99.132:46067): tablet 2ac8004211e64c3481e75377092fa93b (table rebalance_test_table_4 [id=d52e5f2a44ec4f12b7ef6d85bd14d4ac]) successfully deleted
I20240503 01:59:40.906445 6363 consensus_queue.cc:237] T 885b787d0cbb4ddabd47cb7c42a78c74 P 4127249ba42644388d225d614c935a25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: true } }
I20240503 01:59:40.906703 8355 raft_consensus.cc:1261] T 1d392e08c15a4d4098a350287b3163ef P de2f076398ee44008622fb5db2a75952 [term 1 LEARNER]: Refusing update from remote peer 0c4821d9898d42799c5789d4d2da3f63: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:40.907193 10117 raft_consensus.cc:1050] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 4127249ba42644388d225d614c935a25: attempting to promote NON_VOTER 9264f24d028347d98f7e4e2d1aedeb79 to VOTER
I20240503 01:59:40.907533 10117 consensus_queue.cc:237] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 4127249ba42644388d225d614c935a25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } }
I20240503 01:59:40.907563 7917 consensus_queue.cc:1035] T 1d392e08c15a4d4098a350287b3163ef P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:40.907763 6758 raft_consensus.cc:1261] T 885b787d0cbb4ddabd47cb7c42a78c74 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Refusing update from remote peer 4127249ba42644388d225d614c935a25: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:40.908340 6758 raft_consensus.cc:1261] T 1d392e08c15a4d4098a350287b3163ef P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Refusing update from remote peer 0c4821d9898d42799c5789d4d2da3f63: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:40.908730 7917 consensus_queue.cc:1035] T 1d392e08c15a4d4098a350287b3163ef P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:40.908993 10117 consensus_queue.cc:1035] T 885b787d0cbb4ddabd47cb7c42a78c74 P 4127249ba42644388d225d614c935a25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:40.909468 6494 raft_consensus.cc:1261] T 885b787d0cbb4ddabd47cb7c42a78c74 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Refusing update from remote peer 4127249ba42644388d225d614c935a25: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:40.909492 8080 raft_consensus.cc:1261] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 9264f24d028347d98f7e4e2d1aedeb79 [term 1 LEARNER]: Refusing update from remote peer 4127249ba42644388d225d614c935a25: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240503 01:59:40.910171 6298 consensus_peers.cc:485] T 885b787d0cbb4ddabd47cb7c42a78c74 P 4127249ba42644388d225d614c935a25 -> Peer 9264f24d028347d98f7e4e2d1aedeb79 (127.0.99.133:41713): Couldn't send request to peer 9264f24d028347d98f7e4e2d1aedeb79. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 885b787d0cbb4ddabd47cb7c42a78c74. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:40.910212 6494 raft_consensus.cc:1261] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Refusing update from remote peer 4127249ba42644388d225d614c935a25: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:40.910356 10121 consensus_queue.cc:1035] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 4127249ba42644388d225d614c935a25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:40.910841 10121 consensus_queue.cc:1035] T 885b787d0cbb4ddabd47cb7c42a78c74 P 4127249ba42644388d225d614c935a25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:40.911758 6625 raft_consensus.cc:1261] T 1d392e08c15a4d4098a350287b3163ef P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Refusing update from remote peer 0c4821d9898d42799c5789d4d2da3f63: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:40.912108 10121 consensus_queue.cc:1035] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 4127249ba42644388d225d614c935a25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:40.912111 7917 consensus_queue.cc:1035] T 1d392e08c15a4d4098a350287b3163ef P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:40.912866 6625 raft_consensus.cc:1261] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Refusing update from remote peer 4127249ba42644388d225d614c935a25: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:40.913180 10120 consensus_queue.cc:1035] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 4127249ba42644388d225d614c935a25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:40.913357 10121 raft_consensus.cc:2932] T 885b787d0cbb4ddabd47cb7c42a78c74 P 4127249ba42644388d225d614c935a25 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9264f24d028347d98f7e4e2d1aedeb79 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: true } } }
I20240503 01:59:40.914378 9924 raft_consensus.cc:2932] T 1d392e08c15a4d4098a350287b3163ef P 0c4821d9898d42799c5789d4d2da3f63 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, de2f076398ee44008622fb5db2a75952 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } }
I20240503 01:59:40.915185 10120 raft_consensus.cc:2932] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 4127249ba42644388d225d614c935a25 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9264f24d028347d98f7e4e2d1aedeb79 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } }
I20240503 01:59:40.915350 6758 raft_consensus.cc:2932] T 885b787d0cbb4ddabd47cb7c42a78c74 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9264f24d028347d98f7e4e2d1aedeb79 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: true } } }
I20240503 01:59:40.917349 8080 raft_consensus.cc:2932] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 9264f24d028347d98f7e4e2d1aedeb79 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9264f24d028347d98f7e4e2d1aedeb79 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } }
I20240503 01:59:40.917407 6218 catalog_manager.cc:5484] T 1d392e08c15a4d4098a350287b3163ef P 0c4821d9898d42799c5789d4d2da3f63 reported cstate change: config changed from index 4 to 5, de2f076398ee44008622fb5db2a75952 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0c4821d9898d42799c5789d4d2da3f63" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:40.918130 6494 raft_consensus.cc:2932] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9264f24d028347d98f7e4e2d1aedeb79 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } }
I20240503 01:59:40.919214 8355 raft_consensus.cc:2932] T 1d392e08c15a4d4098a350287b3163ef P de2f076398ee44008622fb5db2a75952 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, de2f076398ee44008622fb5db2a75952 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } }
I20240503 01:59:40.920372 6219 catalog_manager.cc:5484] T 885b787d0cbb4ddabd47cb7c42a78c74 P 4127249ba42644388d225d614c935a25 reported cstate change: config changed from index -1 to 4, NON_VOTER 9264f24d028347d98f7e4e2d1aedeb79 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "4127249ba42644388d225d614c935a25" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:40.920373 6495 raft_consensus.cc:2932] T 885b787d0cbb4ddabd47cb7c42a78c74 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9264f24d028347d98f7e4e2d1aedeb79 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: true } } }
I20240503 01:59:40.921156 6214 catalog_manager.cc:5484] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 9264f24d028347d98f7e4e2d1aedeb79 reported cstate change: config changed from index 4 to 5, 9264f24d028347d98f7e4e2d1aedeb79 (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4127249ba42644388d225d614c935a25" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } }
I20240503 01:59:40.921459 6627 raft_consensus.cc:2932] T 1d392e08c15a4d4098a350287b3163ef P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, de2f076398ee44008622fb5db2a75952 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } }
I20240503 01:59:40.923604 6758 raft_consensus.cc:2932] T 1d392e08c15a4d4098a350287b3163ef P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, de2f076398ee44008622fb5db2a75952 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } }
I20240503 01:59:40.925724 6495 consensus_queue.cc:237] T 1d392e08c15a4d4098a350287b3163ef P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } }
I20240503 01:59:40.926877 6757 raft_consensus.cc:1261] T 1d392e08c15a4d4098a350287b3163ef P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Refusing update from remote peer 0c4821d9898d42799c5789d4d2da3f63: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:40.927708 9923 consensus_queue.cc:1035] T 1d392e08c15a4d4098a350287b3163ef P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:40.928009 6625 raft_consensus.cc:2932] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9264f24d028347d98f7e4e2d1aedeb79 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } }
I20240503 01:59:40.930470 8355 raft_consensus.cc:1261] T 1d392e08c15a4d4098a350287b3163ef P de2f076398ee44008622fb5db2a75952 [term 1 FOLLOWER]: Refusing update from remote peer 0c4821d9898d42799c5789d4d2da3f63: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:40.930827 9923 consensus_queue.cc:1035] T 1d392e08c15a4d4098a350287b3163ef P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:40.931160 9924 raft_consensus.cc:2932] T 1d392e08c15a4d4098a350287b3163ef P 0c4821d9898d42799c5789d4d2da3f63 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } }
I20240503 01:59:40.931710 8355 raft_consensus.cc:2932] T 1d392e08c15a4d4098a350287b3163ef P de2f076398ee44008622fb5db2a75952 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } }
I20240503 01:59:40.932034 6363 consensus_queue.cc:237] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 4127249ba42644388d225d614c935a25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } }
I20240503 01:59:40.933142 6210 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1d392e08c15a4d4098a350287b3163ef with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:40.933631 8080 raft_consensus.cc:1261] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 9264f24d028347d98f7e4e2d1aedeb79 [term 1 FOLLOWER]: Refusing update from remote peer 4127249ba42644388d225d614c935a25: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:40.933748 6222 catalog_manager.cc:5484] T 1d392e08c15a4d4098a350287b3163ef P 0c4821d9898d42799c5789d4d2da3f63 reported cstate change: config changed from index 5 to 6, VOTER 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "0c4821d9898d42799c5789d4d2da3f63" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:40.934029 6757 raft_consensus.cc:2932] T 1d392e08c15a4d4098a350287b3163ef P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } }
I20240503 01:59:40.934211 6495 consensus_queue.cc:237] T ccb34a4b52244d90850ab04e21299f55 P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: true } }
I20240503 01:59:40.934795 10117 consensus_queue.cc:1035] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 4127249ba42644388d225d614c935a25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:40.936143 6607 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d392e08c15a4d4098a350287b3163ef with delete_type TABLET_DATA_TOMBSTONED (TS 79163c69f6f04a9abd0208d74e1c61be not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:47560
I20240503 01:59:40.936383 6625 raft_consensus.cc:1261] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Refusing update from remote peer 4127249ba42644388d225d614c935a25: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:40.936518 10128 tablet_replica.cc:330] T 1d392e08c15a4d4098a350287b3163ef P 79163c69f6f04a9abd0208d74e1c61be: stopping tablet replica
I20240503 01:59:40.936697 10128 raft_consensus.cc:2229] T 1d392e08c15a4d4098a350287b3163ef P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:40.936859 10128 raft_consensus.cc:2258] T 1d392e08c15a4d4098a350287b3163ef P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:40.937450 10117 consensus_queue.cc:1035] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 4127249ba42644388d225d614c935a25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:40.938347 10128 ts_tablet_manager.cc:1875] T 1d392e08c15a4d4098a350287b3163ef P 79163c69f6f04a9abd0208d74e1c61be: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:40.938457 10121 raft_consensus.cc:2932] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 4127249ba42644388d225d614c935a25 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } }
I20240503 01:59:40.938966 8080 raft_consensus.cc:2932] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 9264f24d028347d98f7e4e2d1aedeb79 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } }
I20240503 01:59:40.940575 10128 ts_tablet_manager.cc:1888] T 1d392e08c15a4d4098a350287b3163ef P 79163c69f6f04a9abd0208d74e1c61be: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 01:59:40.940699 10128 log.cc:1196] T 1d392e08c15a4d4098a350287b3163ef P 79163c69f6f04a9abd0208d74e1c61be: Deleting WAL directory at /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/1d392e08c15a4d4098a350287b3163ef
I20240503 01:59:40.940966 6219 catalog_manager.cc:5484] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 4127249ba42644388d225d614c935a25 reported cstate change: config changed from index 5 to 6, VOTER 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "4127249ba42644388d225d614c935a25" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:40.941236 6208 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2088c84e4eef4ccfbc70d7b668b8f2cf with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:40.941524 6363 raft_consensus.cc:1261] T ccb34a4b52244d90850ab04e21299f55 P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Refusing update from remote peer 0c4821d9898d42799c5789d4d2da3f63: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 01:59:40.941859 6430 consensus_peers.cc:485] T ccb34a4b52244d90850ab04e21299f55 P 0c4821d9898d42799c5789d4d2da3f63 -> Peer 7a1404fe1c1b4b388b821c8261b53104 (127.0.99.136:46759): Couldn't send request to peer 7a1404fe1c1b4b388b821c8261b53104. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ccb34a4b52244d90850ab04e21299f55. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:40.942029 6625 raft_consensus.cc:1261] T ccb34a4b52244d90850ab04e21299f55 P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Refusing update from remote peer 0c4821d9898d42799c5789d4d2da3f63: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:40.942065 9931 consensus_queue.cc:1035] T ccb34a4b52244d90850ab04e21299f55 P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:40.942399 9923 consensus_queue.cc:1035] T ccb34a4b52244d90850ab04e21299f55 P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:40.943835 6476 tablet_service.cc:1514] Processing DeleteTablet for tablet 2088c84e4eef4ccfbc70d7b668b8f2cf with delete_type TABLET_DATA_TOMBSTONED (TS 0c4821d9898d42799c5789d4d2da3f63 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:39850
I20240503 01:59:40.943894 6208 catalog_manager.cc:4830] TS 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131:44005): tablet 1d392e08c15a4d4098a350287b3163ef (table rebalance_test_table_5 [id=ba15a57b1f0e43fd9bae24116dc815d0]) successfully deleted
I20240503 01:59:40.944028 10112 tablet_replica.cc:330] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 0c4821d9898d42799c5789d4d2da3f63: stopping tablet replica
I20240503 01:59:40.944155 10112 raft_consensus.cc:2229] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:40.944316 10112 raft_consensus.cc:2258] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:40.945020 10112 ts_tablet_manager.cc:1875] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 0c4821d9898d42799c5789d4d2da3f63: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:40.944957 6625 raft_consensus.cc:2932] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } }
I20240503 01:59:40.944921 7917 raft_consensus.cc:2932] T ccb34a4b52244d90850ab04e21299f55 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 7a1404fe1c1b4b388b821c8261b53104 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: true } } }
I20240503 01:59:40.946638 6625 raft_consensus.cc:2932] T ccb34a4b52244d90850ab04e21299f55 P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 7a1404fe1c1b4b388b821c8261b53104 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: true } } }
I20240503 01:59:40.947065 10112 ts_tablet_manager.cc:1888] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 0c4821d9898d42799c5789d4d2da3f63: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 01:59:40.947135 10112 log.cc:1196] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 0c4821d9898d42799c5789d4d2da3f63: Deleting WAL directory at /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/2088c84e4eef4ccfbc70d7b668b8f2cf
I20240503 01:59:40.947628 6210 catalog_manager.cc:4830] TS 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130:34189): tablet 2088c84e4eef4ccfbc70d7b668b8f2cf (table rebalance_test_table_5 [id=ba15a57b1f0e43fd9bae24116dc815d0]) successfully deleted
I20240503 01:59:40.948053 10101 raft_consensus.cc:1050] T 0bdcea4e59ec4e2da63b9316f9d718be P 79163c69f6f04a9abd0208d74e1c61be: attempting to promote NON_VOTER de2f076398ee44008622fb5db2a75952 to VOTER
I20240503 01:59:40.948063 6222 catalog_manager.cc:5484] T ccb34a4b52244d90850ab04e21299f55 P 0c4821d9898d42799c5789d4d2da3f63 reported cstate change: config changed from index -1 to 4, NON_VOTER 7a1404fe1c1b4b388b821c8261b53104 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "0c4821d9898d42799c5789d4d2da3f63" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:40.948460 10101 consensus_queue.cc:237] T 0bdcea4e59ec4e2da63b9316f9d718be P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } }
I20240503 01:59:40.949837 6758 raft_consensus.cc:1261] T 0bdcea4e59ec4e2da63b9316f9d718be P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:40.950230 10108 consensus_queue.cc:1035] T 0bdcea4e59ec4e2da63b9316f9d718be P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:40.950428 8355 raft_consensus.cc:1261] T 0bdcea4e59ec4e2da63b9316f9d718be P de2f076398ee44008622fb5db2a75952 [term 1 LEARNER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:40.951535 10108 consensus_queue.cc:1035] T 0bdcea4e59ec4e2da63b9316f9d718be P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:40.952656 6363 raft_consensus.cc:1261] T 0bdcea4e59ec4e2da63b9316f9d718be P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:40.952734 6365 raft_consensus.cc:2932] T ccb34a4b52244d90850ab04e21299f55 P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 7a1404fe1c1b4b388b821c8261b53104 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: true } } }
I20240503 01:59:40.953943 10108 consensus_queue.cc:1035] T 0bdcea4e59ec4e2da63b9316f9d718be P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:40.956907 7647 raft_consensus.cc:2932] T 0bdcea4e59ec4e2da63b9316f9d718be P 79163c69f6f04a9abd0208d74e1c61be [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, de2f076398ee44008622fb5db2a75952 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } }
I20240503 01:59:40.957509 6758 raft_consensus.cc:2932] T 0bdcea4e59ec4e2da63b9316f9d718be P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, de2f076398ee44008622fb5db2a75952 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } }
I20240503 01:59:40.959777 6219 catalog_manager.cc:5484] T 0bdcea4e59ec4e2da63b9316f9d718be P b6b1d5677b8d47788b7968f64b3dac3c reported cstate change: config changed from index 4 to 5, de2f076398ee44008622fb5db2a75952 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "79163c69f6f04a9abd0208d74e1c61be" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } }
I20240503 01:59:40.960243 8354 raft_consensus.cc:2932] T 0bdcea4e59ec4e2da63b9316f9d718be P de2f076398ee44008622fb5db2a75952 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, de2f076398ee44008622fb5db2a75952 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } }
I20240503 01:59:40.961987 9472 raft_consensus.cc:1050] T a5926dae08184952bc693342c2c61ebf P 79163c69f6f04a9abd0208d74e1c61be: attempting to promote NON_VOTER de2f076398ee44008622fb5db2a75952 to VOTER
I20240503 01:59:40.962395 9472 consensus_queue.cc:237] T a5926dae08184952bc693342c2c61ebf P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } }
I20240503 01:59:40.964079 6758 raft_consensus.cc:1261] T a5926dae08184952bc693342c2c61ebf P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:40.964102 8354 raft_consensus.cc:1261] T a5926dae08184952bc693342c2c61ebf P de2f076398ee44008622fb5db2a75952 [term 1 LEARNER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:40.964461 10108 consensus_queue.cc:1035] T a5926dae08184952bc693342c2c61ebf P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:40.964699 6365 raft_consensus.cc:2932] T 0bdcea4e59ec4e2da63b9316f9d718be P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, de2f076398ee44008622fb5db2a75952 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } }
I20240503 01:59:40.964855 10108 consensus_queue.cc:1035] T a5926dae08184952bc693342c2c61ebf P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:40.964926 6363 raft_consensus.cc:1261] T a5926dae08184952bc693342c2c61ebf P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:40.967481 6625 consensus_queue.cc:237] T 0bdcea4e59ec4e2da63b9316f9d718be P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } }
I20240503 01:59:40.967978 10101 consensus_queue.cc:1035] T a5926dae08184952bc693342c2c61ebf P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:40.968958 8355 raft_consensus.cc:1261] T 0bdcea4e59ec4e2da63b9316f9d718be P de2f076398ee44008622fb5db2a75952 [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:40.968984 6758 raft_consensus.cc:1261] T 0bdcea4e59ec4e2da63b9316f9d718be P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:40.969326 10101 consensus_queue.cc:1035] T 0bdcea4e59ec4e2da63b9316f9d718be P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:40.969805 10101 consensus_queue.cc:1035] T 0bdcea4e59ec4e2da63b9316f9d718be P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:40.970698 10101 raft_consensus.cc:2932] T 0bdcea4e59ec4e2da63b9316f9d718be P 79163c69f6f04a9abd0208d74e1c61be [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4127249ba42644388d225d614c935a25 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } }
I20240503 01:59:40.971148 6758 raft_consensus.cc:2932] T 0bdcea4e59ec4e2da63b9316f9d718be P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4127249ba42644388d225d614c935a25 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } }
I20240503 01:59:40.971191 9472 raft_consensus.cc:2932] T a5926dae08184952bc693342c2c61ebf P 79163c69f6f04a9abd0208d74e1c61be [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, de2f076398ee44008622fb5db2a75952 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } }
I20240503 01:59:40.972321 6208 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0bdcea4e59ec4e2da63b9316f9d718be with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:40.973181 8355 raft_consensus.cc:2932] T 0bdcea4e59ec4e2da63b9316f9d718be P de2f076398ee44008622fb5db2a75952 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4127249ba42644388d225d614c935a25 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } }
I20240503 01:59:40.973713 6757 raft_consensus.cc:2932] T a5926dae08184952bc693342c2c61ebf P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, de2f076398ee44008622fb5db2a75952 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } }
I20240503 01:59:40.974738 6365 raft_consensus.cc:2932] T a5926dae08184952bc693342c2c61ebf P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, de2f076398ee44008622fb5db2a75952 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } }
I20240503 01:59:40.974972 6219 catalog_manager.cc:5484] T 0bdcea4e59ec4e2da63b9316f9d718be P b6b1d5677b8d47788b7968f64b3dac3c reported cstate change: config changed from index 5 to 6, VOTER 4127249ba42644388d225d614c935a25 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "79163c69f6f04a9abd0208d74e1c61be" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } }
I20240503 01:59:40.977141 6221 catalog_manager.cc:5484] T a5926dae08184952bc693342c2c61ebf P 79163c69f6f04a9abd0208d74e1c61be reported cstate change: config changed from index 4 to 5, de2f076398ee44008622fb5db2a75952 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "79163c69f6f04a9abd0208d74e1c61be" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:40.977955 6344 tablet_service.cc:1514] Processing DeleteTablet for tablet 0bdcea4e59ec4e2da63b9316f9d718be with delete_type TABLET_DATA_TOMBSTONED (TS 4127249ba42644388d225d614c935a25 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:57398
I20240503 01:59:40.978121 9950 tablet_replica.cc:330] T 0bdcea4e59ec4e2da63b9316f9d718be P 4127249ba42644388d225d614c935a25: stopping tablet replica
I20240503 01:59:40.978233 9950 raft_consensus.cc:2229] T 0bdcea4e59ec4e2da63b9316f9d718be P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:40.978390 9950 raft_consensus.cc:2258] T 0bdcea4e59ec4e2da63b9316f9d718be P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:40.979182 9950 ts_tablet_manager.cc:1875] T 0bdcea4e59ec4e2da63b9316f9d718be P 4127249ba42644388d225d614c935a25: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:40.981169 9950 ts_tablet_manager.cc:1888] T 0bdcea4e59ec4e2da63b9316f9d718be P 4127249ba42644388d225d614c935a25: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 01:59:40.981241 9950 log.cc:1196] T 0bdcea4e59ec4e2da63b9316f9d718be P 4127249ba42644388d225d614c935a25: Deleting WAL directory at /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/0bdcea4e59ec4e2da63b9316f9d718be
I20240503 01:59:40.981694 6208 catalog_manager.cc:4830] TS 4127249ba42644388d225d614c935a25 (127.0.99.129:35613): tablet 0bdcea4e59ec4e2da63b9316f9d718be (table rebalance_test_table_8 [id=85186d17f9464a408a3790b0978eb502]) successfully deleted
I20240503 01:59:40.983718 8354 raft_consensus.cc:2932] T a5926dae08184952bc693342c2c61ebf P de2f076398ee44008622fb5db2a75952 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, de2f076398ee44008622fb5db2a75952 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } }
I20240503 01:59:40.986377 6625 consensus_queue.cc:237] T a5926dae08184952bc693342c2c61ebf P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } }
I20240503 01:59:40.987562 6757 raft_consensus.cc:1261] T a5926dae08184952bc693342c2c61ebf P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:40.987920 8354 raft_consensus.cc:1261] T a5926dae08184952bc693342c2c61ebf P de2f076398ee44008622fb5db2a75952 [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:40.988271 9472 consensus_queue.cc:1035] T a5926dae08184952bc693342c2c61ebf P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:40.988778 10305 consensus_queue.cc:1035] T a5926dae08184952bc693342c2c61ebf P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:40.990365 9472 raft_consensus.cc:2932] T a5926dae08184952bc693342c2c61ebf P 79163c69f6f04a9abd0208d74e1c61be [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4127249ba42644388d225d614c935a25 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } }
I20240503 01:59:40.991067 8354 raft_consensus.cc:2932] T a5926dae08184952bc693342c2c61ebf P de2f076398ee44008622fb5db2a75952 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4127249ba42644388d225d614c935a25 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } }
I20240503 01:59:40.991602 6208 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a5926dae08184952bc693342c2c61ebf with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:40.992008 6758 raft_consensus.cc:2932] T a5926dae08184952bc693342c2c61ebf P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4127249ba42644388d225d614c935a25 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } }
I20240503 01:59:40.993193 6222 catalog_manager.cc:5484] T a5926dae08184952bc693342c2c61ebf P 79163c69f6f04a9abd0208d74e1c61be reported cstate change: config changed from index 5 to 6, VOTER 4127249ba42644388d225d614c935a25 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "79163c69f6f04a9abd0208d74e1c61be" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:40.995641 6344 tablet_service.cc:1514] Processing DeleteTablet for tablet a5926dae08184952bc693342c2c61ebf with delete_type TABLET_DATA_TOMBSTONED (TS 4127249ba42644388d225d614c935a25 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:57398
I20240503 01:59:40.995880 9950 tablet_replica.cc:330] T a5926dae08184952bc693342c2c61ebf P 4127249ba42644388d225d614c935a25: stopping tablet replica
I20240503 01:59:40.996012 9950 raft_consensus.cc:2229] T a5926dae08184952bc693342c2c61ebf P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:40.996152 9950 raft_consensus.cc:2258] T a5926dae08184952bc693342c2c61ebf P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:40.996867 9950 ts_tablet_manager.cc:1875] T a5926dae08184952bc693342c2c61ebf P 4127249ba42644388d225d614c935a25: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:40.998842 9950 ts_tablet_manager.cc:1888] T a5926dae08184952bc693342c2c61ebf P 4127249ba42644388d225d614c935a25: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 01:59:40.998921 9950 log.cc:1196] T a5926dae08184952bc693342c2c61ebf P 4127249ba42644388d225d614c935a25: Deleting WAL directory at /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/a5926dae08184952bc693342c2c61ebf
I20240503 01:59:40.999416 6208 catalog_manager.cc:4830] TS 4127249ba42644388d225d614c935a25 (127.0.99.129:35613): tablet a5926dae08184952bc693342c2c61ebf (table rebalance_test_table_3 [id=2c07d364168a409a98aed9a7c3c1015c]) successfully deleted
I20240503 01:59:41.024207 9923 raft_consensus.cc:1050] T 120d5dd4f26c42399702141094ce7982 P 0c4821d9898d42799c5789d4d2da3f63: attempting to promote NON_VOTER 7a1404fe1c1b4b388b821c8261b53104 to VOTER
I20240503 01:59:41.024631 9923 consensus_queue.cc:237] T 120d5dd4f26c42399702141094ce7982 P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } }
I20240503 01:59:41.026124 6365 raft_consensus.cc:1261] T 120d5dd4f26c42399702141094ce7982 P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Refusing update from remote peer 0c4821d9898d42799c5789d4d2da3f63: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:41.026124 6758 raft_consensus.cc:1261] T 120d5dd4f26c42399702141094ce7982 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Refusing update from remote peer 0c4821d9898d42799c5789d4d2da3f63: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:41.026525 9924 consensus_queue.cc:1035] T 120d5dd4f26c42399702141094ce7982 P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:41.026597 8485 raft_consensus.cc:1261] T 120d5dd4f26c42399702141094ce7982 P 7a1404fe1c1b4b388b821c8261b53104 [term 1 LEARNER]: Refusing update from remote peer 0c4821d9898d42799c5789d4d2da3f63: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:41.026795 7917 consensus_queue.cc:1035] T 120d5dd4f26c42399702141094ce7982 P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:41.028070 7917 consensus_queue.cc:1035] T 120d5dd4f26c42399702141094ce7982 P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:41.028882 9931 raft_consensus.cc:2932] T 120d5dd4f26c42399702141094ce7982 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7a1404fe1c1b4b388b821c8261b53104 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } }
I20240503 01:59:41.030561 6758 raft_consensus.cc:2932] T 120d5dd4f26c42399702141094ce7982 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7a1404fe1c1b4b388b821c8261b53104 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } }
I20240503 01:59:41.031121 8485 raft_consensus.cc:2932] T 120d5dd4f26c42399702141094ce7982 P 7a1404fe1c1b4b388b821c8261b53104 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7a1404fe1c1b4b388b821c8261b53104 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } }
I20240503 01:59:41.031121 6365 raft_consensus.cc:2932] T 120d5dd4f26c42399702141094ce7982 P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7a1404fe1c1b4b388b821c8261b53104 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } }
I20240503 01:59:41.031407 6221 catalog_manager.cc:5484] T 120d5dd4f26c42399702141094ce7982 P 0c4821d9898d42799c5789d4d2da3f63 reported cstate change: config changed from index 4 to 5, 7a1404fe1c1b4b388b821c8261b53104 (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0c4821d9898d42799c5789d4d2da3f63" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:41.034803 6495 consensus_queue.cc:237] T 120d5dd4f26c42399702141094ce7982 P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } }
I20240503 01:59:41.036145 8485 raft_consensus.cc:1261] T 120d5dd4f26c42399702141094ce7982 P 7a1404fe1c1b4b388b821c8261b53104 [term 1 FOLLOWER]: Refusing update from remote peer 0c4821d9898d42799c5789d4d2da3f63: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:41.036208 6365 raft_consensus.cc:1261] T 120d5dd4f26c42399702141094ce7982 P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Refusing update from remote peer 0c4821d9898d42799c5789d4d2da3f63: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:41.036605 9924 consensus_queue.cc:1035] T 120d5dd4f26c42399702141094ce7982 P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:41.037007 9931 consensus_queue.cc:1035] T 120d5dd4f26c42399702141094ce7982 P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:41.038404 9931 raft_consensus.cc:2932] T 120d5dd4f26c42399702141094ce7982 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b6b1d5677b8d47788b7968f64b3dac3c (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } }
I20240503 01:59:41.038739 8701 raft_consensus.cc:1050] T 4e7ceb1453804246a41bd1dda0e39d02 P 4127249ba42644388d225d614c935a25: attempting to promote NON_VOTER 65d4632bcdd54ba596acdbbe1b5ef8af to VOTER
I20240503 01:59:41.039099 8701 consensus_queue.cc:237] T 4e7ceb1453804246a41bd1dda0e39d02 P 4127249ba42644388d225d614c935a25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } }
I20240503 01:59:41.039438 10129 raft_consensus.cc:1050] T e5a00d4ce3a64ca5a7c5bd65c556f429 P b6b1d5677b8d47788b7968f64b3dac3c: attempting to promote NON_VOTER de2f076398ee44008622fb5db2a75952 to VOTER
I20240503 01:59:41.039808 10129 consensus_queue.cc:237] T e5a00d4ce3a64ca5a7c5bd65c556f429 P b6b1d5677b8d47788b7968f64b3dac3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } }
I20240503 01:59:41.039904 6365 raft_consensus.cc:2932] T 120d5dd4f26c42399702141094ce7982 P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b6b1d5677b8d47788b7968f64b3dac3c (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } }
I20240503 01:59:41.040936 8224 raft_consensus.cc:1261] T 4e7ceb1453804246a41bd1dda0e39d02 P 65d4632bcdd54ba596acdbbe1b5ef8af [term 1 LEARNER]: Refusing update from remote peer 4127249ba42644388d225d614c935a25: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:41.041378 6625 raft_consensus.cc:1261] T 4e7ceb1453804246a41bd1dda0e39d02 P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Refusing update from remote peer 4127249ba42644388d225d614c935a25: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:41.041399 6494 raft_consensus.cc:1261] T e5a00d4ce3a64ca5a7c5bd65c556f429 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Refusing update from remote peer b6b1d5677b8d47788b7968f64b3dac3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:41.041401 8354 raft_consensus.cc:1261] T e5a00d4ce3a64ca5a7c5bd65c556f429 P de2f076398ee44008622fb5db2a75952 [term 1 LEARNER]: Refusing update from remote peer b6b1d5677b8d47788b7968f64b3dac3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:41.041829 10132 consensus_queue.cc:1035] T e5a00d4ce3a64ca5a7c5bd65c556f429 P b6b1d5677b8d47788b7968f64b3dac3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:41.041970 10117 consensus_queue.cc:1035] T 4e7ceb1453804246a41bd1dda0e39d02 P 4127249ba42644388d225d614c935a25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:41.042039 8485 raft_consensus.cc:2932] T 120d5dd4f26c42399702141094ce7982 P 7a1404fe1c1b4b388b821c8261b53104 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b6b1d5677b8d47788b7968f64b3dac3c (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } }
I20240503 01:59:41.042150 6625 raft_consensus.cc:1261] T e5a00d4ce3a64ca5a7c5bd65c556f429 P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Refusing update from remote peer b6b1d5677b8d47788b7968f64b3dac3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:41.043181 10132 consensus_queue.cc:1035] T e5a00d4ce3a64ca5a7c5bd65c556f429 P b6b1d5677b8d47788b7968f64b3dac3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:41.044178 6495 raft_consensus.cc:1261] T 4e7ceb1453804246a41bd1dda0e39d02 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Refusing update from remote peer 4127249ba42644388d225d614c935a25: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:41.044391 10132 consensus_queue.cc:1035] T e5a00d4ce3a64ca5a7c5bd65c556f429 P b6b1d5677b8d47788b7968f64b3dac3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:41.044919 10117 consensus_queue.cc:1035] T 4e7ceb1453804246a41bd1dda0e39d02 P 4127249ba42644388d225d614c935a25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:41.045462 10117 consensus_queue.cc:1035] T 4e7ceb1453804246a41bd1dda0e39d02 P 4127249ba42644388d225d614c935a25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:41.046273 6219 catalog_manager.cc:5484] T 120d5dd4f26c42399702141094ce7982 P 7a1404fe1c1b4b388b821c8261b53104 reported cstate change: config changed from index 5 to 6, VOTER b6b1d5677b8d47788b7968f64b3dac3c (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "0c4821d9898d42799c5789d4d2da3f63" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } }
I20240503 01:59:41.046979 10108 raft_consensus.cc:1050] T a9a74e04ce3745b2b479e96a3f3dc86e P 79163c69f6f04a9abd0208d74e1c61be: attempting to promote NON_VOTER 7a1404fe1c1b4b388b821c8261b53104 to VOTER
I20240503 01:59:41.047387 10108 consensus_queue.cc:237] T a9a74e04ce3745b2b479e96a3f3dc86e P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } }
I20240503 01:59:41.048043 10132 raft_consensus.cc:2932] T e5a00d4ce3a64ca5a7c5bd65c556f429 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, de2f076398ee44008622fb5db2a75952 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } }
I20240503 01:59:41.048440 6494 raft_consensus.cc:2932] T e5a00d4ce3a64ca5a7c5bd65c556f429 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, de2f076398ee44008622fb5db2a75952 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } }
I20240503 01:59:41.050631 8485 raft_consensus.cc:1261] T a9a74e04ce3745b2b479e96a3f3dc86e P 7a1404fe1c1b4b388b821c8261b53104 [term 1 LEARNER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:41.050741 6625 raft_consensus.cc:2932] T e5a00d4ce3a64ca5a7c5bd65c556f429 P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, de2f076398ee44008622fb5db2a75952 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } }
I20240503 01:59:41.051065 6365 raft_consensus.cc:1261] T a9a74e04ce3745b2b479e96a3f3dc86e P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:41.051584 6210 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 120d5dd4f26c42399702141094ce7982 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:41.051990 6758 raft_consensus.cc:1261] T a9a74e04ce3745b2b479e96a3f3dc86e P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:41.051990 6222 catalog_manager.cc:5484] T e5a00d4ce3a64ca5a7c5bd65c556f429 P b6b1d5677b8d47788b7968f64b3dac3c reported cstate change: config changed from index 4 to 5, de2f076398ee44008622fb5db2a75952 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:41.052568 10315 consensus_queue.cc:1035] T a9a74e04ce3745b2b479e96a3f3dc86e P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:41.053355 10316 consensus_queue.cc:1035] T a9a74e04ce3745b2b479e96a3f3dc86e P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:41.053735 7647 consensus_queue.cc:1035] T a9a74e04ce3745b2b479e96a3f3dc86e P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:41.054181 8701 raft_consensus.cc:2932] T 4e7ceb1453804246a41bd1dda0e39d02 P 4127249ba42644388d225d614c935a25 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 65d4632bcdd54ba596acdbbe1b5ef8af (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } }
I20240503 01:59:41.056037 6494 raft_consensus.cc:2932] T 4e7ceb1453804246a41bd1dda0e39d02 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 65d4632bcdd54ba596acdbbe1b5ef8af (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } }
I20240503 01:59:41.056670 8224 raft_consensus.cc:2932] T 4e7ceb1453804246a41bd1dda0e39d02 P 65d4632bcdd54ba596acdbbe1b5ef8af [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 65d4632bcdd54ba596acdbbe1b5ef8af (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } }
I20240503 01:59:41.058055 6738 tablet_service.cc:1514] Processing DeleteTablet for tablet 120d5dd4f26c42399702141094ce7982 with delete_type TABLET_DATA_TOMBSTONED (TS b6b1d5677b8d47788b7968f64b3dac3c not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:51980
I20240503 01:59:41.058225 10222 tablet_replica.cc:330] T 120d5dd4f26c42399702141094ce7982 P b6b1d5677b8d47788b7968f64b3dac3c: stopping tablet replica
I20240503 01:59:41.058369 10222 raft_consensus.cc:2229] T 120d5dd4f26c42399702141094ce7982 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:41.058523 10222 raft_consensus.cc:2258] T 120d5dd4f26c42399702141094ce7982 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:41.059211 6218 catalog_manager.cc:5484] T 4e7ceb1453804246a41bd1dda0e39d02 P 65d4632bcdd54ba596acdbbe1b5ef8af reported cstate change: config changed from index 4 to 5, 65d4632bcdd54ba596acdbbe1b5ef8af (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4127249ba42644388d225d614c935a25" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } }
I20240503 01:59:41.059762 10222 ts_tablet_manager.cc:1875] T 120d5dd4f26c42399702141094ce7982 P b6b1d5677b8d47788b7968f64b3dac3c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:41.061847 10222 ts_tablet_manager.cc:1888] T 120d5dd4f26c42399702141094ce7982 P b6b1d5677b8d47788b7968f64b3dac3c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 01:59:41.061915 10222 log.cc:1196] T 120d5dd4f26c42399702141094ce7982 P b6b1d5677b8d47788b7968f64b3dac3c: Deleting WAL directory at /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/120d5dd4f26c42399702141094ce7982
I20240503 01:59:41.062346 6209 catalog_manager.cc:4830] TS b6b1d5677b8d47788b7968f64b3dac3c (127.0.99.132:46067): tablet 120d5dd4f26c42399702141094ce7982 (table rebalance_test_table_6 [id=80cb1486f22e4bee930434fdfc3463d2]) successfully deleted
I20240503 01:59:41.063086 8354 raft_consensus.cc:2932] T e5a00d4ce3a64ca5a7c5bd65c556f429 P de2f076398ee44008622fb5db2a75952 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, de2f076398ee44008622fb5db2a75952 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } }
I20240503 01:59:41.064306 6625 raft_consensus.cc:2932] T 4e7ceb1453804246a41bd1dda0e39d02 P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 65d4632bcdd54ba596acdbbe1b5ef8af (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } }
I20240503 01:59:41.064381 7647 raft_consensus.cc:2932] T a9a74e04ce3745b2b479e96a3f3dc86e P 79163c69f6f04a9abd0208d74e1c61be [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7a1404fe1c1b4b388b821c8261b53104 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } }
I20240503 01:59:41.067116 6758 raft_consensus.cc:2932] T a9a74e04ce3745b2b479e96a3f3dc86e P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7a1404fe1c1b4b388b821c8261b53104 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } }
I20240503 01:59:41.067211 6365 raft_consensus.cc:2932] T a9a74e04ce3745b2b479e96a3f3dc86e P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7a1404fe1c1b4b388b821c8261b53104 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } }
I20240503 01:59:41.067334 8485 raft_consensus.cc:2932] T a9a74e04ce3745b2b479e96a3f3dc86e P 7a1404fe1c1b4b388b821c8261b53104 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7a1404fe1c1b4b388b821c8261b53104 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } }
I20240503 01:59:41.067894 6757 consensus_queue.cc:237] T e5a00d4ce3a64ca5a7c5bd65c556f429 P b6b1d5677b8d47788b7968f64b3dac3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } }
I20240503 01:59:41.069888 6494 raft_consensus.cc:1261] T e5a00d4ce3a64ca5a7c5bd65c556f429 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Refusing update from remote peer b6b1d5677b8d47788b7968f64b3dac3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:41.069886 6222 catalog_manager.cc:5484] T a9a74e04ce3745b2b479e96a3f3dc86e P 7a1404fe1c1b4b388b821c8261b53104 reported cstate change: config changed from index 4 to 5, 7a1404fe1c1b4b388b821c8261b53104 (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "79163c69f6f04a9abd0208d74e1c61be" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } }
I20240503 01:59:41.070304 9464 consensus_queue.cc:1035] T e5a00d4ce3a64ca5a7c5bd65c556f429 P b6b1d5677b8d47788b7968f64b3dac3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:41.070755 8354 raft_consensus.cc:1261] T e5a00d4ce3a64ca5a7c5bd65c556f429 P de2f076398ee44008622fb5db2a75952 [term 1 FOLLOWER]: Refusing update from remote peer b6b1d5677b8d47788b7968f64b3dac3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:41.071171 9464 consensus_queue.cc:1035] T e5a00d4ce3a64ca5a7c5bd65c556f429 P b6b1d5677b8d47788b7968f64b3dac3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:41.075814 9464 raft_consensus.cc:2932] T e5a00d4ce3a64ca5a7c5bd65c556f429 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } }
I20240503 01:59:41.076025 6625 consensus_queue.cc:237] T a9a74e04ce3745b2b479e96a3f3dc86e P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } }
I20240503 01:59:41.077170 6365 raft_consensus.cc:1261] T a9a74e04ce3745b2b479e96a3f3dc86e P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:41.077466 8485 raft_consensus.cc:1261] T a9a74e04ce3745b2b479e96a3f3dc86e P 7a1404fe1c1b4b388b821c8261b53104 [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:41.078069 6209 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e5a00d4ce3a64ca5a7c5bd65c556f429 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:41.078089 10316 consensus_queue.cc:1035] T a9a74e04ce3745b2b479e96a3f3dc86e P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:41.078253 6495 raft_consensus.cc:2932] T e5a00d4ce3a64ca5a7c5bd65c556f429 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } }
I20240503 01:59:41.078424 10316 consensus_queue.cc:1035] T a9a74e04ce3745b2b479e96a3f3dc86e P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:41.078649 6218 catalog_manager.cc:5484] T e5a00d4ce3a64ca5a7c5bd65c556f429 P b6b1d5677b8d47788b7968f64b3dac3c reported cstate change: config changed from index 5 to 6, VOTER 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:41.081153 8354 raft_consensus.cc:2932] T e5a00d4ce3a64ca5a7c5bd65c556f429 P de2f076398ee44008622fb5db2a75952 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } }
I20240503 01:59:41.081609 6607 tablet_service.cc:1514] Processing DeleteTablet for tablet e5a00d4ce3a64ca5a7c5bd65c556f429 with delete_type TABLET_DATA_TOMBSTONED (TS 79163c69f6f04a9abd0208d74e1c61be not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:47560
I20240503 01:59:41.081810 10128 tablet_replica.cc:330] T e5a00d4ce3a64ca5a7c5bd65c556f429 P 79163c69f6f04a9abd0208d74e1c61be: stopping tablet replica
I20240503 01:59:41.081943 10128 raft_consensus.cc:2229] T e5a00d4ce3a64ca5a7c5bd65c556f429 P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:41.082087 10128 raft_consensus.cc:2258] T e5a00d4ce3a64ca5a7c5bd65c556f429 P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:41.082897 10128 ts_tablet_manager.cc:1875] T e5a00d4ce3a64ca5a7c5bd65c556f429 P 79163c69f6f04a9abd0208d74e1c61be: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:41.082995 10305 raft_consensus.cc:1050] T a655d914858447b59fe5763e43c0807d P 79163c69f6f04a9abd0208d74e1c61be: attempting to promote NON_VOTER 7a1404fe1c1b4b388b821c8261b53104 to VOTER
I20240503 01:59:41.082966 9472 raft_consensus.cc:2932] T a9a74e04ce3745b2b479e96a3f3dc86e P 79163c69f6f04a9abd0208d74e1c61be [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b6b1d5677b8d47788b7968f64b3dac3c (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } }
I20240503 01:59:41.083375 10305 consensus_queue.cc:237] T a655d914858447b59fe5763e43c0807d P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } }
I20240503 01:59:41.084512 6208 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a9a74e04ce3745b2b479e96a3f3dc86e with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:41.084950 10128 ts_tablet_manager.cc:1888] T e5a00d4ce3a64ca5a7c5bd65c556f429 P 79163c69f6f04a9abd0208d74e1c61be: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 01:59:41.085019 10128 log.cc:1196] T e5a00d4ce3a64ca5a7c5bd65c556f429 P 79163c69f6f04a9abd0208d74e1c61be: Deleting WAL directory at /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/e5a00d4ce3a64ca5a7c5bd65c556f429
I20240503 01:59:41.085064 6363 raft_consensus.cc:1261] T a655d914858447b59fe5763e43c0807d P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:41.085296 6757 raft_consensus.cc:1261] T a655d914858447b59fe5763e43c0807d P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:41.085687 10316 consensus_queue.cc:1035] T a655d914858447b59fe5763e43c0807d P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:41.085815 6208 catalog_manager.cc:4830] TS 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131:44005): tablet e5a00d4ce3a64ca5a7c5bd65c556f429 (table rebalance_test_table_1 [id=91bdb83bc45b4b61b094a0180eacf8e3]) successfully deleted
I20240503 01:59:41.086613 6365 raft_consensus.cc:2932] T a9a74e04ce3745b2b479e96a3f3dc86e P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b6b1d5677b8d47788b7968f64b3dac3c (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } }
I20240503 01:59:41.088716 10316 consensus_queue.cc:1035] T a655d914858447b59fe5763e43c0807d P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:41.089082 8485 raft_consensus.cc:2932] T a9a74e04ce3745b2b479e96a3f3dc86e P 7a1404fe1c1b4b388b821c8261b53104 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b6b1d5677b8d47788b7968f64b3dac3c (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } }
I20240503 01:59:41.089251 8486 raft_consensus.cc:1261] T a655d914858447b59fe5763e43c0807d P 7a1404fe1c1b4b388b821c8261b53104 [term 1 LEARNER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:41.089798 6219 catalog_manager.cc:5484] T a9a74e04ce3745b2b479e96a3f3dc86e P 4127249ba42644388d225d614c935a25 reported cstate change: config changed from index 5 to 6, VOTER b6b1d5677b8d47788b7968f64b3dac3c (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "79163c69f6f04a9abd0208d74e1c61be" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } }
I20240503 01:59:41.090602 10316 consensus_queue.cc:1035] T a655d914858447b59fe5763e43c0807d P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:41.092409 6738 tablet_service.cc:1514] Processing DeleteTablet for tablet a9a74e04ce3745b2b479e96a3f3dc86e with delete_type TABLET_DATA_TOMBSTONED (TS b6b1d5677b8d47788b7968f64b3dac3c not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:51980
I20240503 01:59:41.092602 10222 tablet_replica.cc:330] T a9a74e04ce3745b2b479e96a3f3dc86e P b6b1d5677b8d47788b7968f64b3dac3c: stopping tablet replica
I20240503 01:59:41.092763 10222 raft_consensus.cc:2229] T a9a74e04ce3745b2b479e96a3f3dc86e P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:41.092912 10222 raft_consensus.cc:2258] T a9a74e04ce3745b2b479e96a3f3dc86e P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:41.093189 9472 raft_consensus.cc:2932] T a655d914858447b59fe5763e43c0807d P 79163c69f6f04a9abd0208d74e1c61be [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7a1404fe1c1b4b388b821c8261b53104 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } }
I20240503 01:59:41.093717 10222 ts_tablet_manager.cc:1875] T a9a74e04ce3745b2b479e96a3f3dc86e P b6b1d5677b8d47788b7968f64b3dac3c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:41.094856 6757 raft_consensus.cc:2932] T a655d914858447b59fe5763e43c0807d P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7a1404fe1c1b4b388b821c8261b53104 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } }
I20240503 01:59:41.095635 10222 ts_tablet_manager.cc:1888] T a9a74e04ce3745b2b479e96a3f3dc86e P b6b1d5677b8d47788b7968f64b3dac3c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 01:59:41.095710 10222 log.cc:1196] T a9a74e04ce3745b2b479e96a3f3dc86e P b6b1d5677b8d47788b7968f64b3dac3c: Deleting WAL directory at /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/a9a74e04ce3745b2b479e96a3f3dc86e
I20240503 01:59:41.095669 6365 raft_consensus.cc:2932] T a655d914858447b59fe5763e43c0807d P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7a1404fe1c1b4b388b821c8261b53104 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } }
I20240503 01:59:41.096227 6209 catalog_manager.cc:4830] TS b6b1d5677b8d47788b7968f64b3dac3c (127.0.99.132:46067): tablet a9a74e04ce3745b2b479e96a3f3dc86e (table rebalance_test_table_2 [id=4a248d80763944458620bf40775e2098]) successfully deleted
I20240503 01:59:41.097227 6214 catalog_manager.cc:5484] T a655d914858447b59fe5763e43c0807d P 79163c69f6f04a9abd0208d74e1c61be reported cstate change: config changed from index 4 to 5, 7a1404fe1c1b4b388b821c8261b53104 (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "79163c69f6f04a9abd0208d74e1c61be" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:41.098520 8486 raft_consensus.cc:2932] T a655d914858447b59fe5763e43c0807d P 7a1404fe1c1b4b388b821c8261b53104 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7a1404fe1c1b4b388b821c8261b53104 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } }
I20240503 01:59:41.101526 6625 consensus_queue.cc:237] T a655d914858447b59fe5763e43c0807d P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } }
I20240503 01:59:41.102509 8485 raft_consensus.cc:1261] T a655d914858447b59fe5763e43c0807d P 7a1404fe1c1b4b388b821c8261b53104 [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:41.102715 6365 raft_consensus.cc:1261] T a655d914858447b59fe5763e43c0807d P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:41.102985 10315 consensus_queue.cc:1035] T a655d914858447b59fe5763e43c0807d P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:41.103224 10316 consensus_queue.cc:1035] T a655d914858447b59fe5763e43c0807d P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:41.104913 10316 raft_consensus.cc:2932] T a655d914858447b59fe5763e43c0807d P 79163c69f6f04a9abd0208d74e1c61be [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b6b1d5677b8d47788b7968f64b3dac3c (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } }
I20240503 01:59:41.106467 8485 raft_consensus.cc:2932] T a655d914858447b59fe5763e43c0807d P 7a1404fe1c1b4b388b821c8261b53104 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b6b1d5677b8d47788b7968f64b3dac3c (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } }
I20240503 01:59:41.106799 6365 raft_consensus.cc:2932] T a655d914858447b59fe5763e43c0807d P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b6b1d5677b8d47788b7968f64b3dac3c (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } }
I20240503 01:59:41.108717 6208 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a655d914858447b59fe5763e43c0807d with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:41.109200 6219 catalog_manager.cc:5484] T a655d914858447b59fe5763e43c0807d P 7a1404fe1c1b4b388b821c8261b53104 reported cstate change: config changed from index 5 to 6, VOTER b6b1d5677b8d47788b7968f64b3dac3c (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "79163c69f6f04a9abd0208d74e1c61be" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } }
I20240503 01:59:41.111451 6738 tablet_service.cc:1514] Processing DeleteTablet for tablet a655d914858447b59fe5763e43c0807d with delete_type TABLET_DATA_TOMBSTONED (TS b6b1d5677b8d47788b7968f64b3dac3c not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:51980
I20240503 01:59:41.111688 10222 tablet_replica.cc:330] T a655d914858447b59fe5763e43c0807d P b6b1d5677b8d47788b7968f64b3dac3c: stopping tablet replica
I20240503 01:59:41.111840 10222 raft_consensus.cc:2229] T a655d914858447b59fe5763e43c0807d P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:41.111984 10222 raft_consensus.cc:2258] T a655d914858447b59fe5763e43c0807d P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:41.112769 10222 ts_tablet_manager.cc:1875] T a655d914858447b59fe5763e43c0807d P b6b1d5677b8d47788b7968f64b3dac3c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:41.114627 10222 ts_tablet_manager.cc:1888] T a655d914858447b59fe5763e43c0807d P b6b1d5677b8d47788b7968f64b3dac3c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 01:59:41.114706 10222 log.cc:1196] T a655d914858447b59fe5763e43c0807d P b6b1d5677b8d47788b7968f64b3dac3c: Deleting WAL directory at /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/a655d914858447b59fe5763e43c0807d
I20240503 01:59:41.115178 6209 catalog_manager.cc:4830] TS b6b1d5677b8d47788b7968f64b3dac3c (127.0.99.132:46067): tablet a655d914858447b59fe5763e43c0807d (table rebalance_test_table_0 [id=2e2540d8e10e43848a2e486f767d1576]) successfully deleted
I20240503 01:59:41.148727 6365 consensus_queue.cc:237] T 82cfe99cd6fe4a82a7e35298ec634b8b P 4127249ba42644388d225d614c935a25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: true } }
I20240503 01:59:41.150223 6495 raft_consensus.cc:1261] T 82cfe99cd6fe4a82a7e35298ec634b8b P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Refusing update from remote peer 4127249ba42644388d225d614c935a25: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:41.150364 6757 raft_consensus.cc:1261] T 82cfe99cd6fe4a82a7e35298ec634b8b P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Refusing update from remote peer 4127249ba42644388d225d614c935a25: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 01:59:41.150482 6298 consensus_peers.cc:485] T 82cfe99cd6fe4a82a7e35298ec634b8b P 4127249ba42644388d225d614c935a25 -> Peer 9264f24d028347d98f7e4e2d1aedeb79 (127.0.99.133:41713): Couldn't send request to peer 9264f24d028347d98f7e4e2d1aedeb79. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 82cfe99cd6fe4a82a7e35298ec634b8b. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:41.150741 10117 consensus_queue.cc:1035] T 82cfe99cd6fe4a82a7e35298ec634b8b P 4127249ba42644388d225d614c935a25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:41.150910 10120 consensus_queue.cc:1035] T 82cfe99cd6fe4a82a7e35298ec634b8b P 4127249ba42644388d225d614c935a25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:41.152873 10120 raft_consensus.cc:2932] T 82cfe99cd6fe4a82a7e35298ec634b8b P 4127249ba42644388d225d614c935a25 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9264f24d028347d98f7e4e2d1aedeb79 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: true } } }
I20240503 01:59:41.154254 6495 raft_consensus.cc:2932] T 82cfe99cd6fe4a82a7e35298ec634b8b P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9264f24d028347d98f7e4e2d1aedeb79 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: true } } }
I20240503 01:59:41.154914 6757 raft_consensus.cc:2932] T 82cfe99cd6fe4a82a7e35298ec634b8b P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9264f24d028347d98f7e4e2d1aedeb79 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: true } } }
I20240503 01:59:41.156911 6214 catalog_manager.cc:5484] T 82cfe99cd6fe4a82a7e35298ec634b8b P 4127249ba42644388d225d614c935a25 reported cstate change: config changed from index -1 to 4, NON_VOTER 9264f24d028347d98f7e4e2d1aedeb79 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "4127249ba42644388d225d614c935a25" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:41.172354 6627 consensus_queue.cc:237] T 679fc86ea2ad4bd6a9160fddf4c586d7 P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: true } }
I20240503 01:59:41.173846 6757 raft_consensus.cc:1261] T 679fc86ea2ad4bd6a9160fddf4c586d7 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:41.173858 6365 raft_consensus.cc:1261] T 679fc86ea2ad4bd6a9160fddf4c586d7 P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:41.174305 10305 consensus_queue.cc:1035] T 679fc86ea2ad4bd6a9160fddf4c586d7 P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 01:59:41.174479 6563 consensus_peers.cc:485] T 679fc86ea2ad4bd6a9160fddf4c586d7 P 79163c69f6f04a9abd0208d74e1c61be -> Peer de2f076398ee44008622fb5db2a75952 (127.0.99.135:34035): Couldn't send request to peer de2f076398ee44008622fb5db2a75952. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 679fc86ea2ad4bd6a9160fddf4c586d7. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:41.174623 7647 consensus_queue.cc:1035] T 679fc86ea2ad4bd6a9160fddf4c586d7 P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:41.176254 7647 raft_consensus.cc:2932] T 679fc86ea2ad4bd6a9160fddf4c586d7 P 79163c69f6f04a9abd0208d74e1c61be [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER de2f076398ee44008622fb5db2a75952 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: true } } }
I20240503 01:59:41.177070 6365 raft_consensus.cc:2932] T 679fc86ea2ad4bd6a9160fddf4c586d7 P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER de2f076398ee44008622fb5db2a75952 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: true } } }
I20240503 01:59:41.179159 6219 catalog_manager.cc:5484] T 679fc86ea2ad4bd6a9160fddf4c586d7 P 79163c69f6f04a9abd0208d74e1c61be reported cstate change: config changed from index -1 to 4, NON_VOTER de2f076398ee44008622fb5db2a75952 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "79163c69f6f04a9abd0208d74e1c61be" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:41.179806 6757 raft_consensus.cc:2932] T 679fc86ea2ad4bd6a9160fddf4c586d7 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER de2f076398ee44008622fb5db2a75952 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: true } } }
I20240503 01:59:41.183878 10101 raft_consensus.cc:1050] T a388d59f6ba94e91958f55b340343fa6 P 79163c69f6f04a9abd0208d74e1c61be: attempting to promote NON_VOTER 9264f24d028347d98f7e4e2d1aedeb79 to VOTER
I20240503 01:59:41.184192 10101 consensus_queue.cc:237] T a388d59f6ba94e91958f55b340343fa6 P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } }
I20240503 01:59:41.185413 6365 raft_consensus.cc:1261] T a388d59f6ba94e91958f55b340343fa6 P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:41.185638 8080 raft_consensus.cc:1261] T a388d59f6ba94e91958f55b340343fa6 P 9264f24d028347d98f7e4e2d1aedeb79 [term 1 LEARNER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:41.185638 6495 raft_consensus.cc:1261] T a388d59f6ba94e91958f55b340343fa6 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:41.186036 7647 consensus_queue.cc:1035] T a388d59f6ba94e91958f55b340343fa6 P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:41.186530 7647 consensus_queue.cc:1035] T a388d59f6ba94e91958f55b340343fa6 P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:41.186672 9472 consensus_queue.cc:1035] T a388d59f6ba94e91958f55b340343fa6 P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:41.188591 10315 raft_consensus.cc:2932] T a388d59f6ba94e91958f55b340343fa6 P 79163c69f6f04a9abd0208d74e1c61be [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9264f24d028347d98f7e4e2d1aedeb79 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } }
I20240503 01:59:41.189136 8080 raft_consensus.cc:2932] T a388d59f6ba94e91958f55b340343fa6 P 9264f24d028347d98f7e4e2d1aedeb79 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9264f24d028347d98f7e4e2d1aedeb79 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } }
I20240503 01:59:41.189450 6365 raft_consensus.cc:2932] T a388d59f6ba94e91958f55b340343fa6 P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9264f24d028347d98f7e4e2d1aedeb79 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } }
I20240503 01:59:41.190753 6494 raft_consensus.cc:2932] T a388d59f6ba94e91958f55b340343fa6 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9264f24d028347d98f7e4e2d1aedeb79 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } }
I20240503 01:59:41.191731 6214 catalog_manager.cc:5484] T a388d59f6ba94e91958f55b340343fa6 P 9264f24d028347d98f7e4e2d1aedeb79 reported cstate change: config changed from index 4 to 5, 9264f24d028347d98f7e4e2d1aedeb79 (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "79163c69f6f04a9abd0208d74e1c61be" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } }
I20240503 01:59:41.193377 6494 consensus_queue.cc:237] T dc7d272c4d324e10959c62362e0d81f2 P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: true } }
W20240503 01:59:41.195245 6430 consensus_peers.cc:485] T dc7d272c4d324e10959c62362e0d81f2 P 0c4821d9898d42799c5789d4d2da3f63 -> Peer 7a1404fe1c1b4b388b821c8261b53104 (127.0.99.136:46759): Couldn't send request to peer 7a1404fe1c1b4b388b821c8261b53104. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dc7d272c4d324e10959c62362e0d81f2. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:41.195304 6757 raft_consensus.cc:1261] T dc7d272c4d324e10959c62362e0d81f2 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Refusing update from remote peer 0c4821d9898d42799c5789d4d2da3f63: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:41.195309 6627 raft_consensus.cc:1261] T dc7d272c4d324e10959c62362e0d81f2 P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Refusing update from remote peer 0c4821d9898d42799c5789d4d2da3f63: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:41.195680 9924 consensus_queue.cc:1035] T dc7d272c4d324e10959c62362e0d81f2 P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:41.195998 9924 consensus_queue.cc:1035] T dc7d272c4d324e10959c62362e0d81f2 P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:41.196174 6627 consensus_queue.cc:237] T a388d59f6ba94e91958f55b340343fa6 P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } }
I20240503 01:59:41.197381 8080 raft_consensus.cc:1261] T a388d59f6ba94e91958f55b340343fa6 P 9264f24d028347d98f7e4e2d1aedeb79 [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:41.197553 6365 raft_consensus.cc:1261] T a388d59f6ba94e91958f55b340343fa6 P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:41.197539 9931 raft_consensus.cc:2932] T dc7d272c4d324e10959c62362e0d81f2 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 7a1404fe1c1b4b388b821c8261b53104 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: true } } }
I20240503 01:59:41.197835 10101 consensus_queue.cc:1035] T a388d59f6ba94e91958f55b340343fa6 P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:41.198282 6757 raft_consensus.cc:2932] T dc7d272c4d324e10959c62362e0d81f2 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 7a1404fe1c1b4b388b821c8261b53104 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: true } } }
I20240503 01:59:41.199606 9472 consensus_queue.cc:1035] T a388d59f6ba94e91958f55b340343fa6 P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:41.199493 10101 raft_consensus.cc:2932] T a388d59f6ba94e91958f55b340343fa6 P 79163c69f6f04a9abd0208d74e1c61be [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } }
I20240503 01:59:41.200001 6219 catalog_manager.cc:5484] T dc7d272c4d324e10959c62362e0d81f2 P 0c4821d9898d42799c5789d4d2da3f63 reported cstate change: config changed from index -1 to 4, NON_VOTER 7a1404fe1c1b4b388b821c8261b53104 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "0c4821d9898d42799c5789d4d2da3f63" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:41.201602 6365 raft_consensus.cc:2932] T a388d59f6ba94e91958f55b340343fa6 P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } }
I20240503 01:59:41.201967 6208 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a388d59f6ba94e91958f55b340343fa6 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:41.202659 6625 raft_consensus.cc:2932] T dc7d272c4d324e10959c62362e0d81f2 P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 7a1404fe1c1b4b388b821c8261b53104 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: true } } }
I20240503 01:59:41.203469 6218 catalog_manager.cc:5484] T a388d59f6ba94e91958f55b340343fa6 P 79163c69f6f04a9abd0208d74e1c61be reported cstate change: config changed from index 5 to 6, VOTER 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "79163c69f6f04a9abd0208d74e1c61be" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:41.204216 8080 raft_consensus.cc:2932] T a388d59f6ba94e91958f55b340343fa6 P 9264f24d028347d98f7e4e2d1aedeb79 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } }
I20240503 01:59:41.207924 6476 tablet_service.cc:1514] Processing DeleteTablet for tablet a388d59f6ba94e91958f55b340343fa6 with delete_type TABLET_DATA_TOMBSTONED (TS 0c4821d9898d42799c5789d4d2da3f63 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:39850
I20240503 01:59:41.208164 10112 tablet_replica.cc:330] T a388d59f6ba94e91958f55b340343fa6 P 0c4821d9898d42799c5789d4d2da3f63: stopping tablet replica
I20240503 01:59:41.208294 10112 raft_consensus.cc:2229] T a388d59f6ba94e91958f55b340343fa6 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:41.208565 10112 raft_consensus.cc:2258] T a388d59f6ba94e91958f55b340343fa6 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:41.209314 10112 ts_tablet_manager.cc:1875] T a388d59f6ba94e91958f55b340343fa6 P 0c4821d9898d42799c5789d4d2da3f63: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:41.210992 10112 ts_tablet_manager.cc:1888] T a388d59f6ba94e91958f55b340343fa6 P 0c4821d9898d42799c5789d4d2da3f63: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 01:59:41.211056 10112 log.cc:1196] T a388d59f6ba94e91958f55b340343fa6 P 0c4821d9898d42799c5789d4d2da3f63: Deleting WAL directory at /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/a388d59f6ba94e91958f55b340343fa6
I20240503 01:59:41.211586 6210 catalog_manager.cc:4830] TS 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130:34189): tablet a388d59f6ba94e91958f55b340343fa6 (table rebalance_test_table_1 [id=91bdb83bc45b4b61b094a0180eacf8e3]) successfully deleted
I20240503 01:59:41.216496 6625 consensus_queue.cc:237] T c3337c95b354418faf37684e7440b1c7 P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: true } }
I20240503 01:59:41.217855 6757 raft_consensus.cc:1261] T c3337c95b354418faf37684e7440b1c7 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:41.217953 6494 raft_consensus.cc:1261] T c3337c95b354418faf37684e7440b1c7 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:41.218272 9472 consensus_queue.cc:1035] T c3337c95b354418faf37684e7440b1c7 P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 01:59:41.218441 6563 consensus_peers.cc:485] T c3337c95b354418faf37684e7440b1c7 P 79163c69f6f04a9abd0208d74e1c61be -> Peer de2f076398ee44008622fb5db2a75952 (127.0.99.135:34035): Couldn't send request to peer de2f076398ee44008622fb5db2a75952. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c3337c95b354418faf37684e7440b1c7. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:41.218463 7647 consensus_queue.cc:1035] T c3337c95b354418faf37684e7440b1c7 P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:41.220132 10101 raft_consensus.cc:2932] T c3337c95b354418faf37684e7440b1c7 P 79163c69f6f04a9abd0208d74e1c61be [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER de2f076398ee44008622fb5db2a75952 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: true } } }
I20240503 01:59:41.220552 6494 raft_consensus.cc:2932] T c3337c95b354418faf37684e7440b1c7 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER de2f076398ee44008622fb5db2a75952 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: true } } }
I20240503 01:59:41.221891 6757 raft_consensus.cc:2932] T c3337c95b354418faf37684e7440b1c7 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER de2f076398ee44008622fb5db2a75952 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: true } } }
I20240503 01:59:41.224193 6214 catalog_manager.cc:5484] T c3337c95b354418faf37684e7440b1c7 P 0c4821d9898d42799c5789d4d2da3f63 reported cstate change: config changed from index -1 to 4, NON_VOTER de2f076398ee44008622fb5db2a75952 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "79163c69f6f04a9abd0208d74e1c61be" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: true } } }
I20240503 01:59:41.271021 6365 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4e7ceb1453804246a41bd1dda0e39d02"
dest_uuid: "4127249ba42644388d225d614c935a25"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:33274
I20240503 01:59:41.271174 6365 raft_consensus.cc:595] T 4e7ceb1453804246a41bd1dda0e39d02 P 4127249ba42644388d225d614c935a25 [term 1 LEADER]: Received request to transfer leadership
I20240503 01:59:41.312430 10533 ts_tablet_manager.cc:902] T a4983426b2d843e7a9ed335c79178678 P 65d4632bcdd54ba596acdbbe1b5ef8af: Initiating tablet copy from peer 4127249ba42644388d225d614c935a25 (127.0.99.129:35613)
I20240503 01:59:41.312948 10533 tablet_copy_client.cc:308] T a4983426b2d843e7a9ed335c79178678 P 65d4632bcdd54ba596acdbbe1b5ef8af: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:35613
I20240503 01:59:41.313446 6385 tablet_copy_service.cc:140] P 4127249ba42644388d225d614c935a25: Received BeginTabletCopySession request for tablet a4983426b2d843e7a9ed335c79178678 from peer 65d4632bcdd54ba596acdbbe1b5ef8af ({username='slave'} at 127.0.99.134:49429)
I20240503 01:59:41.313542 6385 tablet_copy_service.cc:161] P 4127249ba42644388d225d614c935a25: Beginning new tablet copy session on tablet a4983426b2d843e7a9ed335c79178678 from peer 65d4632bcdd54ba596acdbbe1b5ef8af at {username='slave'} at 127.0.99.134:49429: session id = 65d4632bcdd54ba596acdbbe1b5ef8af-a4983426b2d843e7a9ed335c79178678
I20240503 01:59:41.314651 6385 tablet_copy_source_session.cc:205] T a4983426b2d843e7a9ed335c79178678 P 4127249ba42644388d225d614c935a25: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:41.315464 10533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4983426b2d843e7a9ed335c79178678. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:41.317749 10533 tablet_copy_client.cc:783] T a4983426b2d843e7a9ed335c79178678 P 65d4632bcdd54ba596acdbbe1b5ef8af: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:41.317905 10533 tablet_copy_client.cc:647] T a4983426b2d843e7a9ed335c79178678 P 65d4632bcdd54ba596acdbbe1b5ef8af: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:41.318917 10534 tablet_copy_client.cc:1002] T a4983426b2d843e7a9ed335c79178678 P 65d4632bcdd54ba596acdbbe1b5ef8af: tablet copy: Injecting latency into file download: 1500
I20240503 01:59:41.338178 6365 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4e7ceb1453804246a41bd1dda0e39d02"
dest_uuid: "4127249ba42644388d225d614c935a25"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:33316
I20240503 01:59:41.338306 6365 raft_consensus.cc:595] T 4e7ceb1453804246a41bd1dda0e39d02 P 4127249ba42644388d225d614c935a25 [term 1 LEADER]: Received request to transfer leadership
I20240503 01:59:41.360817 10573 ts_tablet_manager.cc:902] T 5b599bd1fea6409fbe485c7a35c0891c P 7a1404fe1c1b4b388b821c8261b53104: Initiating tablet copy from peer 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131:44005)
I20240503 01:59:41.361254 10573 tablet_copy_client.cc:308] T 5b599bd1fea6409fbe485c7a35c0891c P 7a1404fe1c1b4b388b821c8261b53104: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:44005
I20240503 01:59:41.361711 6647 tablet_copy_service.cc:140] P 79163c69f6f04a9abd0208d74e1c61be: Received BeginTabletCopySession request for tablet 5b599bd1fea6409fbe485c7a35c0891c from peer 7a1404fe1c1b4b388b821c8261b53104 ({username='slave'} at 127.0.99.136:51745)
I20240503 01:59:41.361830 6647 tablet_copy_service.cc:161] P 79163c69f6f04a9abd0208d74e1c61be: Beginning new tablet copy session on tablet 5b599bd1fea6409fbe485c7a35c0891c from peer 7a1404fe1c1b4b388b821c8261b53104 at {username='slave'} at 127.0.99.136:51745: session id = 7a1404fe1c1b4b388b821c8261b53104-5b599bd1fea6409fbe485c7a35c0891c
I20240503 01:59:41.362608 6647 tablet_copy_source_session.cc:205] T 5b599bd1fea6409fbe485c7a35c0891c P 79163c69f6f04a9abd0208d74e1c61be: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:41.363397 10573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b599bd1fea6409fbe485c7a35c0891c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:41.365202 10573 tablet_copy_client.cc:783] T 5b599bd1fea6409fbe485c7a35c0891c P 7a1404fe1c1b4b388b821c8261b53104: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:41.365358 10573 tablet_copy_client.cc:647] T 5b599bd1fea6409fbe485c7a35c0891c P 7a1404fe1c1b4b388b821c8261b53104: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:41.366348 10582 tablet_copy_client.cc:1002] T 5b599bd1fea6409fbe485c7a35c0891c P 7a1404fe1c1b4b388b821c8261b53104: tablet copy: Injecting latency into file download: 1500
I20240503 01:59:41.366410 10583 ts_tablet_manager.cc:902] T 897e88ac93484a11829189e027de8bae P de2f076398ee44008622fb5db2a75952: Initiating tablet copy from peer 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130:34189)
I20240503 01:59:41.366905 10583 tablet_copy_client.cc:308] T 897e88ac93484a11829189e027de8bae P de2f076398ee44008622fb5db2a75952: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:34189
I20240503 01:59:41.367341 6516 tablet_copy_service.cc:140] P 0c4821d9898d42799c5789d4d2da3f63: Received BeginTabletCopySession request for tablet 897e88ac93484a11829189e027de8bae from peer de2f076398ee44008622fb5db2a75952 ({username='slave'} at 127.0.99.135:55869)
I20240503 01:59:41.367431 6516 tablet_copy_service.cc:161] P 0c4821d9898d42799c5789d4d2da3f63: Beginning new tablet copy session on tablet 897e88ac93484a11829189e027de8bae from peer de2f076398ee44008622fb5db2a75952 at {username='slave'} at 127.0.99.135:55869: session id = de2f076398ee44008622fb5db2a75952-897e88ac93484a11829189e027de8bae
I20240503 01:59:41.368073 6516 tablet_copy_source_session.cc:205] T 897e88ac93484a11829189e027de8bae P 0c4821d9898d42799c5789d4d2da3f63: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:41.368880 10583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 897e88ac93484a11829189e027de8bae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:41.370826 10583 tablet_copy_client.cc:783] T 897e88ac93484a11829189e027de8bae P de2f076398ee44008622fb5db2a75952: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:41.370960 10583 tablet_copy_client.cc:647] T 897e88ac93484a11829189e027de8bae P de2f076398ee44008622fb5db2a75952: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:41.371908 10587 tablet_copy_client.cc:1002] T 897e88ac93484a11829189e027de8bae P de2f076398ee44008622fb5db2a75952: tablet copy: Injecting latency into file download: 1500
I20240503 01:59:41.418061 10588 ts_tablet_manager.cc:902] T 885b787d0cbb4ddabd47cb7c42a78c74 P 9264f24d028347d98f7e4e2d1aedeb79: Initiating tablet copy from peer 4127249ba42644388d225d614c935a25 (127.0.99.129:35613)
I20240503 01:59:41.418689 10588 tablet_copy_client.cc:308] T 885b787d0cbb4ddabd47cb7c42a78c74 P 9264f24d028347d98f7e4e2d1aedeb79: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:35613
I20240503 01:59:41.419175 6385 tablet_copy_service.cc:140] P 4127249ba42644388d225d614c935a25: Received BeginTabletCopySession request for tablet 885b787d0cbb4ddabd47cb7c42a78c74 from peer 9264f24d028347d98f7e4e2d1aedeb79 ({username='slave'} at 127.0.99.133:57445)
I20240503 01:59:41.419299 6385 tablet_copy_service.cc:161] P 4127249ba42644388d225d614c935a25: Beginning new tablet copy session on tablet 885b787d0cbb4ddabd47cb7c42a78c74 from peer 9264f24d028347d98f7e4e2d1aedeb79 at {username='slave'} at 127.0.99.133:57445: session id = 9264f24d028347d98f7e4e2d1aedeb79-885b787d0cbb4ddabd47cb7c42a78c74
I20240503 01:59:41.420055 6385 tablet_copy_source_session.cc:205] T 885b787d0cbb4ddabd47cb7c42a78c74 P 4127249ba42644388d225d614c935a25: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:41.420841 10588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 885b787d0cbb4ddabd47cb7c42a78c74. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:41.422550 10588 tablet_copy_client.cc:783] T 885b787d0cbb4ddabd47cb7c42a78c74 P 9264f24d028347d98f7e4e2d1aedeb79: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:41.422763 10588 tablet_copy_client.cc:647] T 885b787d0cbb4ddabd47cb7c42a78c74 P 9264f24d028347d98f7e4e2d1aedeb79: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:41.423081 10593 ts_tablet_manager.cc:902] T ccb34a4b52244d90850ab04e21299f55 P 7a1404fe1c1b4b388b821c8261b53104: Initiating tablet copy from peer 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130:34189)
I20240503 01:59:41.423609 10593 tablet_copy_client.cc:308] T ccb34a4b52244d90850ab04e21299f55 P 7a1404fe1c1b4b388b821c8261b53104: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:34189
I20240503 01:59:41.424052 6516 tablet_copy_service.cc:140] P 0c4821d9898d42799c5789d4d2da3f63: Received BeginTabletCopySession request for tablet ccb34a4b52244d90850ab04e21299f55 from peer 7a1404fe1c1b4b388b821c8261b53104 ({username='slave'} at 127.0.99.136:32823)
I20240503 01:59:41.424096 10592 tablet_copy_client.cc:1002] T 885b787d0cbb4ddabd47cb7c42a78c74 P 9264f24d028347d98f7e4e2d1aedeb79: tablet copy: Injecting latency into file download: 1500
I20240503 01:59:41.424158 6516 tablet_copy_service.cc:161] P 0c4821d9898d42799c5789d4d2da3f63: Beginning new tablet copy session on tablet ccb34a4b52244d90850ab04e21299f55 from peer 7a1404fe1c1b4b388b821c8261b53104 at {username='slave'} at 127.0.99.136:32823: session id = 7a1404fe1c1b4b388b821c8261b53104-ccb34a4b52244d90850ab04e21299f55
I20240503 01:59:41.424912 6516 tablet_copy_source_session.cc:205] T ccb34a4b52244d90850ab04e21299f55 P 0c4821d9898d42799c5789d4d2da3f63: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:41.425738 10593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccb34a4b52244d90850ab04e21299f55. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:41.427304 10593 tablet_copy_client.cc:783] T ccb34a4b52244d90850ab04e21299f55 P 7a1404fe1c1b4b388b821c8261b53104: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:41.427466 10593 tablet_copy_client.cc:647] T ccb34a4b52244d90850ab04e21299f55 P 7a1404fe1c1b4b388b821c8261b53104: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:41.428732 10597 tablet_copy_client.cc:1002] T ccb34a4b52244d90850ab04e21299f55 P 7a1404fe1c1b4b388b821c8261b53104: tablet copy: Injecting latency into file download: 1500
I20240503 01:59:41.486709 10120 raft_consensus.cc:979] T 4e7ceb1453804246a41bd1dda0e39d02 P 4127249ba42644388d225d614c935a25: : Instructing follower 65d4632bcdd54ba596acdbbe1b5ef8af to start an election
I20240503 01:59:41.486819 10120 raft_consensus.cc:1067] T 4e7ceb1453804246a41bd1dda0e39d02 P 4127249ba42644388d225d614c935a25 [term 1 LEADER]: Signalling peer 65d4632bcdd54ba596acdbbe1b5ef8af to start an election
I20240503 01:59:41.487368 8224 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4e7ceb1453804246a41bd1dda0e39d02"
dest_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af"
from {username='slave'} at 127.0.99.129:56679
I20240503 01:59:41.487552 8224 raft_consensus.cc:482] T 4e7ceb1453804246a41bd1dda0e39d02 P 65d4632bcdd54ba596acdbbe1b5ef8af [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 01:59:41.487664 8224 raft_consensus.cc:3037] T 4e7ceb1453804246a41bd1dda0e39d02 P 65d4632bcdd54ba596acdbbe1b5ef8af [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:41.488755 8224 raft_consensus.cc:504] T 4e7ceb1453804246a41bd1dda0e39d02 P 65d4632bcdd54ba596acdbbe1b5ef8af [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } }
I20240503 01:59:41.490063 8224 leader_election.cc:288] T 4e7ceb1453804246a41bd1dda0e39d02 P 65d4632bcdd54ba596acdbbe1b5ef8af [CANDIDATE]: Term 2 election: Requested vote from peers 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131:44005), 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130:34189), 4127249ba42644388d225d614c935a25 (127.0.99.129:35613)
I20240503 01:59:41.490700 6494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e7ceb1453804246a41bd1dda0e39d02" candidate_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "0c4821d9898d42799c5789d4d2da3f63"
I20240503 01:59:41.490873 6494 raft_consensus.cc:3037] T 4e7ceb1453804246a41bd1dda0e39d02 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:41.490846 6365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e7ceb1453804246a41bd1dda0e39d02" candidate_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "4127249ba42644388d225d614c935a25"
I20240503 01:59:41.490971 6365 raft_consensus.cc:3032] T 4e7ceb1453804246a41bd1dda0e39d02 P 4127249ba42644388d225d614c935a25 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 01:59:41.491040 6365 raft_consensus.cc:729] T 4e7ceb1453804246a41bd1dda0e39d02 P 4127249ba42644388d225d614c935a25 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4127249ba42644388d225d614c935a25, State: Running, Role: LEADER
I20240503 01:59:41.491205 6365 consensus_queue.cc:260] T 4e7ceb1453804246a41bd1dda0e39d02 P 4127249ba42644388d225d614c935a25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } }
I20240503 01:59:41.491463 6365 raft_consensus.cc:3037] T 4e7ceb1453804246a41bd1dda0e39d02 P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:41.491842 6494 raft_consensus.cc:2454] T 4e7ceb1453804246a41bd1dda0e39d02 P 0c4821d9898d42799c5789d4d2da3f63 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65d4632bcdd54ba596acdbbe1b5ef8af in term 2.
I20240503 01:59:41.492501 6365 raft_consensus.cc:2454] T 4e7ceb1453804246a41bd1dda0e39d02 P 4127249ba42644388d225d614c935a25 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65d4632bcdd54ba596acdbbe1b5ef8af in term 2.
I20240503 01:59:41.492826 8158 leader_election.cc:302] T 4e7ceb1453804246a41bd1dda0e39d02 P 65d4632bcdd54ba596acdbbe1b5ef8af [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0c4821d9898d42799c5789d4d2da3f63, 4127249ba42644388d225d614c935a25, 65d4632bcdd54ba596acdbbe1b5ef8af; no voters:
I20240503 01:59:41.493041 9896 raft_consensus.cc:2781] T 4e7ceb1453804246a41bd1dda0e39d02 P 65d4632bcdd54ba596acdbbe1b5ef8af [term 2 FOLLOWER]: Leader election won for term 2
I20240503 01:59:41.493247 9896 raft_consensus.cc:686] T 4e7ceb1453804246a41bd1dda0e39d02 P 65d4632bcdd54ba596acdbbe1b5ef8af [term 2 LEADER]: Becoming Leader. State: Replica: 65d4632bcdd54ba596acdbbe1b5ef8af, State: Running, Role: LEADER
I20240503 01:59:41.493522 9896 consensus_queue.cc:237] T 4e7ceb1453804246a41bd1dda0e39d02 P 65d4632bcdd54ba596acdbbe1b5ef8af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } }
I20240503 01:59:41.494747 6625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e7ceb1453804246a41bd1dda0e39d02" candidate_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "79163c69f6f04a9abd0208d74e1c61be"
I20240503 01:59:41.494872 6625 raft_consensus.cc:3037] T 4e7ceb1453804246a41bd1dda0e39d02 P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:41.495707 6625 raft_consensus.cc:2454] T 4e7ceb1453804246a41bd1dda0e39d02 P 79163c69f6f04a9abd0208d74e1c61be [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65d4632bcdd54ba596acdbbe1b5ef8af in term 2.
I20240503 01:59:41.496412 6214 catalog_manager.cc:5484] T 4e7ceb1453804246a41bd1dda0e39d02 P 65d4632bcdd54ba596acdbbe1b5ef8af reported cstate change: term changed from 1 to 2, leader changed from 4127249ba42644388d225d614c935a25 (127.0.99.129) to 65d4632bcdd54ba596acdbbe1b5ef8af (127.0.99.134). New cstate: current_term: 2 leader_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:41.721185 10651 ts_tablet_manager.cc:902] T 679fc86ea2ad4bd6a9160fddf4c586d7 P de2f076398ee44008622fb5db2a75952: Initiating tablet copy from peer 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131:44005)
I20240503 01:59:41.721807 10651 tablet_copy_client.cc:308] T 679fc86ea2ad4bd6a9160fddf4c586d7 P de2f076398ee44008622fb5db2a75952: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:44005
I20240503 01:59:41.722308 6647 tablet_copy_service.cc:140] P 79163c69f6f04a9abd0208d74e1c61be: Received BeginTabletCopySession request for tablet 679fc86ea2ad4bd6a9160fddf4c586d7 from peer de2f076398ee44008622fb5db2a75952 ({username='slave'} at 127.0.99.135:58241)
I20240503 01:59:41.722440 6647 tablet_copy_service.cc:161] P 79163c69f6f04a9abd0208d74e1c61be: Beginning new tablet copy session on tablet 679fc86ea2ad4bd6a9160fddf4c586d7 from peer de2f076398ee44008622fb5db2a75952 at {username='slave'} at 127.0.99.135:58241: session id = de2f076398ee44008622fb5db2a75952-679fc86ea2ad4bd6a9160fddf4c586d7
I20240503 01:59:41.723146 6647 tablet_copy_source_session.cc:205] T 679fc86ea2ad4bd6a9160fddf4c586d7 P 79163c69f6f04a9abd0208d74e1c61be: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:41.723963 10651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 679fc86ea2ad4bd6a9160fddf4c586d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:41.725793 10651 tablet_copy_client.cc:783] T 679fc86ea2ad4bd6a9160fddf4c586d7 P de2f076398ee44008622fb5db2a75952: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:41.725926 10651 tablet_copy_client.cc:647] T 679fc86ea2ad4bd6a9160fddf4c586d7 P de2f076398ee44008622fb5db2a75952: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:41.726914 10654 tablet_copy_client.cc:1002] T 679fc86ea2ad4bd6a9160fddf4c586d7 P de2f076398ee44008622fb5db2a75952: tablet copy: Injecting latency into file download: 1500
I20240503 01:59:41.755561 10656 ts_tablet_manager.cc:902] T c3337c95b354418faf37684e7440b1c7 P de2f076398ee44008622fb5db2a75952: Initiating tablet copy from peer 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131:44005)
I20240503 01:59:41.756062 10656 tablet_copy_client.cc:308] T c3337c95b354418faf37684e7440b1c7 P de2f076398ee44008622fb5db2a75952: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:44005
I20240503 01:59:41.756490 6647 tablet_copy_service.cc:140] P 79163c69f6f04a9abd0208d74e1c61be: Received BeginTabletCopySession request for tablet c3337c95b354418faf37684e7440b1c7 from peer de2f076398ee44008622fb5db2a75952 ({username='slave'} at 127.0.99.135:58241)
I20240503 01:59:41.756582 6647 tablet_copy_service.cc:161] P 79163c69f6f04a9abd0208d74e1c61be: Beginning new tablet copy session on tablet c3337c95b354418faf37684e7440b1c7 from peer de2f076398ee44008622fb5db2a75952 at {username='slave'} at 127.0.99.135:58241: session id = de2f076398ee44008622fb5db2a75952-c3337c95b354418faf37684e7440b1c7
I20240503 01:59:41.757293 6647 tablet_copy_source_session.cc:205] T c3337c95b354418faf37684e7440b1c7 P 79163c69f6f04a9abd0208d74e1c61be: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:41.758018 10656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3337c95b354418faf37684e7440b1c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:41.759685 10656 tablet_copy_client.cc:783] T c3337c95b354418faf37684e7440b1c7 P de2f076398ee44008622fb5db2a75952: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:41.759825 10656 tablet_copy_client.cc:647] T c3337c95b354418faf37684e7440b1c7 P de2f076398ee44008622fb5db2a75952: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:41.760797 10660 tablet_copy_client.cc:1002] T c3337c95b354418faf37684e7440b1c7 P de2f076398ee44008622fb5db2a75952: tablet copy: Injecting latency into file download: 1500
I20240503 01:59:41.762926 10661 ts_tablet_manager.cc:902] T 82cfe99cd6fe4a82a7e35298ec634b8b P 9264f24d028347d98f7e4e2d1aedeb79: Initiating tablet copy from peer 4127249ba42644388d225d614c935a25 (127.0.99.129:35613)
I20240503 01:59:41.763288 10661 tablet_copy_client.cc:308] T 82cfe99cd6fe4a82a7e35298ec634b8b P 9264f24d028347d98f7e4e2d1aedeb79: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:35613
I20240503 01:59:41.763693 6385 tablet_copy_service.cc:140] P 4127249ba42644388d225d614c935a25: Received BeginTabletCopySession request for tablet 82cfe99cd6fe4a82a7e35298ec634b8b from peer 9264f24d028347d98f7e4e2d1aedeb79 ({username='slave'} at 127.0.99.133:57445)
I20240503 01:59:41.763775 6385 tablet_copy_service.cc:161] P 4127249ba42644388d225d614c935a25: Beginning new tablet copy session on tablet 82cfe99cd6fe4a82a7e35298ec634b8b from peer 9264f24d028347d98f7e4e2d1aedeb79 at {username='slave'} at 127.0.99.133:57445: session id = 9264f24d028347d98f7e4e2d1aedeb79-82cfe99cd6fe4a82a7e35298ec634b8b
I20240503 01:59:41.764418 6385 tablet_copy_source_session.cc:205] T 82cfe99cd6fe4a82a7e35298ec634b8b P 4127249ba42644388d225d614c935a25: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:41.765127 10661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82cfe99cd6fe4a82a7e35298ec634b8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:41.766708 10661 tablet_copy_client.cc:783] T 82cfe99cd6fe4a82a7e35298ec634b8b P 9264f24d028347d98f7e4e2d1aedeb79: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:41.766846 10661 tablet_copy_client.cc:647] T 82cfe99cd6fe4a82a7e35298ec634b8b P 9264f24d028347d98f7e4e2d1aedeb79: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:41.767758 10665 tablet_copy_client.cc:1002] T 82cfe99cd6fe4a82a7e35298ec634b8b P 9264f24d028347d98f7e4e2d1aedeb79: tablet copy: Injecting latency into file download: 1500
I20240503 01:59:41.771785 10666 ts_tablet_manager.cc:902] T dc7d272c4d324e10959c62362e0d81f2 P 7a1404fe1c1b4b388b821c8261b53104: Initiating tablet copy from peer 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130:34189)
I20240503 01:59:41.772226 10666 tablet_copy_client.cc:308] T dc7d272c4d324e10959c62362e0d81f2 P 7a1404fe1c1b4b388b821c8261b53104: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:34189
I20240503 01:59:41.772753 6516 tablet_copy_service.cc:140] P 0c4821d9898d42799c5789d4d2da3f63: Received BeginTabletCopySession request for tablet dc7d272c4d324e10959c62362e0d81f2 from peer 7a1404fe1c1b4b388b821c8261b53104 ({username='slave'} at 127.0.99.136:32823)
I20240503 01:59:41.772867 6516 tablet_copy_service.cc:161] P 0c4821d9898d42799c5789d4d2da3f63: Beginning new tablet copy session on tablet dc7d272c4d324e10959c62362e0d81f2 from peer 7a1404fe1c1b4b388b821c8261b53104 at {username='slave'} at 127.0.99.136:32823: session id = 7a1404fe1c1b4b388b821c8261b53104-dc7d272c4d324e10959c62362e0d81f2
I20240503 01:59:41.773808 6516 tablet_copy_source_session.cc:205] T dc7d272c4d324e10959c62362e0d81f2 P 0c4821d9898d42799c5789d4d2da3f63: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:41.774703 10666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc7d272c4d324e10959c62362e0d81f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:41.776404 10666 tablet_copy_client.cc:783] T dc7d272c4d324e10959c62362e0d81f2 P 7a1404fe1c1b4b388b821c8261b53104: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:41.776548 10666 tablet_copy_client.cc:647] T dc7d272c4d324e10959c62362e0d81f2 P 7a1404fe1c1b4b388b821c8261b53104: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:41.777484 10670 tablet_copy_client.cc:1002] T dc7d272c4d324e10959c62362e0d81f2 P 7a1404fe1c1b4b388b821c8261b53104: tablet copy: Injecting latency into file download: 1500
I20240503 01:59:41.926321 6365 raft_consensus.cc:1261] T 4e7ceb1453804246a41bd1dda0e39d02 P 4127249ba42644388d225d614c935a25 [term 2 FOLLOWER]: Refusing update from remote peer 65d4632bcdd54ba596acdbbe1b5ef8af: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 01:59:41.926805 9896 consensus_queue.cc:1035] T 4e7ceb1453804246a41bd1dda0e39d02 P 65d4632bcdd54ba596acdbbe1b5ef8af [LEADER]: Connected to new peer: Peer: permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:41.958369 6494 raft_consensus.cc:1261] T 4e7ceb1453804246a41bd1dda0e39d02 P 0c4821d9898d42799c5789d4d2da3f63 [term 2 FOLLOWER]: Refusing update from remote peer 65d4632bcdd54ba596acdbbe1b5ef8af: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 01:59:41.958784 9896 consensus_queue.cc:1035] T 4e7ceb1453804246a41bd1dda0e39d02 P 65d4632bcdd54ba596acdbbe1b5ef8af [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:41.962359 6625 raft_consensus.cc:1261] T 4e7ceb1453804246a41bd1dda0e39d02 P 79163c69f6f04a9abd0208d74e1c61be [term 2 FOLLOWER]: Refusing update from remote peer 65d4632bcdd54ba596acdbbe1b5ef8af: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 01:59:41.963496 9896 consensus_queue.cc:1035] T 4e7ceb1453804246a41bd1dda0e39d02 P 65d4632bcdd54ba596acdbbe1b5ef8af [LEADER]: Connected to new peer: Peer: permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:42.578469 8224 consensus_queue.cc:237] T 4e7ceb1453804246a41bd1dda0e39d02 P 65d4632bcdd54ba596acdbbe1b5ef8af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } }
I20240503 01:59:42.579766 6625 raft_consensus.cc:1261] T 4e7ceb1453804246a41bd1dda0e39d02 P 79163c69f6f04a9abd0208d74e1c61be [term 2 FOLLOWER]: Refusing update from remote peer 65d4632bcdd54ba596acdbbe1b5ef8af: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 01:59:42.579820 6495 raft_consensus.cc:1261] T 4e7ceb1453804246a41bd1dda0e39d02 P 0c4821d9898d42799c5789d4d2da3f63 [term 2 FOLLOWER]: Refusing update from remote peer 65d4632bcdd54ba596acdbbe1b5ef8af: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 01:59:42.580252 10828 consensus_queue.cc:1035] T 4e7ceb1453804246a41bd1dda0e39d02 P 65d4632bcdd54ba596acdbbe1b5ef8af [LEADER]: Connected to new peer: Peer: permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:42.580524 10827 consensus_queue.cc:1035] T 4e7ceb1453804246a41bd1dda0e39d02 P 65d4632bcdd54ba596acdbbe1b5ef8af [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:42.582769 10830 raft_consensus.cc:2932] T 4e7ceb1453804246a41bd1dda0e39d02 P 65d4632bcdd54ba596acdbbe1b5ef8af [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 4127249ba42644388d225d614c935a25 (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } }
I20240503 01:59:42.583104 6625 raft_consensus.cc:2932] T 4e7ceb1453804246a41bd1dda0e39d02 P 79163c69f6f04a9abd0208d74e1c61be [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 4127249ba42644388d225d614c935a25 (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } }
I20240503 01:59:42.583215 6495 raft_consensus.cc:2932] T 4e7ceb1453804246a41bd1dda0e39d02 P 0c4821d9898d42799c5789d4d2da3f63 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 4127249ba42644388d225d614c935a25 (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } }
I20240503 01:59:42.584893 6207 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4e7ceb1453804246a41bd1dda0e39d02 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:42.586484 6214 catalog_manager.cc:5484] T 4e7ceb1453804246a41bd1dda0e39d02 P 65d4632bcdd54ba596acdbbe1b5ef8af reported cstate change: config changed from index 5 to 7, VOTER 4127249ba42644388d225d614c935a25 (127.0.99.129) evicted. New cstate: current_term: 2 leader_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:42.591043 6344 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e7ceb1453804246a41bd1dda0e39d02 with delete_type TABLET_DATA_TOMBSTONED (TS 4127249ba42644388d225d614c935a25 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:57398
I20240503 01:59:42.591517 10840 tablet_replica.cc:330] T 4e7ceb1453804246a41bd1dda0e39d02 P 4127249ba42644388d225d614c935a25: stopping tablet replica
I20240503 01:59:42.591696 10840 raft_consensus.cc:2229] T 4e7ceb1453804246a41bd1dda0e39d02 P 4127249ba42644388d225d614c935a25 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:42.591830 10840 raft_consensus.cc:2258] T 4e7ceb1453804246a41bd1dda0e39d02 P 4127249ba42644388d225d614c935a25 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:42.592545 10840 ts_tablet_manager.cc:1875] T 4e7ceb1453804246a41bd1dda0e39d02 P 4127249ba42644388d225d614c935a25: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:42.595350 10840 ts_tablet_manager.cc:1888] T 4e7ceb1453804246a41bd1dda0e39d02 P 4127249ba42644388d225d614c935a25: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 01:59:42.595435 10840 log.cc:1196] T 4e7ceb1453804246a41bd1dda0e39d02 P 4127249ba42644388d225d614c935a25: Deleting WAL directory at /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/4e7ceb1453804246a41bd1dda0e39d02
I20240503 01:59:42.596086 6208 catalog_manager.cc:4830] TS 4127249ba42644388d225d614c935a25 (127.0.99.129:35613): tablet 4e7ceb1453804246a41bd1dda0e39d02 (table rebalance_test_table_7 [id=dbd6f15c28634e3dac01249aa7cc2e25]) successfully deleted
I20240503 01:59:42.819340 10533 tablet_copy_client.cc:523] T a4983426b2d843e7a9ed335c79178678 P 65d4632bcdd54ba596acdbbe1b5ef8af: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:42.820822 10533 tablet_bootstrap.cc:492] T a4983426b2d843e7a9ed335c79178678 P 65d4632bcdd54ba596acdbbe1b5ef8af: Bootstrap starting.
I20240503 01:59:42.838076 10533 tablet_bootstrap.cc:492] T a4983426b2d843e7a9ed335c79178678 P 65d4632bcdd54ba596acdbbe1b5ef8af: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:42.838338 10533 tablet_bootstrap.cc:492] T a4983426b2d843e7a9ed335c79178678 P 65d4632bcdd54ba596acdbbe1b5ef8af: Bootstrap complete.
I20240503 01:59:42.838510 10533 ts_tablet_manager.cc:1367] T a4983426b2d843e7a9ed335c79178678 P 65d4632bcdd54ba596acdbbe1b5ef8af: Time spent bootstrapping tablet: real 0.018s user 0.013s sys 0.002s
I20240503 01:59:42.838896 10533 raft_consensus.cc:348] T a4983426b2d843e7a9ed335c79178678 P 65d4632bcdd54ba596acdbbe1b5ef8af [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: true } }
I20240503 01:59:42.838999 10533 raft_consensus.cc:729] T a4983426b2d843e7a9ed335c79178678 P 65d4632bcdd54ba596acdbbe1b5ef8af [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 65d4632bcdd54ba596acdbbe1b5ef8af, State: Initialized, Role: LEARNER
I20240503 01:59:42.839071 10533 consensus_queue.cc:260] T a4983426b2d843e7a9ed335c79178678 P 65d4632bcdd54ba596acdbbe1b5ef8af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: true } }
I20240503 01:59:42.839330 10533 ts_tablet_manager.cc:1398] T a4983426b2d843e7a9ed335c79178678 P 65d4632bcdd54ba596acdbbe1b5ef8af: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:42.840025 6385 tablet_copy_service.cc:342] P 4127249ba42644388d225d614c935a25: Request end of tablet copy session 65d4632bcdd54ba596acdbbe1b5ef8af-a4983426b2d843e7a9ed335c79178678 received from {username='slave'} at 127.0.99.134:49429
I20240503 01:59:42.840106 6385 tablet_copy_service.cc:433] P 4127249ba42644388d225d614c935a25: ending tablet copy session 65d4632bcdd54ba596acdbbe1b5ef8af-a4983426b2d843e7a9ed335c79178678 on tablet a4983426b2d843e7a9ed335c79178678 with peer 65d4632bcdd54ba596acdbbe1b5ef8af
I20240503 01:59:42.866818 10573 tablet_copy_client.cc:523] T 5b599bd1fea6409fbe485c7a35c0891c P 7a1404fe1c1b4b388b821c8261b53104: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:42.868746 10573 tablet_bootstrap.cc:492] T 5b599bd1fea6409fbe485c7a35c0891c P 7a1404fe1c1b4b388b821c8261b53104: Bootstrap starting.
I20240503 01:59:42.872301 10583 tablet_copy_client.cc:523] T 897e88ac93484a11829189e027de8bae P de2f076398ee44008622fb5db2a75952: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:42.874387 10583 tablet_bootstrap.cc:492] T 897e88ac93484a11829189e027de8bae P de2f076398ee44008622fb5db2a75952: Bootstrap starting.
I20240503 01:59:42.890111 10573 tablet_bootstrap.cc:492] T 5b599bd1fea6409fbe485c7a35c0891c P 7a1404fe1c1b4b388b821c8261b53104: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:42.890448 10573 tablet_bootstrap.cc:492] T 5b599bd1fea6409fbe485c7a35c0891c P 7a1404fe1c1b4b388b821c8261b53104: Bootstrap complete.
I20240503 01:59:42.890652 10573 ts_tablet_manager.cc:1367] T 5b599bd1fea6409fbe485c7a35c0891c P 7a1404fe1c1b4b388b821c8261b53104: Time spent bootstrapping tablet: real 0.022s user 0.017s sys 0.003s
I20240503 01:59:42.891119 10573 raft_consensus.cc:348] T 5b599bd1fea6409fbe485c7a35c0891c P 7a1404fe1c1b4b388b821c8261b53104 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: true } }
I20240503 01:59:42.891235 10573 raft_consensus.cc:729] T 5b599bd1fea6409fbe485c7a35c0891c P 7a1404fe1c1b4b388b821c8261b53104 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7a1404fe1c1b4b388b821c8261b53104, State: Initialized, Role: LEARNER
I20240503 01:59:42.891323 10573 consensus_queue.cc:260] T 5b599bd1fea6409fbe485c7a35c0891c P 7a1404fe1c1b4b388b821c8261b53104 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: true } }
I20240503 01:59:42.891830 10573 ts_tablet_manager.cc:1398] T 5b599bd1fea6409fbe485c7a35c0891c P 7a1404fe1c1b4b388b821c8261b53104: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 01:59:42.892439 6647 tablet_copy_service.cc:342] P 79163c69f6f04a9abd0208d74e1c61be: Request end of tablet copy session 7a1404fe1c1b4b388b821c8261b53104-5b599bd1fea6409fbe485c7a35c0891c received from {username='slave'} at 127.0.99.136:51745
I20240503 01:59:42.892517 6647 tablet_copy_service.cc:433] P 79163c69f6f04a9abd0208d74e1c61be: ending tablet copy session 7a1404fe1c1b4b388b821c8261b53104-5b599bd1fea6409fbe485c7a35c0891c on tablet 5b599bd1fea6409fbe485c7a35c0891c with peer 7a1404fe1c1b4b388b821c8261b53104
I20240503 01:59:42.897511 10583 tablet_bootstrap.cc:492] T 897e88ac93484a11829189e027de8bae P de2f076398ee44008622fb5db2a75952: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:42.897809 10583 tablet_bootstrap.cc:492] T 897e88ac93484a11829189e027de8bae P de2f076398ee44008622fb5db2a75952: Bootstrap complete.
I20240503 01:59:42.897986 10583 ts_tablet_manager.cc:1367] T 897e88ac93484a11829189e027de8bae P de2f076398ee44008622fb5db2a75952: Time spent bootstrapping tablet: real 0.024s user 0.018s sys 0.003s
I20240503 01:59:42.898418 10583 raft_consensus.cc:348] T 897e88ac93484a11829189e027de8bae P de2f076398ee44008622fb5db2a75952 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: true } }
I20240503 01:59:42.898540 10583 raft_consensus.cc:729] T 897e88ac93484a11829189e027de8bae P de2f076398ee44008622fb5db2a75952 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: de2f076398ee44008622fb5db2a75952, State: Initialized, Role: LEARNER
I20240503 01:59:42.898646 10583 consensus_queue.cc:260] T 897e88ac93484a11829189e027de8bae P de2f076398ee44008622fb5db2a75952 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: true } }
I20240503 01:59:42.898993 10583 ts_tablet_manager.cc:1398] T 897e88ac93484a11829189e027de8bae P de2f076398ee44008622fb5db2a75952: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 01:59:42.899627 6516 tablet_copy_service.cc:342] P 0c4821d9898d42799c5789d4d2da3f63: Request end of tablet copy session de2f076398ee44008622fb5db2a75952-897e88ac93484a11829189e027de8bae received from {username='slave'} at 127.0.99.135:55869
I20240503 01:59:42.899719 6516 tablet_copy_service.cc:433] P 0c4821d9898d42799c5789d4d2da3f63: ending tablet copy session de2f076398ee44008622fb5db2a75952-897e88ac93484a11829189e027de8bae on tablet 897e88ac93484a11829189e027de8bae with peer de2f076398ee44008622fb5db2a75952
I20240503 01:59:42.924401 10588 tablet_copy_client.cc:523] T 885b787d0cbb4ddabd47cb7c42a78c74 P 9264f24d028347d98f7e4e2d1aedeb79: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:42.925884 10588 tablet_bootstrap.cc:492] T 885b787d0cbb4ddabd47cb7c42a78c74 P 9264f24d028347d98f7e4e2d1aedeb79: Bootstrap starting.
I20240503 01:59:42.929025 10593 tablet_copy_client.cc:523] T ccb34a4b52244d90850ab04e21299f55 P 7a1404fe1c1b4b388b821c8261b53104: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:42.930768 10593 tablet_bootstrap.cc:492] T ccb34a4b52244d90850ab04e21299f55 P 7a1404fe1c1b4b388b821c8261b53104: Bootstrap starting.
I20240503 01:59:42.950086 10588 tablet_bootstrap.cc:492] T 885b787d0cbb4ddabd47cb7c42a78c74 P 9264f24d028347d98f7e4e2d1aedeb79: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:42.950428 10588 tablet_bootstrap.cc:492] T 885b787d0cbb4ddabd47cb7c42a78c74 P 9264f24d028347d98f7e4e2d1aedeb79: Bootstrap complete.
I20240503 01:59:42.950606 10588 ts_tablet_manager.cc:1367] T 885b787d0cbb4ddabd47cb7c42a78c74 P 9264f24d028347d98f7e4e2d1aedeb79: Time spent bootstrapping tablet: real 0.025s user 0.022s sys 0.000s
I20240503 01:59:42.951018 10588 raft_consensus.cc:348] T 885b787d0cbb4ddabd47cb7c42a78c74 P 9264f24d028347d98f7e4e2d1aedeb79 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: true } }
I20240503 01:59:42.951130 10588 raft_consensus.cc:729] T 885b787d0cbb4ddabd47cb7c42a78c74 P 9264f24d028347d98f7e4e2d1aedeb79 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9264f24d028347d98f7e4e2d1aedeb79, State: Initialized, Role: LEARNER
I20240503 01:59:42.951236 10588 consensus_queue.cc:260] T 885b787d0cbb4ddabd47cb7c42a78c74 P 9264f24d028347d98f7e4e2d1aedeb79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: true } }
I20240503 01:59:42.951592 10588 ts_tablet_manager.cc:1398] T 885b787d0cbb4ddabd47cb7c42a78c74 P 9264f24d028347d98f7e4e2d1aedeb79: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240503 01:59:42.952103 6385 tablet_copy_service.cc:342] P 4127249ba42644388d225d614c935a25: Request end of tablet copy session 9264f24d028347d98f7e4e2d1aedeb79-885b787d0cbb4ddabd47cb7c42a78c74 received from {username='slave'} at 127.0.99.133:57445
I20240503 01:59:42.952214 6385 tablet_copy_service.cc:433] P 4127249ba42644388d225d614c935a25: ending tablet copy session 9264f24d028347d98f7e4e2d1aedeb79-885b787d0cbb4ddabd47cb7c42a78c74 on tablet 885b787d0cbb4ddabd47cb7c42a78c74 with peer 9264f24d028347d98f7e4e2d1aedeb79
I20240503 01:59:42.956625 10593 tablet_bootstrap.cc:492] T ccb34a4b52244d90850ab04e21299f55 P 7a1404fe1c1b4b388b821c8261b53104: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:42.957027 10593 tablet_bootstrap.cc:492] T ccb34a4b52244d90850ab04e21299f55 P 7a1404fe1c1b4b388b821c8261b53104: Bootstrap complete.
I20240503 01:59:42.957280 10593 ts_tablet_manager.cc:1367] T ccb34a4b52244d90850ab04e21299f55 P 7a1404fe1c1b4b388b821c8261b53104: Time spent bootstrapping tablet: real 0.027s user 0.023s sys 0.000s
I20240503 01:59:42.957852 10593 raft_consensus.cc:348] T ccb34a4b52244d90850ab04e21299f55 P 7a1404fe1c1b4b388b821c8261b53104 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: true } }
I20240503 01:59:42.958024 10593 raft_consensus.cc:729] T ccb34a4b52244d90850ab04e21299f55 P 7a1404fe1c1b4b388b821c8261b53104 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7a1404fe1c1b4b388b821c8261b53104, State: Initialized, Role: LEARNER
I20240503 01:59:42.958150 10593 consensus_queue.cc:260] T ccb34a4b52244d90850ab04e21299f55 P 7a1404fe1c1b4b388b821c8261b53104 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: true } }
I20240503 01:59:42.958454 10593 ts_tablet_manager.cc:1398] T ccb34a4b52244d90850ab04e21299f55 P 7a1404fe1c1b4b388b821c8261b53104: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:42.959144 6516 tablet_copy_service.cc:342] P 0c4821d9898d42799c5789d4d2da3f63: Request end of tablet copy session 7a1404fe1c1b4b388b821c8261b53104-ccb34a4b52244d90850ab04e21299f55 received from {username='slave'} at 127.0.99.136:32823
I20240503 01:59:42.959224 6516 tablet_copy_service.cc:433] P 0c4821d9898d42799c5789d4d2da3f63: ending tablet copy session 7a1404fe1c1b4b388b821c8261b53104-ccb34a4b52244d90850ab04e21299f55 on tablet ccb34a4b52244d90850ab04e21299f55 with peer 7a1404fe1c1b4b388b821c8261b53104
I20240503 01:59:43.025403 8485 raft_consensus.cc:1203] T ccb34a4b52244d90850ab04e21299f55 P 7a1404fe1c1b4b388b821c8261b53104 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 01:59:43.064292 8080 raft_consensus.cc:1203] T 885b787d0cbb4ddabd47cb7c42a78c74 P 9264f24d028347d98f7e4e2d1aedeb79 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 01:59:43.098291 8485 raft_consensus.cc:1203] T 5b599bd1fea6409fbe485c7a35c0891c P 7a1404fe1c1b4b388b821c8261b53104 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 01:59:43.227319 10651 tablet_copy_client.cc:523] T 679fc86ea2ad4bd6a9160fddf4c586d7 P de2f076398ee44008622fb5db2a75952: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:43.229207 10651 tablet_bootstrap.cc:492] T 679fc86ea2ad4bd6a9160fddf4c586d7 P de2f076398ee44008622fb5db2a75952: Bootstrap starting.
I20240503 01:59:43.249346 10651 tablet_bootstrap.cc:492] T 679fc86ea2ad4bd6a9160fddf4c586d7 P de2f076398ee44008622fb5db2a75952: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:43.249611 10651 tablet_bootstrap.cc:492] T 679fc86ea2ad4bd6a9160fddf4c586d7 P de2f076398ee44008622fb5db2a75952: Bootstrap complete.
I20240503 01:59:43.249792 10651 ts_tablet_manager.cc:1367] T 679fc86ea2ad4bd6a9160fddf4c586d7 P de2f076398ee44008622fb5db2a75952: Time spent bootstrapping tablet: real 0.021s user 0.019s sys 0.000s
I20240503 01:59:43.250149 10651 raft_consensus.cc:348] T 679fc86ea2ad4bd6a9160fddf4c586d7 P de2f076398ee44008622fb5db2a75952 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: true } }
I20240503 01:59:43.250252 10651 raft_consensus.cc:729] T 679fc86ea2ad4bd6a9160fddf4c586d7 P de2f076398ee44008622fb5db2a75952 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: de2f076398ee44008622fb5db2a75952, State: Initialized, Role: LEARNER
I20240503 01:59:43.250331 10651 consensus_queue.cc:260] T 679fc86ea2ad4bd6a9160fddf4c586d7 P de2f076398ee44008622fb5db2a75952 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: true } }
I20240503 01:59:43.250535 10651 ts_tablet_manager.cc:1398] T 679fc86ea2ad4bd6a9160fddf4c586d7 P de2f076398ee44008622fb5db2a75952: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:43.251114 6647 tablet_copy_service.cc:342] P 79163c69f6f04a9abd0208d74e1c61be: Request end of tablet copy session de2f076398ee44008622fb5db2a75952-679fc86ea2ad4bd6a9160fddf4c586d7 received from {username='slave'} at 127.0.99.135:58241
I20240503 01:59:43.251236 6647 tablet_copy_service.cc:433] P 79163c69f6f04a9abd0208d74e1c61be: ending tablet copy session de2f076398ee44008622fb5db2a75952-679fc86ea2ad4bd6a9160fddf4c586d7 on tablet 679fc86ea2ad4bd6a9160fddf4c586d7 with peer de2f076398ee44008622fb5db2a75952
I20240503 01:59:43.261082 10656 tablet_copy_client.cc:523] T c3337c95b354418faf37684e7440b1c7 P de2f076398ee44008622fb5db2a75952: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:43.262590 10656 tablet_bootstrap.cc:492] T c3337c95b354418faf37684e7440b1c7 P de2f076398ee44008622fb5db2a75952: Bootstrap starting.
I20240503 01:59:43.268043 10661 tablet_copy_client.cc:523] T 82cfe99cd6fe4a82a7e35298ec634b8b P 9264f24d028347d98f7e4e2d1aedeb79: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:43.269402 10661 tablet_bootstrap.cc:492] T 82cfe99cd6fe4a82a7e35298ec634b8b P 9264f24d028347d98f7e4e2d1aedeb79: Bootstrap starting.
I20240503 01:59:43.277818 10666 tablet_copy_client.cc:523] T dc7d272c4d324e10959c62362e0d81f2 P 7a1404fe1c1b4b388b821c8261b53104: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:43.279804 10666 tablet_bootstrap.cc:492] T dc7d272c4d324e10959c62362e0d81f2 P 7a1404fe1c1b4b388b821c8261b53104: Bootstrap starting.
I20240503 01:59:43.287727 10656 tablet_bootstrap.cc:492] T c3337c95b354418faf37684e7440b1c7 P de2f076398ee44008622fb5db2a75952: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:43.288039 10656 tablet_bootstrap.cc:492] T c3337c95b354418faf37684e7440b1c7 P de2f076398ee44008622fb5db2a75952: Bootstrap complete.
I20240503 01:59:43.288252 10656 ts_tablet_manager.cc:1367] T c3337c95b354418faf37684e7440b1c7 P de2f076398ee44008622fb5db2a75952: Time spent bootstrapping tablet: real 0.026s user 0.024s sys 0.000s
I20240503 01:59:43.288836 10661 tablet_bootstrap.cc:492] T 82cfe99cd6fe4a82a7e35298ec634b8b P 9264f24d028347d98f7e4e2d1aedeb79: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:43.288818 10656 raft_consensus.cc:348] T c3337c95b354418faf37684e7440b1c7 P de2f076398ee44008622fb5db2a75952 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: true } }
I20240503 01:59:43.288978 10656 raft_consensus.cc:729] T c3337c95b354418faf37684e7440b1c7 P de2f076398ee44008622fb5db2a75952 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: de2f076398ee44008622fb5db2a75952, State: Initialized, Role: LEARNER
I20240503 01:59:43.289119 10661 tablet_bootstrap.cc:492] T 82cfe99cd6fe4a82a7e35298ec634b8b P 9264f24d028347d98f7e4e2d1aedeb79: Bootstrap complete.
I20240503 01:59:43.289103 10656 consensus_queue.cc:260] T c3337c95b354418faf37684e7440b1c7 P de2f076398ee44008622fb5db2a75952 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: true } }
I20240503 01:59:43.289299 10661 ts_tablet_manager.cc:1367] T 82cfe99cd6fe4a82a7e35298ec634b8b P 9264f24d028347d98f7e4e2d1aedeb79: Time spent bootstrapping tablet: real 0.020s user 0.016s sys 0.003s
I20240503 01:59:43.289438 10656 ts_tablet_manager.cc:1398] T c3337c95b354418faf37684e7440b1c7 P de2f076398ee44008622fb5db2a75952: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240503 01:59:43.289783 10661 raft_consensus.cc:348] T 82cfe99cd6fe4a82a7e35298ec634b8b P 9264f24d028347d98f7e4e2d1aedeb79 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: true } }
I20240503 01:59:43.289942 10661 raft_consensus.cc:729] T 82cfe99cd6fe4a82a7e35298ec634b8b P 9264f24d028347d98f7e4e2d1aedeb79 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9264f24d028347d98f7e4e2d1aedeb79, State: Initialized, Role: LEARNER
I20240503 01:59:43.290064 10661 consensus_queue.cc:260] T 82cfe99cd6fe4a82a7e35298ec634b8b P 9264f24d028347d98f7e4e2d1aedeb79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: true } }
I20240503 01:59:43.290230 6647 tablet_copy_service.cc:342] P 79163c69f6f04a9abd0208d74e1c61be: Request end of tablet copy session de2f076398ee44008622fb5db2a75952-c3337c95b354418faf37684e7440b1c7 received from {username='slave'} at 127.0.99.135:58241
I20240503 01:59:43.290311 6647 tablet_copy_service.cc:433] P 79163c69f6f04a9abd0208d74e1c61be: ending tablet copy session de2f076398ee44008622fb5db2a75952-c3337c95b354418faf37684e7440b1c7 on tablet c3337c95b354418faf37684e7440b1c7 with peer de2f076398ee44008622fb5db2a75952
I20240503 01:59:43.291201 10661 ts_tablet_manager.cc:1398] T 82cfe99cd6fe4a82a7e35298ec634b8b P 9264f24d028347d98f7e4e2d1aedeb79: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:43.291662 6385 tablet_copy_service.cc:342] P 4127249ba42644388d225d614c935a25: Request end of tablet copy session 9264f24d028347d98f7e4e2d1aedeb79-82cfe99cd6fe4a82a7e35298ec634b8b received from {username='slave'} at 127.0.99.133:57445
I20240503 01:59:43.291754 6385 tablet_copy_service.cc:433] P 4127249ba42644388d225d614c935a25: ending tablet copy session 9264f24d028347d98f7e4e2d1aedeb79-82cfe99cd6fe4a82a7e35298ec634b8b on tablet 82cfe99cd6fe4a82a7e35298ec634b8b with peer 9264f24d028347d98f7e4e2d1aedeb79
I20240503 01:59:43.307178 10666 tablet_bootstrap.cc:492] T dc7d272c4d324e10959c62362e0d81f2 P 7a1404fe1c1b4b388b821c8261b53104: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:43.307528 10666 tablet_bootstrap.cc:492] T dc7d272c4d324e10959c62362e0d81f2 P 7a1404fe1c1b4b388b821c8261b53104: Bootstrap complete.
I20240503 01:59:43.307751 10666 ts_tablet_manager.cc:1367] T dc7d272c4d324e10959c62362e0d81f2 P 7a1404fe1c1b4b388b821c8261b53104: Time spent bootstrapping tablet: real 0.028s user 0.024s sys 0.000s
I20240503 01:59:43.308233 10666 raft_consensus.cc:348] T dc7d272c4d324e10959c62362e0d81f2 P 7a1404fe1c1b4b388b821c8261b53104 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: true } }
I20240503 01:59:43.308357 10666 raft_consensus.cc:729] T dc7d272c4d324e10959c62362e0d81f2 P 7a1404fe1c1b4b388b821c8261b53104 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7a1404fe1c1b4b388b821c8261b53104, State: Initialized, Role: LEARNER
I20240503 01:59:43.308463 10666 consensus_queue.cc:260] T dc7d272c4d324e10959c62362e0d81f2 P 7a1404fe1c1b4b388b821c8261b53104 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: true } }
I20240503 01:59:43.308800 10666 ts_tablet_manager.cc:1398] T dc7d272c4d324e10959c62362e0d81f2 P 7a1404fe1c1b4b388b821c8261b53104: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240503 01:59:43.309386 6516 tablet_copy_service.cc:342] P 0c4821d9898d42799c5789d4d2da3f63: Request end of tablet copy session 7a1404fe1c1b4b388b821c8261b53104-dc7d272c4d324e10959c62362e0d81f2 received from {username='slave'} at 127.0.99.136:32823
I20240503 01:59:43.309502 6516 tablet_copy_service.cc:433] P 0c4821d9898d42799c5789d4d2da3f63: ending tablet copy session 7a1404fe1c1b4b388b821c8261b53104-dc7d272c4d324e10959c62362e0d81f2 on tablet dc7d272c4d324e10959c62362e0d81f2 with peer 7a1404fe1c1b4b388b821c8261b53104
I20240503 01:59:43.318325 8485 raft_consensus.cc:1203] T dc7d272c4d324e10959c62362e0d81f2 P 7a1404fe1c1b4b388b821c8261b53104 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 01:59:43.367400 8080 raft_consensus.cc:1203] T 82cfe99cd6fe4a82a7e35298ec634b8b P 9264f24d028347d98f7e4e2d1aedeb79 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 01:59:43.405270 8354 raft_consensus.cc:1203] T 897e88ac93484a11829189e027de8bae P de2f076398ee44008622fb5db2a75952 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 01:59:43.421175 8224 raft_consensus.cc:1203] T a4983426b2d843e7a9ed335c79178678 P 65d4632bcdd54ba596acdbbe1b5ef8af [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 01:59:43.455602 8354 raft_consensus.cc:1203] T 679fc86ea2ad4bd6a9160fddf4c586d7 P de2f076398ee44008622fb5db2a75952 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 01:59:43.490059 9924 raft_consensus.cc:1050] T ccb34a4b52244d90850ab04e21299f55 P 0c4821d9898d42799c5789d4d2da3f63: attempting to promote NON_VOTER 7a1404fe1c1b4b388b821c8261b53104 to VOTER
I20240503 01:59:43.490512 9924 consensus_queue.cc:237] T ccb34a4b52244d90850ab04e21299f55 P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } }
I20240503 01:59:43.492178 8485 raft_consensus.cc:1261] T ccb34a4b52244d90850ab04e21299f55 P 7a1404fe1c1b4b388b821c8261b53104 [term 1 LEARNER]: Refusing update from remote peer 0c4821d9898d42799c5789d4d2da3f63: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:43.492269 6363 raft_consensus.cc:1261] T ccb34a4b52244d90850ab04e21299f55 P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Refusing update from remote peer 0c4821d9898d42799c5789d4d2da3f63: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:43.492561 6625 raft_consensus.cc:1261] T ccb34a4b52244d90850ab04e21299f55 P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Refusing update from remote peer 0c4821d9898d42799c5789d4d2da3f63: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:43.492808 11078 consensus_queue.cc:1035] T ccb34a4b52244d90850ab04e21299f55 P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:43.493077 11075 consensus_queue.cc:1035] T ccb34a4b52244d90850ab04e21299f55 P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:43.493203 11077 consensus_queue.cc:1035] T ccb34a4b52244d90850ab04e21299f55 P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:43.495448 11075 raft_consensus.cc:2932] T ccb34a4b52244d90850ab04e21299f55 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7a1404fe1c1b4b388b821c8261b53104 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } }
I20240503 01:59:43.496798 6363 raft_consensus.cc:2932] T ccb34a4b52244d90850ab04e21299f55 P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7a1404fe1c1b4b388b821c8261b53104 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } }
I20240503 01:59:43.499787 6218 catalog_manager.cc:5484] T ccb34a4b52244d90850ab04e21299f55 P 0c4821d9898d42799c5789d4d2da3f63 reported cstate change: config changed from index 4 to 5, 7a1404fe1c1b4b388b821c8261b53104 (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0c4821d9898d42799c5789d4d2da3f63" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:43.499759 8485 raft_consensus.cc:2932] T ccb34a4b52244d90850ab04e21299f55 P 7a1404fe1c1b4b388b821c8261b53104 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7a1404fe1c1b4b388b821c8261b53104 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } }
I20240503 01:59:43.500965 10937 raft_consensus.cc:1050] T 885b787d0cbb4ddabd47cb7c42a78c74 P 4127249ba42644388d225d614c935a25: attempting to promote NON_VOTER 9264f24d028347d98f7e4e2d1aedeb79 to VOTER
I20240503 01:59:43.501399 10937 consensus_queue.cc:237] T 885b787d0cbb4ddabd47cb7c42a78c74 P 4127249ba42644388d225d614c935a25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } }
I20240503 01:59:43.503683 6757 raft_consensus.cc:1261] T 885b787d0cbb4ddabd47cb7c42a78c74 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Refusing update from remote peer 4127249ba42644388d225d614c935a25: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:43.504372 10120 consensus_queue.cc:1035] T 885b787d0cbb4ddabd47cb7c42a78c74 P 4127249ba42644388d225d614c935a25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:43.504854 8080 raft_consensus.cc:1261] T 885b787d0cbb4ddabd47cb7c42a78c74 P 9264f24d028347d98f7e4e2d1aedeb79 [term 1 LEARNER]: Refusing update from remote peer 4127249ba42644388d225d614c935a25: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:43.505875 10120 consensus_queue.cc:1035] T 885b787d0cbb4ddabd47cb7c42a78c74 P 4127249ba42644388d225d614c935a25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:43.505970 6495 consensus_queue.cc:237] T ccb34a4b52244d90850ab04e21299f55 P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } }
I20240503 01:59:43.506435 6625 raft_consensus.cc:2932] T ccb34a4b52244d90850ab04e21299f55 P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7a1404fe1c1b4b388b821c8261b53104 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } }
I20240503 01:59:43.507407 8485 raft_consensus.cc:1261] T ccb34a4b52244d90850ab04e21299f55 P 7a1404fe1c1b4b388b821c8261b53104 [term 1 FOLLOWER]: Refusing update from remote peer 0c4821d9898d42799c5789d4d2da3f63: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:43.507853 11077 consensus_queue.cc:1035] T ccb34a4b52244d90850ab04e21299f55 P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:43.509423 6625 raft_consensus.cc:1261] T ccb34a4b52244d90850ab04e21299f55 P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Refusing update from remote peer 0c4821d9898d42799c5789d4d2da3f63: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:43.509989 11077 consensus_queue.cc:1035] T ccb34a4b52244d90850ab04e21299f55 P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:43.510619 6495 raft_consensus.cc:1261] T 885b787d0cbb4ddabd47cb7c42a78c74 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Refusing update from remote peer 4127249ba42644388d225d614c935a25: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:43.511407 10120 consensus_queue.cc:1035] T 885b787d0cbb4ddabd47cb7c42a78c74 P 4127249ba42644388d225d614c935a25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:43.511518 11075 raft_consensus.cc:2932] T ccb34a4b52244d90850ab04e21299f55 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4127249ba42644388d225d614c935a25 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } }
I20240503 01:59:43.511621 10937 raft_consensus.cc:2932] T 885b787d0cbb4ddabd47cb7c42a78c74 P 4127249ba42644388d225d614c935a25 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9264f24d028347d98f7e4e2d1aedeb79 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } }
I20240503 01:59:43.512027 8485 raft_consensus.cc:2932] T ccb34a4b52244d90850ab04e21299f55 P 7a1404fe1c1b4b388b821c8261b53104 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4127249ba42644388d225d614c935a25 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } }
I20240503 01:59:43.512506 6495 raft_consensus.cc:2932] T 885b787d0cbb4ddabd47cb7c42a78c74 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9264f24d028347d98f7e4e2d1aedeb79 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } }
I20240503 01:59:43.513792 6210 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ccb34a4b52244d90850ab04e21299f55 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:43.513772 6757 raft_consensus.cc:2932] T 885b787d0cbb4ddabd47cb7c42a78c74 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9264f24d028347d98f7e4e2d1aedeb79 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } }
I20240503 01:59:43.514732 6214 catalog_manager.cc:5484] T ccb34a4b52244d90850ab04e21299f55 P 7a1404fe1c1b4b388b821c8261b53104 reported cstate change: config changed from index 5 to 6, VOTER 4127249ba42644388d225d614c935a25 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "0c4821d9898d42799c5789d4d2da3f63" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } }
I20240503 01:59:43.515802 6221 catalog_manager.cc:5484] T 885b787d0cbb4ddabd47cb7c42a78c74 P 4127249ba42644388d225d614c935a25 reported cstate change: config changed from index 4 to 5, 9264f24d028347d98f7e4e2d1aedeb79 (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4127249ba42644388d225d614c935a25" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:43.516759 8080 raft_consensus.cc:2932] T 885b787d0cbb4ddabd47cb7c42a78c74 P 9264f24d028347d98f7e4e2d1aedeb79 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9264f24d028347d98f7e4e2d1aedeb79 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } }
I20240503 01:59:43.517686 6344 tablet_service.cc:1514] Processing DeleteTablet for tablet ccb34a4b52244d90850ab04e21299f55 with delete_type TABLET_DATA_TOMBSTONED (TS 4127249ba42644388d225d614c935a25 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:57398
I20240503 01:59:43.519966 11097 tablet_replica.cc:330] T ccb34a4b52244d90850ab04e21299f55 P 4127249ba42644388d225d614c935a25: stopping tablet replica
I20240503 01:59:43.520140 11097 raft_consensus.cc:2229] T ccb34a4b52244d90850ab04e21299f55 P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:43.520321 11097 raft_consensus.cc:2258] T ccb34a4b52244d90850ab04e21299f55 P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:43.520529 6363 consensus_queue.cc:237] T 885b787d0cbb4ddabd47cb7c42a78c74 P 4127249ba42644388d225d614c935a25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } }
I20240503 01:59:43.521030 11097 ts_tablet_manager.cc:1875] T ccb34a4b52244d90850ab04e21299f55 P 4127249ba42644388d225d614c935a25: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:43.521777 6757 raft_consensus.cc:1261] T 885b787d0cbb4ddabd47cb7c42a78c74 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Refusing update from remote peer 4127249ba42644388d225d614c935a25: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:43.522151 8080 raft_consensus.cc:1261] T 885b787d0cbb4ddabd47cb7c42a78c74 P 9264f24d028347d98f7e4e2d1aedeb79 [term 1 FOLLOWER]: Refusing update from remote peer 4127249ba42644388d225d614c935a25: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:43.522434 6625 raft_consensus.cc:2932] T ccb34a4b52244d90850ab04e21299f55 P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4127249ba42644388d225d614c935a25 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } }
I20240503 01:59:43.523236 11097 ts_tablet_manager.cc:1888] T ccb34a4b52244d90850ab04e21299f55 P 4127249ba42644388d225d614c935a25: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 01:59:43.523308 11097 log.cc:1196] T ccb34a4b52244d90850ab04e21299f55 P 4127249ba42644388d225d614c935a25: Deleting WAL directory at /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/ccb34a4b52244d90850ab04e21299f55
I20240503 01:59:43.523862 6208 catalog_manager.cc:4830] TS 4127249ba42644388d225d614c935a25 (127.0.99.129:35613): tablet ccb34a4b52244d90850ab04e21299f55 (table rebalance_test_table_8 [id=85186d17f9464a408a3790b0978eb502]) successfully deleted
I20240503 01:59:43.524091 11087 consensus_queue.cc:1035] T 885b787d0cbb4ddabd47cb7c42a78c74 P 4127249ba42644388d225d614c935a25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:43.524597 11087 consensus_queue.cc:1035] T 885b787d0cbb4ddabd47cb7c42a78c74 P 4127249ba42644388d225d614c935a25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:43.526093 11090 raft_consensus.cc:2932] T 885b787d0cbb4ddabd47cb7c42a78c74 P 4127249ba42644388d225d614c935a25 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } }
I20240503 01:59:43.527609 6208 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 885b787d0cbb4ddabd47cb7c42a78c74 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:43.527650 8080 raft_consensus.cc:2932] T 885b787d0cbb4ddabd47cb7c42a78c74 P 9264f24d028347d98f7e4e2d1aedeb79 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } }
I20240503 01:59:43.528026 6757 raft_consensus.cc:2932] T 885b787d0cbb4ddabd47cb7c42a78c74 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } }
I20240503 01:59:43.530206 6214 catalog_manager.cc:5484] T 885b787d0cbb4ddabd47cb7c42a78c74 P 9264f24d028347d98f7e4e2d1aedeb79 reported cstate change: config changed from index 5 to 6, VOTER 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "4127249ba42644388d225d614c935a25" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } }
I20240503 01:59:43.531235 11098 raft_consensus.cc:1050] T 5b599bd1fea6409fbe485c7a35c0891c P 79163c69f6f04a9abd0208d74e1c61be: attempting to promote NON_VOTER 7a1404fe1c1b4b388b821c8261b53104 to VOTER
I20240503 01:59:43.531584 11098 consensus_queue.cc:237] T 5b599bd1fea6409fbe485c7a35c0891c P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } }
I20240503 01:59:43.532949 6476 tablet_service.cc:1514] Processing DeleteTablet for tablet 885b787d0cbb4ddabd47cb7c42a78c74 with delete_type TABLET_DATA_TOMBSTONED (TS 0c4821d9898d42799c5789d4d2da3f63 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:39850
I20240503 01:59:43.533360 6363 raft_consensus.cc:1261] T 5b599bd1fea6409fbe485c7a35c0891c P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:43.533403 11100 tablet_replica.cc:330] T 885b787d0cbb4ddabd47cb7c42a78c74 P 0c4821d9898d42799c5789d4d2da3f63: stopping tablet replica
I20240503 01:59:43.533548 11100 raft_consensus.cc:2229] T 885b787d0cbb4ddabd47cb7c42a78c74 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:43.533740 11100 raft_consensus.cc:2258] T 885b787d0cbb4ddabd47cb7c42a78c74 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:43.533807 7647 consensus_queue.cc:1035] T 5b599bd1fea6409fbe485c7a35c0891c P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:43.534497 8485 raft_consensus.cc:1261] T 5b599bd1fea6409fbe485c7a35c0891c P 7a1404fe1c1b4b388b821c8261b53104 [term 1 LEARNER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:43.534957 7647 consensus_queue.cc:1035] T 5b599bd1fea6409fbe485c7a35c0891c P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:43.535310 11100 ts_tablet_manager.cc:1875] T 885b787d0cbb4ddabd47cb7c42a78c74 P 0c4821d9898d42799c5789d4d2da3f63: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:43.535816 6495 raft_consensus.cc:1261] T 5b599bd1fea6409fbe485c7a35c0891c P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:43.536767 11098 consensus_queue.cc:1035] T 5b599bd1fea6409fbe485c7a35c0891c P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:43.537369 7647 raft_consensus.cc:2932] T 5b599bd1fea6409fbe485c7a35c0891c P 79163c69f6f04a9abd0208d74e1c61be [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7a1404fe1c1b4b388b821c8261b53104 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } }
I20240503 01:59:43.538396 11100 ts_tablet_manager.cc:1888] T 885b787d0cbb4ddabd47cb7c42a78c74 P 0c4821d9898d42799c5789d4d2da3f63: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 01:59:43.538517 11100 log.cc:1196] T 885b787d0cbb4ddabd47cb7c42a78c74 P 0c4821d9898d42799c5789d4d2da3f63: Deleting WAL directory at /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/885b787d0cbb4ddabd47cb7c42a78c74
I20240503 01:59:43.539211 6210 catalog_manager.cc:4830] TS 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130:34189): tablet 885b787d0cbb4ddabd47cb7c42a78c74 (table rebalance_test_table_2 [id=4a248d80763944458620bf40775e2098]) successfully deleted
I20240503 01:59:43.539389 6495 raft_consensus.cc:2932] T 5b599bd1fea6409fbe485c7a35c0891c P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7a1404fe1c1b4b388b821c8261b53104 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } }
I20240503 01:59:43.539696 6363 raft_consensus.cc:2932] T 5b599bd1fea6409fbe485c7a35c0891c P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7a1404fe1c1b4b388b821c8261b53104 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } }
I20240503 01:59:43.541338 6221 catalog_manager.cc:5484] T 5b599bd1fea6409fbe485c7a35c0891c P 79163c69f6f04a9abd0208d74e1c61be reported cstate change: config changed from index 4 to 5, 7a1404fe1c1b4b388b821c8261b53104 (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "79163c69f6f04a9abd0208d74e1c61be" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:43.542934 8485 raft_consensus.cc:2932] T 5b599bd1fea6409fbe485c7a35c0891c P 7a1404fe1c1b4b388b821c8261b53104 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7a1404fe1c1b4b388b821c8261b53104 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } }
I20240503 01:59:43.544369 6625 consensus_queue.cc:237] T 5b599bd1fea6409fbe485c7a35c0891c P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } }
I20240503 01:59:43.546229 6495 raft_consensus.cc:1261] T 5b599bd1fea6409fbe485c7a35c0891c P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:43.546486 8485 raft_consensus.cc:1261] T 5b599bd1fea6409fbe485c7a35c0891c P 7a1404fe1c1b4b388b821c8261b53104 [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:43.546577 11098 consensus_queue.cc:1035] T 5b599bd1fea6409fbe485c7a35c0891c P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:43.546866 7647 consensus_queue.cc:1035] T 5b599bd1fea6409fbe485c7a35c0891c P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:43.548554 11098 raft_consensus.cc:2932] T 5b599bd1fea6409fbe485c7a35c0891c P 79163c69f6f04a9abd0208d74e1c61be [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4127249ba42644388d225d614c935a25 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } }
I20240503 01:59:43.549012 6495 raft_consensus.cc:2932] T 5b599bd1fea6409fbe485c7a35c0891c P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4127249ba42644388d225d614c935a25 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } }
I20240503 01:59:43.550491 6208 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5b599bd1fea6409fbe485c7a35c0891c with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:43.550956 8485 raft_consensus.cc:2932] T 5b599bd1fea6409fbe485c7a35c0891c P 7a1404fe1c1b4b388b821c8261b53104 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4127249ba42644388d225d614c935a25 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } }
I20240503 01:59:43.551807 6214 catalog_manager.cc:5484] T 5b599bd1fea6409fbe485c7a35c0891c P 79163c69f6f04a9abd0208d74e1c61be reported cstate change: config changed from index 5 to 6, VOTER 4127249ba42644388d225d614c935a25 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "79163c69f6f04a9abd0208d74e1c61be" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:43.554316 6344 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b599bd1fea6409fbe485c7a35c0891c with delete_type TABLET_DATA_TOMBSTONED (TS 4127249ba42644388d225d614c935a25 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:57398
I20240503 01:59:43.554978 11097 tablet_replica.cc:330] T 5b599bd1fea6409fbe485c7a35c0891c P 4127249ba42644388d225d614c935a25: stopping tablet replica
I20240503 01:59:43.555939 11097 raft_consensus.cc:2229] T 5b599bd1fea6409fbe485c7a35c0891c P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:43.556149 11097 raft_consensus.cc:2258] T 5b599bd1fea6409fbe485c7a35c0891c P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:43.557076 11097 ts_tablet_manager.cc:1875] T 5b599bd1fea6409fbe485c7a35c0891c P 4127249ba42644388d225d614c935a25: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:43.558755 11097 ts_tablet_manager.cc:1888] T 5b599bd1fea6409fbe485c7a35c0891c P 4127249ba42644388d225d614c935a25: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 01:59:43.558851 11097 log.cc:1196] T 5b599bd1fea6409fbe485c7a35c0891c P 4127249ba42644388d225d614c935a25: Deleting WAL directory at /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/5b599bd1fea6409fbe485c7a35c0891c
I20240503 01:59:43.559311 6208 catalog_manager.cc:4830] TS 4127249ba42644388d225d614c935a25 (127.0.99.129:35613): tablet 5b599bd1fea6409fbe485c7a35c0891c (table rebalance_test_table_4 [id=d52e5f2a44ec4f12b7ef6d85bd14d4ac]) successfully deleted
I20240503 01:59:43.762148 11090 raft_consensus.cc:1050] T 82cfe99cd6fe4a82a7e35298ec634b8b P 4127249ba42644388d225d614c935a25: attempting to promote NON_VOTER 9264f24d028347d98f7e4e2d1aedeb79 to VOTER
I20240503 01:59:43.762485 11090 consensus_queue.cc:237] T 82cfe99cd6fe4a82a7e35298ec634b8b P 4127249ba42644388d225d614c935a25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } }
I20240503 01:59:43.763803 6495 raft_consensus.cc:1261] T 82cfe99cd6fe4a82a7e35298ec634b8b P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Refusing update from remote peer 4127249ba42644388d225d614c935a25: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:43.763927 6757 raft_consensus.cc:1261] T 82cfe99cd6fe4a82a7e35298ec634b8b P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Refusing update from remote peer 4127249ba42644388d225d614c935a25: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:43.764178 8080 raft_consensus.cc:1261] T 82cfe99cd6fe4a82a7e35298ec634b8b P 9264f24d028347d98f7e4e2d1aedeb79 [term 1 LEARNER]: Refusing update from remote peer 4127249ba42644388d225d614c935a25: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:43.764309 11087 consensus_queue.cc:1035] T 82cfe99cd6fe4a82a7e35298ec634b8b P 4127249ba42644388d225d614c935a25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:43.764469 10120 consensus_queue.cc:1035] T 82cfe99cd6fe4a82a7e35298ec634b8b P 4127249ba42644388d225d614c935a25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:43.764578 11090 consensus_queue.cc:1035] T 82cfe99cd6fe4a82a7e35298ec634b8b P 4127249ba42644388d225d614c935a25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:43.768095 11087 raft_consensus.cc:2932] T 82cfe99cd6fe4a82a7e35298ec634b8b P 4127249ba42644388d225d614c935a25 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9264f24d028347d98f7e4e2d1aedeb79 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } }
I20240503 01:59:43.768658 6757 raft_consensus.cc:2932] T 82cfe99cd6fe4a82a7e35298ec634b8b P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9264f24d028347d98f7e4e2d1aedeb79 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } }
I20240503 01:59:43.771106 6221 catalog_manager.cc:5484] T 82cfe99cd6fe4a82a7e35298ec634b8b P 4127249ba42644388d225d614c935a25 reported cstate change: config changed from index 4 to 5, 9264f24d028347d98f7e4e2d1aedeb79 (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4127249ba42644388d225d614c935a25" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:43.771116 6495 raft_consensus.cc:2932] T 82cfe99cd6fe4a82a7e35298ec634b8b P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9264f24d028347d98f7e4e2d1aedeb79 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } }
I20240503 01:59:43.772624 8080 raft_consensus.cc:2932] T 82cfe99cd6fe4a82a7e35298ec634b8b P 9264f24d028347d98f7e4e2d1aedeb79 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9264f24d028347d98f7e4e2d1aedeb79 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } }
I20240503 01:59:43.777197 6363 consensus_queue.cc:237] T 82cfe99cd6fe4a82a7e35298ec634b8b P 4127249ba42644388d225d614c935a25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } }
I20240503 01:59:43.778156 6495 raft_consensus.cc:1261] T 82cfe99cd6fe4a82a7e35298ec634b8b P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Refusing update from remote peer 4127249ba42644388d225d614c935a25: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:43.778173 8080 raft_consensus.cc:1261] T 82cfe99cd6fe4a82a7e35298ec634b8b P 9264f24d028347d98f7e4e2d1aedeb79 [term 1 FOLLOWER]: Refusing update from remote peer 4127249ba42644388d225d614c935a25: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:43.778489 10937 consensus_queue.cc:1035] T 82cfe99cd6fe4a82a7e35298ec634b8b P 4127249ba42644388d225d614c935a25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:43.778621 11090 consensus_queue.cc:1035] T 82cfe99cd6fe4a82a7e35298ec634b8b P 4127249ba42644388d225d614c935a25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:43.780179 11087 raft_consensus.cc:2932] T 82cfe99cd6fe4a82a7e35298ec634b8b P 4127249ba42644388d225d614c935a25 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b6b1d5677b8d47788b7968f64b3dac3c (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } }
I20240503 01:59:43.780779 8080 raft_consensus.cc:2932] T 82cfe99cd6fe4a82a7e35298ec634b8b P 9264f24d028347d98f7e4e2d1aedeb79 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b6b1d5677b8d47788b7968f64b3dac3c (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } }
I20240503 01:59:43.780864 6495 raft_consensus.cc:2932] T 82cfe99cd6fe4a82a7e35298ec634b8b P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b6b1d5677b8d47788b7968f64b3dac3c (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } }
I20240503 01:59:43.783043 6208 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 82cfe99cd6fe4a82a7e35298ec634b8b with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:43.783401 6221 catalog_manager.cc:5484] T 82cfe99cd6fe4a82a7e35298ec634b8b P 0c4821d9898d42799c5789d4d2da3f63 reported cstate change: config changed from index 5 to 6, VOTER b6b1d5677b8d47788b7968f64b3dac3c (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "4127249ba42644388d225d614c935a25" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } }
I20240503 01:59:43.785795 6738 tablet_service.cc:1514] Processing DeleteTablet for tablet 82cfe99cd6fe4a82a7e35298ec634b8b with delete_type TABLET_DATA_TOMBSTONED (TS b6b1d5677b8d47788b7968f64b3dac3c not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:51980
I20240503 01:59:43.786342 11173 tablet_replica.cc:330] T 82cfe99cd6fe4a82a7e35298ec634b8b P b6b1d5677b8d47788b7968f64b3dac3c: stopping tablet replica
I20240503 01:59:43.786485 11173 raft_consensus.cc:2229] T 82cfe99cd6fe4a82a7e35298ec634b8b P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:43.786670 11173 raft_consensus.cc:2258] T 82cfe99cd6fe4a82a7e35298ec634b8b P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:43.787302 11173 ts_tablet_manager.cc:1875] T 82cfe99cd6fe4a82a7e35298ec634b8b P b6b1d5677b8d47788b7968f64b3dac3c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:43.789436 11173 ts_tablet_manager.cc:1888] T 82cfe99cd6fe4a82a7e35298ec634b8b P b6b1d5677b8d47788b7968f64b3dac3c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 01:59:43.789574 11173 log.cc:1196] T 82cfe99cd6fe4a82a7e35298ec634b8b P b6b1d5677b8d47788b7968f64b3dac3c: Deleting WAL directory at /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/82cfe99cd6fe4a82a7e35298ec634b8b
I20240503 01:59:43.790114 6209 catalog_manager.cc:4830] TS b6b1d5677b8d47788b7968f64b3dac3c (127.0.99.132:46067): tablet 82cfe99cd6fe4a82a7e35298ec634b8b (table rebalance_test_table_6 [id=80cb1486f22e4bee930434fdfc3463d2]) successfully deleted
I20240503 01:59:43.809854 8355 raft_consensus.cc:1203] T c3337c95b354418faf37684e7440b1c7 P de2f076398ee44008622fb5db2a75952 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 01:59:43.810673 11075 raft_consensus.cc:1050] T 897e88ac93484a11829189e027de8bae P 0c4821d9898d42799c5789d4d2da3f63: attempting to promote NON_VOTER de2f076398ee44008622fb5db2a75952 to VOTER
I20240503 01:59:43.811071 11075 consensus_queue.cc:237] T 897e88ac93484a11829189e027de8bae P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } }
I20240503 01:59:43.812479 6625 raft_consensus.cc:1261] T 897e88ac93484a11829189e027de8bae P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Refusing update from remote peer 0c4821d9898d42799c5789d4d2da3f63: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:43.812501 6757 raft_consensus.cc:1261] T 897e88ac93484a11829189e027de8bae P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Refusing update from remote peer 0c4821d9898d42799c5789d4d2da3f63: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:43.812901 11078 consensus_queue.cc:1035] T 897e88ac93484a11829189e027de8bae P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:43.812969 8355 raft_consensus.cc:1261] T 897e88ac93484a11829189e027de8bae P de2f076398ee44008622fb5db2a75952 [term 1 LEARNER]: Refusing update from remote peer 0c4821d9898d42799c5789d4d2da3f63: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:43.813091 9924 consensus_queue.cc:1035] T 897e88ac93484a11829189e027de8bae P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:43.813285 11075 consensus_queue.cc:1035] T 897e88ac93484a11829189e027de8bae P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:43.815217 9924 raft_consensus.cc:2932] T 897e88ac93484a11829189e027de8bae P 0c4821d9898d42799c5789d4d2da3f63 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, de2f076398ee44008622fb5db2a75952 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } }
I20240503 01:59:43.815620 6625 raft_consensus.cc:2932] T 897e88ac93484a11829189e027de8bae P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, de2f076398ee44008622fb5db2a75952 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } }
I20240503 01:59:43.816843 8355 raft_consensus.cc:2932] T 897e88ac93484a11829189e027de8bae P de2f076398ee44008622fb5db2a75952 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, de2f076398ee44008622fb5db2a75952 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } }
I20240503 01:59:43.817435 6757 raft_consensus.cc:2932] T 897e88ac93484a11829189e027de8bae P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, de2f076398ee44008622fb5db2a75952 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } }
I20240503 01:59:43.817741 6218 catalog_manager.cc:5484] T 897e88ac93484a11829189e027de8bae P 0c4821d9898d42799c5789d4d2da3f63 reported cstate change: config changed from index 4 to 5, de2f076398ee44008622fb5db2a75952 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0c4821d9898d42799c5789d4d2da3f63" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:43.820816 6495 consensus_queue.cc:237] T 897e88ac93484a11829189e027de8bae P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } }
I20240503 01:59:43.822204 8355 raft_consensus.cc:1261] T 897e88ac93484a11829189e027de8bae P de2f076398ee44008622fb5db2a75952 [term 1 FOLLOWER]: Refusing update from remote peer 0c4821d9898d42799c5789d4d2da3f63: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:43.822319 6757 raft_consensus.cc:1261] T 897e88ac93484a11829189e027de8bae P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Refusing update from remote peer 0c4821d9898d42799c5789d4d2da3f63: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:43.822607 11078 consensus_queue.cc:1035] T 897e88ac93484a11829189e027de8bae P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:43.823082 11078 consensus_queue.cc:1035] T 897e88ac93484a11829189e027de8bae P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:43.824084 9924 raft_consensus.cc:2932] T 897e88ac93484a11829189e027de8bae P 0c4821d9898d42799c5789d4d2da3f63 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } }
I20240503 01:59:43.824630 8355 raft_consensus.cc:2932] T 897e88ac93484a11829189e027de8bae P de2f076398ee44008622fb5db2a75952 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } }
I20240503 01:59:43.825706 6210 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 897e88ac93484a11829189e027de8bae with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:43.826406 6214 catalog_manager.cc:5484] T 897e88ac93484a11829189e027de8bae P 0c4821d9898d42799c5789d4d2da3f63 reported cstate change: config changed from index 5 to 6, VOTER 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "0c4821d9898d42799c5789d4d2da3f63" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:43.827014 6757 raft_consensus.cc:2932] T 897e88ac93484a11829189e027de8bae P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } }
I20240503 01:59:43.828685 6607 tablet_service.cc:1514] Processing DeleteTablet for tablet 897e88ac93484a11829189e027de8bae with delete_type TABLET_DATA_TOMBSTONED (TS 79163c69f6f04a9abd0208d74e1c61be not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:47560
I20240503 01:59:43.829010 11188 tablet_replica.cc:330] T 897e88ac93484a11829189e027de8bae P 79163c69f6f04a9abd0208d74e1c61be: stopping tablet replica
I20240503 01:59:43.829155 11188 raft_consensus.cc:2229] T 897e88ac93484a11829189e027de8bae P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:43.829370 11188 raft_consensus.cc:2258] T 897e88ac93484a11829189e027de8bae P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:43.829962 11188 ts_tablet_manager.cc:1875] T 897e88ac93484a11829189e027de8bae P 79163c69f6f04a9abd0208d74e1c61be: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:43.831732 11188 ts_tablet_manager.cc:1888] T 897e88ac93484a11829189e027de8bae P 79163c69f6f04a9abd0208d74e1c61be: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 01:59:43.831800 11188 log.cc:1196] T 897e88ac93484a11829189e027de8bae P 79163c69f6f04a9abd0208d74e1c61be: Deleting WAL directory at /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/897e88ac93484a11829189e027de8bae
I20240503 01:59:43.832273 6208 catalog_manager.cc:4830] TS 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131:44005): tablet 897e88ac93484a11829189e027de8bae (table rebalance_test_table_9 [id=27f301d87e884061bddb2605833c29a6]) successfully deleted
I20240503 01:59:43.876969 11098 raft_consensus.cc:1050] T 679fc86ea2ad4bd6a9160fddf4c586d7 P 79163c69f6f04a9abd0208d74e1c61be: attempting to promote NON_VOTER de2f076398ee44008622fb5db2a75952 to VOTER
I20240503 01:59:43.877424 11098 consensus_queue.cc:237] T 679fc86ea2ad4bd6a9160fddf4c586d7 P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } }
I20240503 01:59:43.878748 6757 raft_consensus.cc:1261] T 679fc86ea2ad4bd6a9160fddf4c586d7 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:43.878748 6363 raft_consensus.cc:1261] T 679fc86ea2ad4bd6a9160fddf4c586d7 P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:43.879073 8355 raft_consensus.cc:1261] T 679fc86ea2ad4bd6a9160fddf4c586d7 P de2f076398ee44008622fb5db2a75952 [term 1 LEARNER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:43.879150 11098 consensus_queue.cc:1035] T 679fc86ea2ad4bd6a9160fddf4c586d7 P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:43.879344 11190 consensus_queue.cc:1035] T 679fc86ea2ad4bd6a9160fddf4c586d7 P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:43.879508 7647 consensus_queue.cc:1035] T 679fc86ea2ad4bd6a9160fddf4c586d7 P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:43.881425 11098 raft_consensus.cc:2932] T 679fc86ea2ad4bd6a9160fddf4c586d7 P 79163c69f6f04a9abd0208d74e1c61be [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, de2f076398ee44008622fb5db2a75952 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } }
I20240503 01:59:43.881848 6363 raft_consensus.cc:2932] T 679fc86ea2ad4bd6a9160fddf4c586d7 P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, de2f076398ee44008622fb5db2a75952 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } }
I20240503 01:59:43.881842 8355 raft_consensus.cc:2932] T 679fc86ea2ad4bd6a9160fddf4c586d7 P de2f076398ee44008622fb5db2a75952 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, de2f076398ee44008622fb5db2a75952 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } }
I20240503 01:59:43.883968 6757 raft_consensus.cc:2932] T 679fc86ea2ad4bd6a9160fddf4c586d7 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, de2f076398ee44008622fb5db2a75952 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } }
I20240503 01:59:43.884581 6221 catalog_manager.cc:5484] T 679fc86ea2ad4bd6a9160fddf4c586d7 P 4127249ba42644388d225d614c935a25 reported cstate change: config changed from index 4 to 5, de2f076398ee44008622fb5db2a75952 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "79163c69f6f04a9abd0208d74e1c61be" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } }
I20240503 01:59:43.894914 9924 raft_consensus.cc:1050] T dc7d272c4d324e10959c62362e0d81f2 P 0c4821d9898d42799c5789d4d2da3f63: attempting to promote NON_VOTER 7a1404fe1c1b4b388b821c8261b53104 to VOTER
I20240503 01:59:43.895380 9924 consensus_queue.cc:237] T dc7d272c4d324e10959c62362e0d81f2 P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } }
I20240503 01:59:43.896792 6625 raft_consensus.cc:1261] T dc7d272c4d324e10959c62362e0d81f2 P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Refusing update from remote peer 0c4821d9898d42799c5789d4d2da3f63: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:43.896792 6757 raft_consensus.cc:1261] T dc7d272c4d324e10959c62362e0d81f2 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Refusing update from remote peer 0c4821d9898d42799c5789d4d2da3f63: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:43.896919 8485 raft_consensus.cc:1261] T dc7d272c4d324e10959c62362e0d81f2 P 7a1404fe1c1b4b388b821c8261b53104 [term 1 LEARNER]: Refusing update from remote peer 0c4821d9898d42799c5789d4d2da3f63: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:43.897197 11075 consensus_queue.cc:1035] T dc7d272c4d324e10959c62362e0d81f2 P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:43.897380 11078 consensus_queue.cc:1035] T dc7d272c4d324e10959c62362e0d81f2 P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:43.897542 11077 consensus_queue.cc:1035] T dc7d272c4d324e10959c62362e0d81f2 P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:43.899355 9924 raft_consensus.cc:2932] T dc7d272c4d324e10959c62362e0d81f2 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7a1404fe1c1b4b388b821c8261b53104 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } }
I20240503 01:59:43.899852 6625 raft_consensus.cc:2932] T dc7d272c4d324e10959c62362e0d81f2 P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7a1404fe1c1b4b388b821c8261b53104 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } }
I20240503 01:59:43.899919 8485 raft_consensus.cc:2932] T dc7d272c4d324e10959c62362e0d81f2 P 7a1404fe1c1b4b388b821c8261b53104 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7a1404fe1c1b4b388b821c8261b53104 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } }
I20240503 01:59:43.902189 6218 catalog_manager.cc:5484] T dc7d272c4d324e10959c62362e0d81f2 P 79163c69f6f04a9abd0208d74e1c61be reported cstate change: config changed from index 4 to 5, 7a1404fe1c1b4b388b821c8261b53104 (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0c4821d9898d42799c5789d4d2da3f63" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } }
I20240503 01:59:43.902526 6757 raft_consensus.cc:2932] T dc7d272c4d324e10959c62362e0d81f2 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7a1404fe1c1b4b388b821c8261b53104 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } }
I20240503 01:59:43.906977 6495 consensus_queue.cc:237] T dc7d272c4d324e10959c62362e0d81f2 P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } }
I20240503 01:59:43.907960 6625 raft_consensus.cc:1261] T dc7d272c4d324e10959c62362e0d81f2 P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Refusing update from remote peer 0c4821d9898d42799c5789d4d2da3f63: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:43.907960 8485 raft_consensus.cc:1261] T dc7d272c4d324e10959c62362e0d81f2 P 7a1404fe1c1b4b388b821c8261b53104 [term 1 FOLLOWER]: Refusing update from remote peer 0c4821d9898d42799c5789d4d2da3f63: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:43.908325 11075 consensus_queue.cc:1035] T dc7d272c4d324e10959c62362e0d81f2 P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:43.908507 11078 consensus_queue.cc:1035] T dc7d272c4d324e10959c62362e0d81f2 P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:43.910037 11078 raft_consensus.cc:2932] T dc7d272c4d324e10959c62362e0d81f2 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b6b1d5677b8d47788b7968f64b3dac3c (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } }
I20240503 01:59:43.910581 6625 raft_consensus.cc:2932] T dc7d272c4d324e10959c62362e0d81f2 P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b6b1d5677b8d47788b7968f64b3dac3c (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } }
I20240503 01:59:43.911496 6210 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet dc7d272c4d324e10959c62362e0d81f2 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:43.912207 8485 raft_consensus.cc:2932] T dc7d272c4d324e10959c62362e0d81f2 P 7a1404fe1c1b4b388b821c8261b53104 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b6b1d5677b8d47788b7968f64b3dac3c (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } }
I20240503 01:59:43.913231 6221 catalog_manager.cc:5484] T dc7d272c4d324e10959c62362e0d81f2 P 79163c69f6f04a9abd0208d74e1c61be reported cstate change: config changed from index 5 to 6, VOTER b6b1d5677b8d47788b7968f64b3dac3c (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "0c4821d9898d42799c5789d4d2da3f63" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } }
I20240503 01:59:43.915480 6738 tablet_service.cc:1514] Processing DeleteTablet for tablet dc7d272c4d324e10959c62362e0d81f2 with delete_type TABLET_DATA_TOMBSTONED (TS b6b1d5677b8d47788b7968f64b3dac3c not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:51980
I20240503 01:59:43.915747 11173 tablet_replica.cc:330] T dc7d272c4d324e10959c62362e0d81f2 P b6b1d5677b8d47788b7968f64b3dac3c: stopping tablet replica
I20240503 01:59:43.915926 11173 raft_consensus.cc:2229] T dc7d272c4d324e10959c62362e0d81f2 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:43.916069 11173 raft_consensus.cc:2258] T dc7d272c4d324e10959c62362e0d81f2 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:43.916882 11173 ts_tablet_manager.cc:1875] T dc7d272c4d324e10959c62362e0d81f2 P b6b1d5677b8d47788b7968f64b3dac3c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:43.918524 11173 ts_tablet_manager.cc:1888] T dc7d272c4d324e10959c62362e0d81f2 P b6b1d5677b8d47788b7968f64b3dac3c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 01:59:43.918586 11173 log.cc:1196] T dc7d272c4d324e10959c62362e0d81f2 P b6b1d5677b8d47788b7968f64b3dac3c: Deleting WAL directory at /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/dc7d272c4d324e10959c62362e0d81f2
I20240503 01:59:43.919030 6209 catalog_manager.cc:4830] TS b6b1d5677b8d47788b7968f64b3dac3c (127.0.99.132:46067): tablet dc7d272c4d324e10959c62362e0d81f2 (table rebalance_test_table_5 [id=ba15a57b1f0e43fd9bae24116dc815d0]) successfully deleted
I20240503 01:59:43.964677 10120 raft_consensus.cc:1050] T a4983426b2d843e7a9ed335c79178678 P 4127249ba42644388d225d614c935a25: attempting to promote NON_VOTER 65d4632bcdd54ba596acdbbe1b5ef8af to VOTER
I20240503 01:59:43.965143 10120 consensus_queue.cc:237] T a4983426b2d843e7a9ed335c79178678 P 4127249ba42644388d225d614c935a25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } }
I20240503 01:59:43.966522 8224 raft_consensus.cc:1261] T a4983426b2d843e7a9ed335c79178678 P 65d4632bcdd54ba596acdbbe1b5ef8af [term 1 LEARNER]: Refusing update from remote peer 4127249ba42644388d225d614c935a25: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:43.966961 10120 consensus_queue.cc:1035] T a4983426b2d843e7a9ed335c79178678 P 4127249ba42644388d225d614c935a25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:43.967268 6495 raft_consensus.cc:1261] T a4983426b2d843e7a9ed335c79178678 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Refusing update from remote peer 4127249ba42644388d225d614c935a25: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:43.968835 11090 consensus_queue.cc:1035] T a4983426b2d843e7a9ed335c79178678 P 4127249ba42644388d225d614c935a25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:43.970113 6757 raft_consensus.cc:1261] T a4983426b2d843e7a9ed335c79178678 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Refusing update from remote peer 4127249ba42644388d225d614c935a25: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:43.972265 11087 consensus_queue.cc:1035] T a4983426b2d843e7a9ed335c79178678 P 4127249ba42644388d225d614c935a25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:43.973953 10937 raft_consensus.cc:2932] T a4983426b2d843e7a9ed335c79178678 P 4127249ba42644388d225d614c935a25 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 65d4632bcdd54ba596acdbbe1b5ef8af (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } }
I20240503 01:59:43.974503 6495 raft_consensus.cc:2932] T a4983426b2d843e7a9ed335c79178678 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 65d4632bcdd54ba596acdbbe1b5ef8af (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } }
I20240503 01:59:43.976554 8224 raft_consensus.cc:2932] T a4983426b2d843e7a9ed335c79178678 P 65d4632bcdd54ba596acdbbe1b5ef8af [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 65d4632bcdd54ba596acdbbe1b5ef8af (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } }
I20240503 01:59:43.976658 6219 catalog_manager.cc:5484] T a4983426b2d843e7a9ed335c79178678 P 4127249ba42644388d225d614c935a25 reported cstate change: config changed from index 4 to 5, 65d4632bcdd54ba596acdbbe1b5ef8af (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4127249ba42644388d225d614c935a25" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:43.982132 6757 raft_consensus.cc:2932] T a4983426b2d843e7a9ed335c79178678 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 65d4632bcdd54ba596acdbbe1b5ef8af (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } }
I20240503 01:59:43.984831 6363 consensus_queue.cc:237] T a4983426b2d843e7a9ed335c79178678 P 4127249ba42644388d225d614c935a25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } }
I20240503 01:59:43.985814 6758 raft_consensus.cc:1261] T a4983426b2d843e7a9ed335c79178678 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Refusing update from remote peer 4127249ba42644388d225d614c935a25: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:43.985867 8224 raft_consensus.cc:1261] T a4983426b2d843e7a9ed335c79178678 P 65d4632bcdd54ba596acdbbe1b5ef8af [term 1 FOLLOWER]: Refusing update from remote peer 4127249ba42644388d225d614c935a25: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:43.986215 11090 consensus_queue.cc:1035] T a4983426b2d843e7a9ed335c79178678 P 4127249ba42644388d225d614c935a25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:43.986382 10120 consensus_queue.cc:1035] T a4983426b2d843e7a9ed335c79178678 P 4127249ba42644388d225d614c935a25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:43.987857 11087 raft_consensus.cc:2932] T a4983426b2d843e7a9ed335c79178678 P 4127249ba42644388d225d614c935a25 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } }
I20240503 01:59:43.988363 6758 raft_consensus.cc:2932] T a4983426b2d843e7a9ed335c79178678 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } }
I20240503 01:59:43.988852 8224 raft_consensus.cc:2932] T a4983426b2d843e7a9ed335c79178678 P 65d4632bcdd54ba596acdbbe1b5ef8af [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } }
I20240503 01:59:43.989570 6208 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a4983426b2d843e7a9ed335c79178678 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:43.991056 6221 catalog_manager.cc:5484] T a4983426b2d843e7a9ed335c79178678 P 4127249ba42644388d225d614c935a25 reported cstate change: config changed from index 5 to 6, VOTER 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "4127249ba42644388d225d614c935a25" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:43.993314 6476 tablet_service.cc:1514] Processing DeleteTablet for tablet a4983426b2d843e7a9ed335c79178678 with delete_type TABLET_DATA_TOMBSTONED (TS 0c4821d9898d42799c5789d4d2da3f63 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:39850
I20240503 01:59:43.993539 11100 tablet_replica.cc:330] T a4983426b2d843e7a9ed335c79178678 P 0c4821d9898d42799c5789d4d2da3f63: stopping tablet replica
I20240503 01:59:43.993708 11100 raft_consensus.cc:2229] T a4983426b2d843e7a9ed335c79178678 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:43.993883 11100 raft_consensus.cc:2258] T a4983426b2d843e7a9ed335c79178678 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:43.994673 11100 ts_tablet_manager.cc:1875] T a4983426b2d843e7a9ed335c79178678 P 0c4821d9898d42799c5789d4d2da3f63: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:43.996532 11100 ts_tablet_manager.cc:1888] T a4983426b2d843e7a9ed335c79178678 P 0c4821d9898d42799c5789d4d2da3f63: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 01:59:43.996603 11100 log.cc:1196] T a4983426b2d843e7a9ed335c79178678 P 0c4821d9898d42799c5789d4d2da3f63: Deleting WAL directory at /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/a4983426b2d843e7a9ed335c79178678
I20240503 01:59:43.997076 6210 catalog_manager.cc:4830] TS 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130:34189): tablet a4983426b2d843e7a9ed335c79178678 (table rebalance_test_table_8 [id=85186d17f9464a408a3790b0978eb502]) successfully deleted
I20240503 01:59:44.032222 6625 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "679fc86ea2ad4bd6a9160fddf4c586d7"
dest_uuid: "79163c69f6f04a9abd0208d74e1c61be"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:57592
I20240503 01:59:44.032372 6625 raft_consensus.cc:595] T 679fc86ea2ad4bd6a9160fddf4c586d7 P 79163c69f6f04a9abd0208d74e1c61be [term 1 LEADER]: Received request to transfer leadership
I20240503 01:59:44.054093 6625 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "679fc86ea2ad4bd6a9160fddf4c586d7"
dest_uuid: "79163c69f6f04a9abd0208d74e1c61be"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:57630
I20240503 01:59:44.054206 6625 raft_consensus.cc:595] T 679fc86ea2ad4bd6a9160fddf4c586d7 P 79163c69f6f04a9abd0208d74e1c61be [term 1 LEADER]: Received request to transfer leadership
I20240503 01:59:44.270677 6625 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "679fc86ea2ad4bd6a9160fddf4c586d7"
dest_uuid: "79163c69f6f04a9abd0208d74e1c61be"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:57646
I20240503 01:59:44.270823 6625 raft_consensus.cc:595] T 679fc86ea2ad4bd6a9160fddf4c586d7 P 79163c69f6f04a9abd0208d74e1c61be [term 1 LEADER]: Received request to transfer leadership
I20240503 01:59:44.276814 7647 raft_consensus.cc:1050] T c3337c95b354418faf37684e7440b1c7 P 79163c69f6f04a9abd0208d74e1c61be: attempting to promote NON_VOTER de2f076398ee44008622fb5db2a75952 to VOTER
I20240503 01:59:44.277168 7647 consensus_queue.cc:237] T c3337c95b354418faf37684e7440b1c7 P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } }
I20240503 01:59:44.278594 6758 raft_consensus.cc:1261] T c3337c95b354418faf37684e7440b1c7 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:44.278594 6495 raft_consensus.cc:1261] T c3337c95b354418faf37684e7440b1c7 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:44.278653 8355 raft_consensus.cc:1261] T c3337c95b354418faf37684e7440b1c7 P de2f076398ee44008622fb5db2a75952 [term 1 LEARNER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:44.279044 7647 consensus_queue.cc:1035] T c3337c95b354418faf37684e7440b1c7 P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:44.279228 9472 consensus_queue.cc:1035] T c3337c95b354418faf37684e7440b1c7 P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:44.279481 11098 consensus_queue.cc:1035] T c3337c95b354418faf37684e7440b1c7 P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:44.281241 11098 raft_consensus.cc:2932] T c3337c95b354418faf37684e7440b1c7 P 79163c69f6f04a9abd0208d74e1c61be [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, de2f076398ee44008622fb5db2a75952 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } }
I20240503 01:59:44.281641 6758 raft_consensus.cc:2932] T c3337c95b354418faf37684e7440b1c7 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, de2f076398ee44008622fb5db2a75952 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } }
I20240503 01:59:44.281744 8355 raft_consensus.cc:2932] T c3337c95b354418faf37684e7440b1c7 P de2f076398ee44008622fb5db2a75952 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, de2f076398ee44008622fb5db2a75952 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } }
I20240503 01:59:44.283620 6495 raft_consensus.cc:2932] T c3337c95b354418faf37684e7440b1c7 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, de2f076398ee44008622fb5db2a75952 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } }
I20240503 01:59:44.284060 6219 catalog_manager.cc:5484] T c3337c95b354418faf37684e7440b1c7 P 79163c69f6f04a9abd0208d74e1c61be reported cstate change: config changed from index 4 to 5, de2f076398ee44008622fb5db2a75952 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "79163c69f6f04a9abd0208d74e1c61be" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:44.287092 6625 consensus_queue.cc:237] T c3337c95b354418faf37684e7440b1c7 P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } }
I20240503 01:59:44.288074 6758 raft_consensus.cc:1261] T c3337c95b354418faf37684e7440b1c7 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:44.288158 8355 raft_consensus.cc:1261] T c3337c95b354418faf37684e7440b1c7 P de2f076398ee44008622fb5db2a75952 [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:44.288484 9472 consensus_queue.cc:1035] T c3337c95b354418faf37684e7440b1c7 P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:44.288695 7647 consensus_queue.cc:1035] T c3337c95b354418faf37684e7440b1c7 P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:44.290087 9472 raft_consensus.cc:2932] T c3337c95b354418faf37684e7440b1c7 P 79163c69f6f04a9abd0208d74e1c61be [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } }
I20240503 01:59:44.290550 6758 raft_consensus.cc:2932] T c3337c95b354418faf37684e7440b1c7 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } }
I20240503 01:59:44.290875 8355 raft_consensus.cc:2932] T c3337c95b354418faf37684e7440b1c7 P de2f076398ee44008622fb5db2a75952 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } }
I20240503 01:59:44.291919 6208 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c3337c95b354418faf37684e7440b1c7 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:44.292837 6214 catalog_manager.cc:5484] T c3337c95b354418faf37684e7440b1c7 P 79163c69f6f04a9abd0208d74e1c61be reported cstate change: config changed from index 5 to 6, VOTER 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "79163c69f6f04a9abd0208d74e1c61be" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:44.295215 6476 tablet_service.cc:1514] Processing DeleteTablet for tablet c3337c95b354418faf37684e7440b1c7 with delete_type TABLET_DATA_TOMBSTONED (TS 0c4821d9898d42799c5789d4d2da3f63 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:39850
I20240503 01:59:44.295430 11100 tablet_replica.cc:330] T c3337c95b354418faf37684e7440b1c7 P 0c4821d9898d42799c5789d4d2da3f63: stopping tablet replica
I20240503 01:59:44.295583 11100 raft_consensus.cc:2229] T c3337c95b354418faf37684e7440b1c7 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:44.295768 11100 raft_consensus.cc:2258] T c3337c95b354418faf37684e7440b1c7 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:44.296375 11100 ts_tablet_manager.cc:1875] T c3337c95b354418faf37684e7440b1c7 P 0c4821d9898d42799c5789d4d2da3f63: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:44.298255 11100 ts_tablet_manager.cc:1888] T c3337c95b354418faf37684e7440b1c7 P 0c4821d9898d42799c5789d4d2da3f63: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 01:59:44.298331 11100 log.cc:1196] T c3337c95b354418faf37684e7440b1c7 P 0c4821d9898d42799c5789d4d2da3f63: Deleting WAL directory at /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/c3337c95b354418faf37684e7440b1c7
I20240503 01:59:44.299074 6210 catalog_manager.cc:4830] TS 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130:34189): tablet c3337c95b354418faf37684e7440b1c7 (table rebalance_test_table_7 [id=dbd6f15c28634e3dac01249aa7cc2e25]) successfully deleted
I20240503 01:59:44.304828 9472 raft_consensus.cc:979] T 679fc86ea2ad4bd6a9160fddf4c586d7 P 79163c69f6f04a9abd0208d74e1c61be: : Instructing follower b6b1d5677b8d47788b7968f64b3dac3c to start an election
I20240503 01:59:44.304924 9472 raft_consensus.cc:1067] T 679fc86ea2ad4bd6a9160fddf4c586d7 P 79163c69f6f04a9abd0208d74e1c61be [term 1 LEADER]: Signalling peer b6b1d5677b8d47788b7968f64b3dac3c to start an election
I20240503 01:59:44.305451 6758 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "679fc86ea2ad4bd6a9160fddf4c586d7"
dest_uuid: "b6b1d5677b8d47788b7968f64b3dac3c"
from {username='slave'} at 127.0.99.131:46507
I20240503 01:59:44.305586 6758 raft_consensus.cc:482] T 679fc86ea2ad4bd6a9160fddf4c586d7 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 01:59:44.305670 6758 raft_consensus.cc:3037] T 679fc86ea2ad4bd6a9160fddf4c586d7 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:44.306424 6758 raft_consensus.cc:504] T 679fc86ea2ad4bd6a9160fddf4c586d7 P b6b1d5677b8d47788b7968f64b3dac3c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } }
I20240503 01:59:44.306917 6758 leader_election.cc:288] T 679fc86ea2ad4bd6a9160fddf4c586d7 P b6b1d5677b8d47788b7968f64b3dac3c [CANDIDATE]: Term 2 election: Requested vote from peers 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131:44005), 4127249ba42644388d225d614c935a25 (127.0.99.129:35613), de2f076398ee44008622fb5db2a75952 (127.0.99.135:34035)
I20240503 01:59:44.307224 6625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "679fc86ea2ad4bd6a9160fddf4c586d7" candidate_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "79163c69f6f04a9abd0208d74e1c61be"
I20240503 01:59:44.307332 6625 raft_consensus.cc:3032] T 679fc86ea2ad4bd6a9160fddf4c586d7 P 79163c69f6f04a9abd0208d74e1c61be [term 1 LEADER]: Stepping down as leader of term 1
I20240503 01:59:44.307374 6625 raft_consensus.cc:729] T 679fc86ea2ad4bd6a9160fddf4c586d7 P 79163c69f6f04a9abd0208d74e1c61be [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 79163c69f6f04a9abd0208d74e1c61be, State: Running, Role: LEADER
I20240503 01:59:44.307350 8355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "679fc86ea2ad4bd6a9160fddf4c586d7" candidate_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "de2f076398ee44008622fb5db2a75952"
I20240503 01:59:44.307518 8355 raft_consensus.cc:3037] T 679fc86ea2ad4bd6a9160fddf4c586d7 P de2f076398ee44008622fb5db2a75952 [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:44.307530 6625 consensus_queue.cc:260] T 679fc86ea2ad4bd6a9160fddf4c586d7 P 79163c69f6f04a9abd0208d74e1c61be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } }
I20240503 01:59:44.307767 6625 raft_consensus.cc:3037] T 679fc86ea2ad4bd6a9160fddf4c586d7 P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:44.308633 8355 raft_consensus.cc:2454] T 679fc86ea2ad4bd6a9160fddf4c586d7 P de2f076398ee44008622fb5db2a75952 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6b1d5677b8d47788b7968f64b3dac3c in term 2.
I20240503 01:59:44.308790 6625 raft_consensus.cc:2454] T 679fc86ea2ad4bd6a9160fddf4c586d7 P 79163c69f6f04a9abd0208d74e1c61be [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6b1d5677b8d47788b7968f64b3dac3c in term 2.
I20240503 01:59:44.309067 6692 leader_election.cc:302] T 679fc86ea2ad4bd6a9160fddf4c586d7 P b6b1d5677b8d47788b7968f64b3dac3c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 79163c69f6f04a9abd0208d74e1c61be, b6b1d5677b8d47788b7968f64b3dac3c, de2f076398ee44008622fb5db2a75952; no voters:
I20240503 01:59:44.309257 10130 raft_consensus.cc:2781] T 679fc86ea2ad4bd6a9160fddf4c586d7 P b6b1d5677b8d47788b7968f64b3dac3c [term 2 FOLLOWER]: Leader election won for term 2
I20240503 01:59:44.309442 10130 raft_consensus.cc:686] T 679fc86ea2ad4bd6a9160fddf4c586d7 P b6b1d5677b8d47788b7968f64b3dac3c [term 2 LEADER]: Becoming Leader. State: Replica: b6b1d5677b8d47788b7968f64b3dac3c, State: Running, Role: LEADER
I20240503 01:59:44.309576 10130 consensus_queue.cc:237] T 679fc86ea2ad4bd6a9160fddf4c586d7 P b6b1d5677b8d47788b7968f64b3dac3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } }
I20240503 01:59:44.311396 6221 catalog_manager.cc:5484] T 679fc86ea2ad4bd6a9160fddf4c586d7 P b6b1d5677b8d47788b7968f64b3dac3c reported cstate change: term changed from 1 to 2, leader changed from 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131) to b6b1d5677b8d47788b7968f64b3dac3c (127.0.99.132). New cstate: current_term: 2 leader_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:44.312685 6363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "679fc86ea2ad4bd6a9160fddf4c586d7" candidate_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "4127249ba42644388d225d614c935a25"
I20240503 01:59:44.312829 6363 raft_consensus.cc:3037] T 679fc86ea2ad4bd6a9160fddf4c586d7 P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:44.313757 6363 raft_consensus.cc:2454] T 679fc86ea2ad4bd6a9160fddf4c586d7 P 4127249ba42644388d225d614c935a25 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6b1d5677b8d47788b7968f64b3dac3c in term 2.
I20240503 01:59:44.835453 6363 raft_consensus.cc:1261] T 679fc86ea2ad4bd6a9160fddf4c586d7 P 4127249ba42644388d225d614c935a25 [term 2 FOLLOWER]: Refusing update from remote peer b6b1d5677b8d47788b7968f64b3dac3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 01:59:44.835840 10130 consensus_queue.cc:1035] T 679fc86ea2ad4bd6a9160fddf4c586d7 P b6b1d5677b8d47788b7968f64b3dac3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:44.849135 8355 raft_consensus.cc:1261] T 679fc86ea2ad4bd6a9160fddf4c586d7 P de2f076398ee44008622fb5db2a75952 [term 2 FOLLOWER]: Refusing update from remote peer b6b1d5677b8d47788b7968f64b3dac3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 01:59:44.849797 10130 consensus_queue.cc:1035] T 679fc86ea2ad4bd6a9160fddf4c586d7 P b6b1d5677b8d47788b7968f64b3dac3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:44.853749 6625 raft_consensus.cc:1261] T 679fc86ea2ad4bd6a9160fddf4c586d7 P 79163c69f6f04a9abd0208d74e1c61be [term 2 FOLLOWER]: Refusing update from remote peer b6b1d5677b8d47788b7968f64b3dac3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 01:59:44.854400 10130 consensus_queue.cc:1035] T 679fc86ea2ad4bd6a9160fddf4c586d7 P b6b1d5677b8d47788b7968f64b3dac3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:44.856253 6758 consensus_queue.cc:237] T 679fc86ea2ad4bd6a9160fddf4c586d7 P b6b1d5677b8d47788b7968f64b3dac3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } }
I20240503 01:59:44.857528 8355 raft_consensus.cc:1261] T 679fc86ea2ad4bd6a9160fddf4c586d7 P de2f076398ee44008622fb5db2a75952 [term 2 FOLLOWER]: Refusing update from remote peer b6b1d5677b8d47788b7968f64b3dac3c: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 01:59:44.857842 6363 raft_consensus.cc:1261] T 679fc86ea2ad4bd6a9160fddf4c586d7 P 4127249ba42644388d225d614c935a25 [term 2 FOLLOWER]: Refusing update from remote peer b6b1d5677b8d47788b7968f64b3dac3c: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 01:59:44.857965 10130 consensus_queue.cc:1035] T 679fc86ea2ad4bd6a9160fddf4c586d7 P b6b1d5677b8d47788b7968f64b3dac3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:44.858541 10130 consensus_queue.cc:1035] T 679fc86ea2ad4bd6a9160fddf4c586d7 P b6b1d5677b8d47788b7968f64b3dac3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:44.860046 11296 raft_consensus.cc:2932] T 679fc86ea2ad4bd6a9160fddf4c586d7 P b6b1d5677b8d47788b7968f64b3dac3c [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } }
I20240503 01:59:44.860586 6363 raft_consensus.cc:2932] T 679fc86ea2ad4bd6a9160fddf4c586d7 P 4127249ba42644388d225d614c935a25 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } }
I20240503 01:59:44.860571 8355 raft_consensus.cc:2932] T 679fc86ea2ad4bd6a9160fddf4c586d7 P de2f076398ee44008622fb5db2a75952 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } }
I20240503 01:59:44.861729 6209 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 679fc86ea2ad4bd6a9160fddf4c586d7 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:44.862844 6221 catalog_manager.cc:5484] T 679fc86ea2ad4bd6a9160fddf4c586d7 P b6b1d5677b8d47788b7968f64b3dac3c reported cstate change: config changed from index 5 to 7, VOTER 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131) evicted. New cstate: current_term: 2 leader_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:44.865962 6607 tablet_service.cc:1514] Processing DeleteTablet for tablet 679fc86ea2ad4bd6a9160fddf4c586d7 with delete_type TABLET_DATA_TOMBSTONED (TS 79163c69f6f04a9abd0208d74e1c61be not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:47560
I20240503 01:59:44.866261 11306 tablet_replica.cc:330] T 679fc86ea2ad4bd6a9160fddf4c586d7 P 79163c69f6f04a9abd0208d74e1c61be: stopping tablet replica
I20240503 01:59:44.866393 11306 raft_consensus.cc:2229] T 679fc86ea2ad4bd6a9160fddf4c586d7 P 79163c69f6f04a9abd0208d74e1c61be [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:44.866490 11306 raft_consensus.cc:2258] T 679fc86ea2ad4bd6a9160fddf4c586d7 P 79163c69f6f04a9abd0208d74e1c61be [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:44.867102 11306 ts_tablet_manager.cc:1875] T 679fc86ea2ad4bd6a9160fddf4c586d7 P 79163c69f6f04a9abd0208d74e1c61be: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:44.869181 11306 ts_tablet_manager.cc:1888] T 679fc86ea2ad4bd6a9160fddf4c586d7 P 79163c69f6f04a9abd0208d74e1c61be: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 01:59:44.869292 11306 log.cc:1196] T 679fc86ea2ad4bd6a9160fddf4c586d7 P 79163c69f6f04a9abd0208d74e1c61be: Deleting WAL directory at /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/679fc86ea2ad4bd6a9160fddf4c586d7
I20240503 01:59:44.869788 6208 catalog_manager.cc:4830] TS 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131:44005): tablet 679fc86ea2ad4bd6a9160fddf4c586d7 (table rebalance_test_table_4 [id=d52e5f2a44ec4f12b7ef6d85bd14d4ac]) successfully deleted
I20240503 01:59:44.978221 6344 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 01:59:44.980844 9390 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:44.981685 8204 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:44.983352 6607 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 01:59:44.987893 6476 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 01:59:44.994415 6738 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 01:59:45.001232 8073 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:45.002511 8466 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:45.015046 8335 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:45.062532 6625 consensus_queue.cc:237] T a9a74e04ce3745b2b479e96a3f3dc86e P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: true } }
I20240503 01:59:45.064224 8486 raft_consensus.cc:1261] T a9a74e04ce3745b2b479e96a3f3dc86e P 7a1404fe1c1b4b388b821c8261b53104 [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:45.064220 6363 raft_consensus.cc:1261] T a9a74e04ce3745b2b479e96a3f3dc86e P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:45.064723 9472 consensus_queue.cc:1035] T a9a74e04ce3745b2b479e96a3f3dc86e P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:45.064996 11365 consensus_queue.cc:1035] T a9a74e04ce3745b2b479e96a3f3dc86e P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:45.066993 9472 raft_consensus.cc:2932] T a9a74e04ce3745b2b479e96a3f3dc86e P 79163c69f6f04a9abd0208d74e1c61be [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2d0fd64aaa6548e09e1ebde581c54658 (127.0.99.137) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: true } } }
I20240503 01:59:45.067409 6363 raft_consensus.cc:2932] T a9a74e04ce3745b2b479e96a3f3dc86e P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2d0fd64aaa6548e09e1ebde581c54658 (127.0.99.137) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: true } } }
W20240503 01:59:45.068316 6562 consensus_peers.cc:485] T a9a74e04ce3745b2b479e96a3f3dc86e P 79163c69f6f04a9abd0208d74e1c61be -> Peer 2d0fd64aaa6548e09e1ebde581c54658 (127.0.99.137:40613): Couldn't send request to peer 2d0fd64aaa6548e09e1ebde581c54658. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a9a74e04ce3745b2b479e96a3f3dc86e. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:45.070027 6221 catalog_manager.cc:5484] T a9a74e04ce3745b2b479e96a3f3dc86e P 4127249ba42644388d225d614c935a25 reported cstate change: config changed from index 6 to 7, NON_VOTER 2d0fd64aaa6548e09e1ebde581c54658 (127.0.99.137) added. New cstate: current_term: 1 leader_uuid: "79163c69f6f04a9abd0208d74e1c61be" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: true } } }
I20240503 01:59:45.070845 8486 raft_consensus.cc:2932] T a9a74e04ce3745b2b479e96a3f3dc86e P 7a1404fe1c1b4b388b821c8261b53104 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2d0fd64aaa6548e09e1ebde581c54658 (127.0.99.137) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: true } } }
I20240503 01:59:45.078116 6495 consensus_queue.cc:237] T ff8f06c5ad8442ee942bf900212fe858 P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: true } }
I20240503 01:59:45.079468 6625 raft_consensus.cc:1261] T ff8f06c5ad8442ee942bf900212fe858 P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Refusing update from remote peer 0c4821d9898d42799c5789d4d2da3f63: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:45.079568 8224 raft_consensus.cc:1261] T ff8f06c5ad8442ee942bf900212fe858 P 65d4632bcdd54ba596acdbbe1b5ef8af [term 1 FOLLOWER]: Refusing update from remote peer 0c4821d9898d42799c5789d4d2da3f63: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 01:59:45.079732 6429 consensus_peers.cc:485] T ff8f06c5ad8442ee942bf900212fe858 P 0c4821d9898d42799c5789d4d2da3f63 -> Peer 9264f24d028347d98f7e4e2d1aedeb79 (127.0.99.133:41713): Couldn't send request to peer 9264f24d028347d98f7e4e2d1aedeb79. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ff8f06c5ad8442ee942bf900212fe858. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:45.079890 11380 consensus_queue.cc:1035] T ff8f06c5ad8442ee942bf900212fe858 P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:45.080078 11075 consensus_queue.cc:1035] T ff8f06c5ad8442ee942bf900212fe858 P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:45.081729 11380 raft_consensus.cc:2932] T ff8f06c5ad8442ee942bf900212fe858 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9264f24d028347d98f7e4e2d1aedeb79 (127.0.99.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: true } } }
I20240503 01:59:45.082249 6625 raft_consensus.cc:2932] T ff8f06c5ad8442ee942bf900212fe858 P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9264f24d028347d98f7e4e2d1aedeb79 (127.0.99.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: true } } }
I20240503 01:59:45.082747 8224 raft_consensus.cc:2932] T ff8f06c5ad8442ee942bf900212fe858 P 65d4632bcdd54ba596acdbbe1b5ef8af [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9264f24d028347d98f7e4e2d1aedeb79 (127.0.99.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: true } } }
I20240503 01:59:45.084461 6219 catalog_manager.cc:5484] T ff8f06c5ad8442ee942bf900212fe858 P 0c4821d9898d42799c5789d4d2da3f63 reported cstate change: config changed from index 6 to 7, NON_VOTER 9264f24d028347d98f7e4e2d1aedeb79 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "0c4821d9898d42799c5789d4d2da3f63" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:45.092470 6625 consensus_queue.cc:237] T 5b599bd1fea6409fbe485c7a35c0891c P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: true } }
I20240503 01:59:45.093786 6495 raft_consensus.cc:1261] T 5b599bd1fea6409fbe485c7a35c0891c P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:45.094172 11365 consensus_queue.cc:1035] T 5b599bd1fea6409fbe485c7a35c0891c P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:45.094349 8486 raft_consensus.cc:1261] T 5b599bd1fea6409fbe485c7a35c0891c P 7a1404fe1c1b4b388b821c8261b53104 [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:45.094688 11365 consensus_queue.cc:1035] T 5b599bd1fea6409fbe485c7a35c0891c P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:45.096262 11366 raft_consensus.cc:2932] T 5b599bd1fea6409fbe485c7a35c0891c P 79163c69f6f04a9abd0208d74e1c61be [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 65d4632bcdd54ba596acdbbe1b5ef8af (127.0.99.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: true } } }
W20240503 01:59:45.097280 6560 consensus_peers.cc:485] T 5b599bd1fea6409fbe485c7a35c0891c P 79163c69f6f04a9abd0208d74e1c61be -> Peer 65d4632bcdd54ba596acdbbe1b5ef8af (127.0.99.134:35795): Couldn't send request to peer 65d4632bcdd54ba596acdbbe1b5ef8af. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5b599bd1fea6409fbe485c7a35c0891c. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:45.098078 8486 raft_consensus.cc:2932] T 5b599bd1fea6409fbe485c7a35c0891c P 7a1404fe1c1b4b388b821c8261b53104 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 65d4632bcdd54ba596acdbbe1b5ef8af (127.0.99.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: true } } }
I20240503 01:59:45.098084 6495 raft_consensus.cc:2932] T 5b599bd1fea6409fbe485c7a35c0891c P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 65d4632bcdd54ba596acdbbe1b5ef8af (127.0.99.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: true } } }
I20240503 01:59:45.098837 6222 catalog_manager.cc:5484] T 5b599bd1fea6409fbe485c7a35c0891c P 79163c69f6f04a9abd0208d74e1c61be reported cstate change: config changed from index 6 to 7, NON_VOTER 65d4632bcdd54ba596acdbbe1b5ef8af (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "79163c69f6f04a9abd0208d74e1c61be" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:45.107975 6495 consensus_queue.cc:237] T 897e88ac93484a11829189e027de8bae P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: true } }
I20240503 01:59:45.109299 8355 raft_consensus.cc:1261] T 897e88ac93484a11829189e027de8bae P de2f076398ee44008622fb5db2a75952 [term 1 FOLLOWER]: Refusing update from remote peer 0c4821d9898d42799c5789d4d2da3f63: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:45.109299 6758 raft_consensus.cc:1261] T 897e88ac93484a11829189e027de8bae P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Refusing update from remote peer 0c4821d9898d42799c5789d4d2da3f63: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 01:59:45.109519 6430 consensus_peers.cc:485] T 897e88ac93484a11829189e027de8bae P 0c4821d9898d42799c5789d4d2da3f63 -> Peer 7a1404fe1c1b4b388b821c8261b53104 (127.0.99.136:46759): Couldn't send request to peer 7a1404fe1c1b4b388b821c8261b53104. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 897e88ac93484a11829189e027de8bae. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:45.109714 11380 consensus_queue.cc:1035] T 897e88ac93484a11829189e027de8bae P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:45.109882 11379 consensus_queue.cc:1035] T 897e88ac93484a11829189e027de8bae P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:45.111450 11380 raft_consensus.cc:2932] T 897e88ac93484a11829189e027de8bae P 0c4821d9898d42799c5789d4d2da3f63 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 7a1404fe1c1b4b388b821c8261b53104 (127.0.99.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: true } } }
I20240503 01:59:45.111965 6758 raft_consensus.cc:2932] T 897e88ac93484a11829189e027de8bae P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 7a1404fe1c1b4b388b821c8261b53104 (127.0.99.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: true } } }
I20240503 01:59:45.112442 8355 raft_consensus.cc:2932] T 897e88ac93484a11829189e027de8bae P de2f076398ee44008622fb5db2a75952 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 7a1404fe1c1b4b388b821c8261b53104 (127.0.99.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: true } } }
I20240503 01:59:45.114972 6219 catalog_manager.cc:5484] T 897e88ac93484a11829189e027de8bae P b6b1d5677b8d47788b7968f64b3dac3c reported cstate change: config changed from index 6 to 7, NON_VOTER 7a1404fe1c1b4b388b821c8261b53104 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "0c4821d9898d42799c5789d4d2da3f63" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: true } } }
I20240503 01:59:45.122360 6495 consensus_queue.cc:237] T 120d5dd4f26c42399702141094ce7982 P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: true } }
I20240503 01:59:45.124042 6363 raft_consensus.cc:1261] T 120d5dd4f26c42399702141094ce7982 P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Refusing update from remote peer 0c4821d9898d42799c5789d4d2da3f63: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:45.124358 8486 raft_consensus.cc:1261] T 120d5dd4f26c42399702141094ce7982 P 7a1404fe1c1b4b388b821c8261b53104 [term 1 FOLLOWER]: Refusing update from remote peer 0c4821d9898d42799c5789d4d2da3f63: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:45.124425 11379 consensus_queue.cc:1035] T 120d5dd4f26c42399702141094ce7982 P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:45.124737 11075 consensus_queue.cc:1035] T 120d5dd4f26c42399702141094ce7982 P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:45.126295 11379 raft_consensus.cc:2932] T 120d5dd4f26c42399702141094ce7982 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2d0fd64aaa6548e09e1ebde581c54658 (127.0.99.137) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: true } } }
I20240503 01:59:45.127053 6363 raft_consensus.cc:2932] T 120d5dd4f26c42399702141094ce7982 P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2d0fd64aaa6548e09e1ebde581c54658 (127.0.99.137) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: true } } }
W20240503 01:59:45.127627 6431 consensus_peers.cc:485] T 120d5dd4f26c42399702141094ce7982 P 0c4821d9898d42799c5789d4d2da3f63 -> Peer 2d0fd64aaa6548e09e1ebde581c54658 (127.0.99.137:40613): Couldn't send request to peer 2d0fd64aaa6548e09e1ebde581c54658. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 120d5dd4f26c42399702141094ce7982. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:45.127902 8486 raft_consensus.cc:2932] T 120d5dd4f26c42399702141094ce7982 P 7a1404fe1c1b4b388b821c8261b53104 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2d0fd64aaa6548e09e1ebde581c54658 (127.0.99.137) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: true } } }
I20240503 01:59:45.129637 6220 catalog_manager.cc:5484] T 120d5dd4f26c42399702141094ce7982 P 4127249ba42644388d225d614c935a25 reported cstate change: config changed from index 6 to 7, NON_VOTER 2d0fd64aaa6548e09e1ebde581c54658 (127.0.99.137) added. New cstate: current_term: 1 leader_uuid: "0c4821d9898d42799c5789d4d2da3f63" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: true } } }
I20240503 01:59:45.137544 6625 consensus_queue.cc:237] T 21d27cca81e8455e9934e1cc604fc2b2 P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: true } }
I20240503 01:59:45.138774 8486 raft_consensus.cc:1261] T 21d27cca81e8455e9934e1cc604fc2b2 P 7a1404fe1c1b4b388b821c8261b53104 [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:45.138840 6363 raft_consensus.cc:1261] T 21d27cca81e8455e9934e1cc604fc2b2 P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 01:59:45.139058 6560 consensus_peers.cc:485] T 21d27cca81e8455e9934e1cc604fc2b2 P 79163c69f6f04a9abd0208d74e1c61be -> Peer 9264f24d028347d98f7e4e2d1aedeb79 (127.0.99.133:41713): Couldn't send request to peer 9264f24d028347d98f7e4e2d1aedeb79. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 21d27cca81e8455e9934e1cc604fc2b2. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:45.139243 11366 consensus_queue.cc:1035] T 21d27cca81e8455e9934e1cc604fc2b2 P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:45.139422 11365 consensus_queue.cc:1035] T 21d27cca81e8455e9934e1cc604fc2b2 P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:45.141085 11366 raft_consensus.cc:2932] T 21d27cca81e8455e9934e1cc604fc2b2 P 79163c69f6f04a9abd0208d74e1c61be [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9264f24d028347d98f7e4e2d1aedeb79 (127.0.99.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: true } } }
I20240503 01:59:45.141913 8486 raft_consensus.cc:2932] T 21d27cca81e8455e9934e1cc604fc2b2 P 7a1404fe1c1b4b388b821c8261b53104 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9264f24d028347d98f7e4e2d1aedeb79 (127.0.99.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: true } } }
I20240503 01:59:45.144130 6221 catalog_manager.cc:5484] T 21d27cca81e8455e9934e1cc604fc2b2 P 79163c69f6f04a9abd0208d74e1c61be reported cstate change: config changed from index 6 to 7, NON_VOTER 9264f24d028347d98f7e4e2d1aedeb79 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "79163c69f6f04a9abd0208d74e1c61be" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:45.145304 6363 raft_consensus.cc:2932] T 21d27cca81e8455e9934e1cc604fc2b2 P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9264f24d028347d98f7e4e2d1aedeb79 (127.0.99.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: true } } }
I20240503 01:59:45.154026 6363 consensus_queue.cc:237] T 82cfe99cd6fe4a82a7e35298ec634b8b P 4127249ba42644388d225d614c935a25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: true } }
I20240503 01:59:45.155311 8080 raft_consensus.cc:1261] T 82cfe99cd6fe4a82a7e35298ec634b8b P 9264f24d028347d98f7e4e2d1aedeb79 [term 1 FOLLOWER]: Refusing update from remote peer 4127249ba42644388d225d614c935a25: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:45.155607 6495 raft_consensus.cc:1261] T 82cfe99cd6fe4a82a7e35298ec634b8b P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Refusing update from remote peer 4127249ba42644388d225d614c935a25: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:45.155714 10937 consensus_queue.cc:1035] T 82cfe99cd6fe4a82a7e35298ec634b8b P 4127249ba42644388d225d614c935a25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240503 01:59:45.155993 6298 consensus_peers.cc:485] T 82cfe99cd6fe4a82a7e35298ec634b8b P 4127249ba42644388d225d614c935a25 -> Peer 65d4632bcdd54ba596acdbbe1b5ef8af (127.0.99.134:35795): Couldn't send request to peer 65d4632bcdd54ba596acdbbe1b5ef8af. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 82cfe99cd6fe4a82a7e35298ec634b8b. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:45.156023 10937 consensus_queue.cc:1035] T 82cfe99cd6fe4a82a7e35298ec634b8b P 4127249ba42644388d225d614c935a25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:45.157557 11358 raft_consensus.cc:2932] T 82cfe99cd6fe4a82a7e35298ec634b8b P 4127249ba42644388d225d614c935a25 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 65d4632bcdd54ba596acdbbe1b5ef8af (127.0.99.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: true } } }
I20240503 01:59:45.157961 8080 raft_consensus.cc:2932] T 82cfe99cd6fe4a82a7e35298ec634b8b P 9264f24d028347d98f7e4e2d1aedeb79 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 65d4632bcdd54ba596acdbbe1b5ef8af (127.0.99.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: true } } }
I20240503 01:59:45.158134 6495 raft_consensus.cc:2932] T 82cfe99cd6fe4a82a7e35298ec634b8b P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 65d4632bcdd54ba596acdbbe1b5ef8af (127.0.99.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: true } } }
I20240503 01:59:45.160631 6220 catalog_manager.cc:5484] T 82cfe99cd6fe4a82a7e35298ec634b8b P 0c4821d9898d42799c5789d4d2da3f63 reported cstate change: config changed from index 6 to 7, NON_VOTER 65d4632bcdd54ba596acdbbe1b5ef8af (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "4127249ba42644388d225d614c935a25" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: true } } }
I20240503 01:59:45.168303 6625 consensus_queue.cc:237] T a5926dae08184952bc693342c2c61ebf P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: true } }
I20240503 01:59:45.169857 8355 raft_consensus.cc:1261] T a5926dae08184952bc693342c2c61ebf P de2f076398ee44008622fb5db2a75952 [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:45.169857 6758 raft_consensus.cc:1261] T a5926dae08184952bc693342c2c61ebf P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 01:59:45.169927 6562 consensus_peers.cc:485] T a5926dae08184952bc693342c2c61ebf P 79163c69f6f04a9abd0208d74e1c61be -> Peer 2d0fd64aaa6548e09e1ebde581c54658 (127.0.99.137:40613): Couldn't send request to peer 2d0fd64aaa6548e09e1ebde581c54658. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a5926dae08184952bc693342c2c61ebf. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:45.170238 11370 consensus_queue.cc:1035] T a5926dae08184952bc693342c2c61ebf P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:45.170433 11366 consensus_queue.cc:1035] T a5926dae08184952bc693342c2c61ebf P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:45.172435 11370 raft_consensus.cc:2932] T a5926dae08184952bc693342c2c61ebf P 79163c69f6f04a9abd0208d74e1c61be [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2d0fd64aaa6548e09e1ebde581c54658 (127.0.99.137) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: true } } }
I20240503 01:59:45.172971 6758 raft_consensus.cc:2932] T a5926dae08184952bc693342c2c61ebf P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2d0fd64aaa6548e09e1ebde581c54658 (127.0.99.137) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: true } } }
I20240503 01:59:45.175766 8355 raft_consensus.cc:2932] T a5926dae08184952bc693342c2c61ebf P de2f076398ee44008622fb5db2a75952 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2d0fd64aaa6548e09e1ebde581c54658 (127.0.99.137) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: true } } }
I20240503 01:59:45.175807 6221 catalog_manager.cc:5484] T a5926dae08184952bc693342c2c61ebf P 79163c69f6f04a9abd0208d74e1c61be reported cstate change: config changed from index 6 to 7, NON_VOTER 2d0fd64aaa6548e09e1ebde581c54658 (127.0.99.137) added. New cstate: current_term: 1 leader_uuid: "79163c69f6f04a9abd0208d74e1c61be" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:45.182714 8224 consensus_queue.cc:237] T 4e7ceb1453804246a41bd1dda0e39d02 P 65d4632bcdd54ba596acdbbe1b5ef8af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: true } }
I20240503 01:59:45.184505 6495 raft_consensus.cc:1261] T 4e7ceb1453804246a41bd1dda0e39d02 P 0c4821d9898d42799c5789d4d2da3f63 [term 2 FOLLOWER]: Refusing update from remote peer 65d4632bcdd54ba596acdbbe1b5ef8af: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 01:59:45.184515 6625 raft_consensus.cc:1261] T 4e7ceb1453804246a41bd1dda0e39d02 P 79163c69f6f04a9abd0208d74e1c61be [term 2 FOLLOWER]: Refusing update from remote peer 65d4632bcdd54ba596acdbbe1b5ef8af: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 01:59:45.184949 11440 consensus_queue.cc:1035] T 4e7ceb1453804246a41bd1dda0e39d02 P 65d4632bcdd54ba596acdbbe1b5ef8af [LEADER]: Connected to new peer: Peer: permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:45.185380 11440 consensus_queue.cc:1035] T 4e7ceb1453804246a41bd1dda0e39d02 P 65d4632bcdd54ba596acdbbe1b5ef8af [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240503 01:59:45.187878 8158 consensus_peers.cc:485] T 4e7ceb1453804246a41bd1dda0e39d02 P 65d4632bcdd54ba596acdbbe1b5ef8af -> Peer 7a1404fe1c1b4b388b821c8261b53104 (127.0.99.136:46759): Couldn't send request to peer 7a1404fe1c1b4b388b821c8261b53104. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4e7ceb1453804246a41bd1dda0e39d02. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:45.187799 11294 raft_consensus.cc:2932] T 4e7ceb1453804246a41bd1dda0e39d02 P 65d4632bcdd54ba596acdbbe1b5ef8af [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 7a1404fe1c1b4b388b821c8261b53104 (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: true } } }
I20240503 01:59:45.188243 6495 raft_consensus.cc:2932] T 4e7ceb1453804246a41bd1dda0e39d02 P 0c4821d9898d42799c5789d4d2da3f63 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 7a1404fe1c1b4b388b821c8261b53104 (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: true } } }
I20240503 01:59:45.188340 6625 raft_consensus.cc:2932] T 4e7ceb1453804246a41bd1dda0e39d02 P 79163c69f6f04a9abd0208d74e1c61be [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 7a1404fe1c1b4b388b821c8261b53104 (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: true } } }
I20240503 01:59:45.190994 6219 catalog_manager.cc:5484] T 4e7ceb1453804246a41bd1dda0e39d02 P 0c4821d9898d42799c5789d4d2da3f63 reported cstate change: config changed from index 7 to 8, NON_VOTER 7a1404fe1c1b4b388b821c8261b53104 (127.0.99.136) added. New cstate: current_term: 2 leader_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: true } } }
I20240503 01:59:45.198498 6495 consensus_queue.cc:237] T 2ac8004211e64c3481e75377092fa93b P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: true } }
I20240503 01:59:45.199955 8080 raft_consensus.cc:1261] T 2ac8004211e64c3481e75377092fa93b P 9264f24d028347d98f7e4e2d1aedeb79 [term 1 FOLLOWER]: Refusing update from remote peer 0c4821d9898d42799c5789d4d2da3f63: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:45.200006 6363 raft_consensus.cc:1261] T 2ac8004211e64c3481e75377092fa93b P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Refusing update from remote peer 0c4821d9898d42799c5789d4d2da3f63: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 01:59:45.200212 6431 consensus_peers.cc:485] T 2ac8004211e64c3481e75377092fa93b P 0c4821d9898d42799c5789d4d2da3f63 -> Peer 2d0fd64aaa6548e09e1ebde581c54658 (127.0.99.137:40613): Couldn't send request to peer 2d0fd64aaa6548e09e1ebde581c54658. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2ac8004211e64c3481e75377092fa93b. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:45.200445 11379 consensus_queue.cc:1035] T 2ac8004211e64c3481e75377092fa93b P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:45.200621 11075 consensus_queue.cc:1035] T 2ac8004211e64c3481e75377092fa93b P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:45.202524 11380 raft_consensus.cc:2932] T 2ac8004211e64c3481e75377092fa93b P 0c4821d9898d42799c5789d4d2da3f63 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2d0fd64aaa6548e09e1ebde581c54658 (127.0.99.137) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: true } } }
I20240503 01:59:45.202950 8080 raft_consensus.cc:2932] T 2ac8004211e64c3481e75377092fa93b P 9264f24d028347d98f7e4e2d1aedeb79 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2d0fd64aaa6548e09e1ebde581c54658 (127.0.99.137) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: true } } }
I20240503 01:59:45.202989 6363 raft_consensus.cc:2932] T 2ac8004211e64c3481e75377092fa93b P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2d0fd64aaa6548e09e1ebde581c54658 (127.0.99.137) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: true } } }
I20240503 01:59:45.205742 6222 catalog_manager.cc:5484] T 2ac8004211e64c3481e75377092fa93b P 9264f24d028347d98f7e4e2d1aedeb79 reported cstate change: config changed from index 6 to 7, NON_VOTER 2d0fd64aaa6548e09e1ebde581c54658 (127.0.99.137) added. New cstate: current_term: 1 leader_uuid: "0c4821d9898d42799c5789d4d2da3f63" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: true } } }
I20240503 01:59:45.212502 6495 consensus_queue.cc:237] T ccb34a4b52244d90850ab04e21299f55 P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: true } }
I20240503 01:59:45.213802 8486 raft_consensus.cc:1261] T ccb34a4b52244d90850ab04e21299f55 P 7a1404fe1c1b4b388b821c8261b53104 [term 1 FOLLOWER]: Refusing update from remote peer 0c4821d9898d42799c5789d4d2da3f63: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:45.213820 6625 raft_consensus.cc:1261] T ccb34a4b52244d90850ab04e21299f55 P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Refusing update from remote peer 0c4821d9898d42799c5789d4d2da3f63: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 01:59:45.213898 6429 consensus_peers.cc:485] T ccb34a4b52244d90850ab04e21299f55 P 0c4821d9898d42799c5789d4d2da3f63 -> Peer 9264f24d028347d98f7e4e2d1aedeb79 (127.0.99.133:41713): Couldn't send request to peer 9264f24d028347d98f7e4e2d1aedeb79. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ccb34a4b52244d90850ab04e21299f55. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:45.214232 11380 consensus_queue.cc:1035] T ccb34a4b52244d90850ab04e21299f55 P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:45.214411 11075 consensus_queue.cc:1035] T ccb34a4b52244d90850ab04e21299f55 P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:45.216126 11379 raft_consensus.cc:2932] T ccb34a4b52244d90850ab04e21299f55 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9264f24d028347d98f7e4e2d1aedeb79 (127.0.99.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: true } } }
I20240503 01:59:45.217039 8486 raft_consensus.cc:2932] T ccb34a4b52244d90850ab04e21299f55 P 7a1404fe1c1b4b388b821c8261b53104 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9264f24d028347d98f7e4e2d1aedeb79 (127.0.99.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: true } } }
I20240503 01:59:45.217114 6625 raft_consensus.cc:2932] T ccb34a4b52244d90850ab04e21299f55 P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9264f24d028347d98f7e4e2d1aedeb79 (127.0.99.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: true } } }
I20240503 01:59:45.219220 6219 catalog_manager.cc:5484] T ccb34a4b52244d90850ab04e21299f55 P 7a1404fe1c1b4b388b821c8261b53104 reported cstate change: config changed from index 6 to 7, NON_VOTER 9264f24d028347d98f7e4e2d1aedeb79 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "0c4821d9898d42799c5789d4d2da3f63" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: true } } }
I20240503 01:59:45.227008 6495 consensus_queue.cc:237] T 1d392e08c15a4d4098a350287b3163ef P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: true } }
I20240503 01:59:45.228351 8355 raft_consensus.cc:1261] T 1d392e08c15a4d4098a350287b3163ef P de2f076398ee44008622fb5db2a75952 [term 1 FOLLOWER]: Refusing update from remote peer 0c4821d9898d42799c5789d4d2da3f63: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 01:59:45.228494 6429 consensus_peers.cc:485] T 1d392e08c15a4d4098a350287b3163ef P 0c4821d9898d42799c5789d4d2da3f63 -> Peer 65d4632bcdd54ba596acdbbe1b5ef8af (127.0.99.134:35795): Couldn't send request to peer 65d4632bcdd54ba596acdbbe1b5ef8af. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1d392e08c15a4d4098a350287b3163ef. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:45.228842 11075 consensus_queue.cc:1035] T 1d392e08c15a4d4098a350287b3163ef P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:45.228891 6758 raft_consensus.cc:1261] T 1d392e08c15a4d4098a350287b3163ef P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Refusing update from remote peer 0c4821d9898d42799c5789d4d2da3f63: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:45.229262 11075 consensus_queue.cc:1035] T 1d392e08c15a4d4098a350287b3163ef P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:45.230713 11380 raft_consensus.cc:2932] T 1d392e08c15a4d4098a350287b3163ef P 0c4821d9898d42799c5789d4d2da3f63 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 65d4632bcdd54ba596acdbbe1b5ef8af (127.0.99.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: true } } }
I20240503 01:59:45.231521 8355 raft_consensus.cc:2932] T 1d392e08c15a4d4098a350287b3163ef P de2f076398ee44008622fb5db2a75952 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 65d4632bcdd54ba596acdbbe1b5ef8af (127.0.99.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: true } } }
I20240503 01:59:45.231742 6758 raft_consensus.cc:2932] T 1d392e08c15a4d4098a350287b3163ef P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 65d4632bcdd54ba596acdbbe1b5ef8af (127.0.99.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: true } } }
I20240503 01:59:45.234243 6222 catalog_manager.cc:5484] T 1d392e08c15a4d4098a350287b3163ef P b6b1d5677b8d47788b7968f64b3dac3c reported cstate change: config changed from index 6 to 7, NON_VOTER 65d4632bcdd54ba596acdbbe1b5ef8af (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "0c4821d9898d42799c5789d4d2da3f63" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: true } } }
I20240503 01:59:45.241515 6625 consensus_queue.cc:237] T d6ee8175874c4063986033b0a80ccb82 P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: true } }
I20240503 01:59:45.242852 8486 raft_consensus.cc:1261] T d6ee8175874c4063986033b0a80ccb82 P 7a1404fe1c1b4b388b821c8261b53104 [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:45.242906 6758 raft_consensus.cc:1261] T d6ee8175874c4063986033b0a80ccb82 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 01:59:45.243155 6560 consensus_peers.cc:485] T d6ee8175874c4063986033b0a80ccb82 P 79163c69f6f04a9abd0208d74e1c61be -> Peer 65d4632bcdd54ba596acdbbe1b5ef8af (127.0.99.134:35795): Couldn't send request to peer 65d4632bcdd54ba596acdbbe1b5ef8af. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d6ee8175874c4063986033b0a80ccb82. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:45.243326 9472 consensus_queue.cc:1035] T d6ee8175874c4063986033b0a80ccb82 P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:45.243513 11365 consensus_queue.cc:1035] T d6ee8175874c4063986033b0a80ccb82 P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:45.245263 9472 raft_consensus.cc:2932] T d6ee8175874c4063986033b0a80ccb82 P 79163c69f6f04a9abd0208d74e1c61be [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 65d4632bcdd54ba596acdbbe1b5ef8af (127.0.99.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: true } } }
I20240503 01:59:45.245797 6758 raft_consensus.cc:2932] T d6ee8175874c4063986033b0a80ccb82 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 65d4632bcdd54ba596acdbbe1b5ef8af (127.0.99.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: true } } }
I20240503 01:59:45.245847 8486 raft_consensus.cc:2932] T d6ee8175874c4063986033b0a80ccb82 P 7a1404fe1c1b4b388b821c8261b53104 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 65d4632bcdd54ba596acdbbe1b5ef8af (127.0.99.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: true } } }
I20240503 01:59:45.249094 6220 catalog_manager.cc:5484] T d6ee8175874c4063986033b0a80ccb82 P 79163c69f6f04a9abd0208d74e1c61be reported cstate change: config changed from index 6 to 7, NON_VOTER 65d4632bcdd54ba596acdbbe1b5ef8af (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "79163c69f6f04a9abd0208d74e1c61be" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:45.257249 6758 consensus_queue.cc:237] T f11a8034585a4d57be0e9b1e5e8bcd85 P b6b1d5677b8d47788b7968f64b3dac3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: true } }
I20240503 01:59:45.258718 6495 raft_consensus.cc:1261] T f11a8034585a4d57be0e9b1e5e8bcd85 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Refusing update from remote peer b6b1d5677b8d47788b7968f64b3dac3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:45.259037 8080 raft_consensus.cc:1261] T f11a8034585a4d57be0e9b1e5e8bcd85 P 9264f24d028347d98f7e4e2d1aedeb79 [term 1 FOLLOWER]: Refusing update from remote peer b6b1d5677b8d47788b7968f64b3dac3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:45.259204 10130 consensus_queue.cc:1035] T f11a8034585a4d57be0e9b1e5e8bcd85 P b6b1d5677b8d47788b7968f64b3dac3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:45.259770 10130 consensus_queue.cc:1035] T f11a8034585a4d57be0e9b1e5e8bcd85 P b6b1d5677b8d47788b7968f64b3dac3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:45.261158 11296 raft_consensus.cc:2932] T f11a8034585a4d57be0e9b1e5e8bcd85 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2d0fd64aaa6548e09e1ebde581c54658 (127.0.99.137) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: true } } }
I20240503 01:59:45.261551 6495 raft_consensus.cc:2932] T f11a8034585a4d57be0e9b1e5e8bcd85 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2d0fd64aaa6548e09e1ebde581c54658 (127.0.99.137) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: true } } }
I20240503 01:59:45.262054 8080 raft_consensus.cc:2932] T f11a8034585a4d57be0e9b1e5e8bcd85 P 9264f24d028347d98f7e4e2d1aedeb79 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2d0fd64aaa6548e09e1ebde581c54658 (127.0.99.137) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: true } } }
I20240503 01:59:45.264350 6219 catalog_manager.cc:5484] T f11a8034585a4d57be0e9b1e5e8bcd85 P 0c4821d9898d42799c5789d4d2da3f63 reported cstate change: config changed from index 6 to 7, NON_VOTER 2d0fd64aaa6548e09e1ebde581c54658 (127.0.99.137) added. New cstate: current_term: 1 leader_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: true } } }
W20240503 01:59:45.264892 6693 consensus_peers.cc:485] T f11a8034585a4d57be0e9b1e5e8bcd85 P b6b1d5677b8d47788b7968f64b3dac3c -> Peer 2d0fd64aaa6548e09e1ebde581c54658 (127.0.99.137:40613): Couldn't send request to peer 2d0fd64aaa6548e09e1ebde581c54658. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f11a8034585a4d57be0e9b1e5e8bcd85. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:45.477797 11556 ts_tablet_manager.cc:902] T ff8f06c5ad8442ee942bf900212fe858 P 9264f24d028347d98f7e4e2d1aedeb79: Initiating tablet copy from peer 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130:34189)
I20240503 01:59:45.478333 11556 tablet_copy_client.cc:308] T ff8f06c5ad8442ee942bf900212fe858 P 9264f24d028347d98f7e4e2d1aedeb79: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:34189
I20240503 01:59:45.478797 6516 tablet_copy_service.cc:140] P 0c4821d9898d42799c5789d4d2da3f63: Received BeginTabletCopySession request for tablet ff8f06c5ad8442ee942bf900212fe858 from peer 9264f24d028347d98f7e4e2d1aedeb79 ({username='slave'} at 127.0.99.133:35409)
I20240503 01:59:45.478899 6516 tablet_copy_service.cc:161] P 0c4821d9898d42799c5789d4d2da3f63: Beginning new tablet copy session on tablet ff8f06c5ad8442ee942bf900212fe858 from peer 9264f24d028347d98f7e4e2d1aedeb79 at {username='slave'} at 127.0.99.133:35409: session id = 9264f24d028347d98f7e4e2d1aedeb79-ff8f06c5ad8442ee942bf900212fe858
I20240503 01:59:45.479650 6516 tablet_copy_source_session.cc:205] T ff8f06c5ad8442ee942bf900212fe858 P 0c4821d9898d42799c5789d4d2da3f63: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:45.480522 11556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff8f06c5ad8442ee942bf900212fe858. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:45.482728 11556 tablet_copy_client.cc:783] T ff8f06c5ad8442ee942bf900212fe858 P 9264f24d028347d98f7e4e2d1aedeb79: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:45.482883 11556 tablet_copy_client.cc:647] T ff8f06c5ad8442ee942bf900212fe858 P 9264f24d028347d98f7e4e2d1aedeb79: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:45.483942 11560 tablet_copy_client.cc:1002] T ff8f06c5ad8442ee942bf900212fe858 P 9264f24d028347d98f7e4e2d1aedeb79: tablet copy: Injecting latency into file download: 1500
I20240503 01:59:45.508174 11561 ts_tablet_manager.cc:902] T 120d5dd4f26c42399702141094ce7982 P 2d0fd64aaa6548e09e1ebde581c54658: Initiating tablet copy from peer 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130:34189)
I20240503 01:59:45.508978 11561 tablet_copy_client.cc:308] T 120d5dd4f26c42399702141094ce7982 P 2d0fd64aaa6548e09e1ebde581c54658: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:34189
I20240503 01:59:45.510129 11567 ts_tablet_manager.cc:902] T 897e88ac93484a11829189e027de8bae P 7a1404fe1c1b4b388b821c8261b53104: Initiating tablet copy from peer 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130:34189)
I20240503 01:59:45.510674 11567 tablet_copy_client.cc:308] T 897e88ac93484a11829189e027de8bae P 7a1404fe1c1b4b388b821c8261b53104: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:34189
I20240503 01:59:45.511134 6516 tablet_copy_service.cc:140] P 0c4821d9898d42799c5789d4d2da3f63: Received BeginTabletCopySession request for tablet 897e88ac93484a11829189e027de8bae from peer 7a1404fe1c1b4b388b821c8261b53104 ({username='slave'} at 127.0.99.136:32823)
I20240503 01:59:45.511219 6516 tablet_copy_service.cc:161] P 0c4821d9898d42799c5789d4d2da3f63: Beginning new tablet copy session on tablet 897e88ac93484a11829189e027de8bae from peer 7a1404fe1c1b4b388b821c8261b53104 at {username='slave'} at 127.0.99.136:32823: session id = 7a1404fe1c1b4b388b821c8261b53104-897e88ac93484a11829189e027de8bae
I20240503 01:59:45.511879 6516 tablet_copy_source_session.cc:205] T 897e88ac93484a11829189e027de8bae P 0c4821d9898d42799c5789d4d2da3f63: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:45.512701 11567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 897e88ac93484a11829189e027de8bae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:45.513001 6516 tablet_copy_service.cc:140] P 0c4821d9898d42799c5789d4d2da3f63: Received BeginTabletCopySession request for tablet 120d5dd4f26c42399702141094ce7982 from peer 2d0fd64aaa6548e09e1ebde581c54658 ({username='slave'} at 127.0.99.137:56177)
I20240503 01:59:45.513108 6516 tablet_copy_service.cc:161] P 0c4821d9898d42799c5789d4d2da3f63: Beginning new tablet copy session on tablet 120d5dd4f26c42399702141094ce7982 from peer 2d0fd64aaa6548e09e1ebde581c54658 at {username='slave'} at 127.0.99.137:56177: session id = 2d0fd64aaa6548e09e1ebde581c54658-120d5dd4f26c42399702141094ce7982
I20240503 01:59:45.513935 6516 tablet_copy_source_session.cc:205] T 120d5dd4f26c42399702141094ce7982 P 0c4821d9898d42799c5789d4d2da3f63: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:45.514881 11567 tablet_copy_client.cc:783] T 897e88ac93484a11829189e027de8bae P 7a1404fe1c1b4b388b821c8261b53104: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:45.515031 11567 tablet_copy_client.cc:647] T 897e88ac93484a11829189e027de8bae P 7a1404fe1c1b4b388b821c8261b53104: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:45.516116 11571 tablet_copy_client.cc:1002] T 897e88ac93484a11829189e027de8bae P 7a1404fe1c1b4b388b821c8261b53104: tablet copy: Injecting latency into file download: 1500
I20240503 01:59:45.517143 11561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 120d5dd4f26c42399702141094ce7982. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:45.521466 11561 tablet_copy_client.cc:783] T 120d5dd4f26c42399702141094ce7982 P 2d0fd64aaa6548e09e1ebde581c54658: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:45.521633 11561 tablet_copy_client.cc:647] T 120d5dd4f26c42399702141094ce7982 P 2d0fd64aaa6548e09e1ebde581c54658: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:45.522821 11565 tablet_copy_client.cc:1002] T 120d5dd4f26c42399702141094ce7982 P 2d0fd64aaa6548e09e1ebde581c54658: tablet copy: Injecting latency into file download: 1500
I20240503 01:59:45.583107 11607 ts_tablet_manager.cc:902] T a9a74e04ce3745b2b479e96a3f3dc86e P 2d0fd64aaa6548e09e1ebde581c54658: Initiating tablet copy from peer 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131:44005)
I20240503 01:59:45.583623 11607 tablet_copy_client.cc:308] T a9a74e04ce3745b2b479e96a3f3dc86e P 2d0fd64aaa6548e09e1ebde581c54658: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:44005
I20240503 01:59:45.586956 6647 tablet_copy_service.cc:140] P 79163c69f6f04a9abd0208d74e1c61be: Received BeginTabletCopySession request for tablet a9a74e04ce3745b2b479e96a3f3dc86e from peer 2d0fd64aaa6548e09e1ebde581c54658 ({username='slave'} at 127.0.99.137:43265)
I20240503 01:59:45.587055 6647 tablet_copy_service.cc:161] P 79163c69f6f04a9abd0208d74e1c61be: Beginning new tablet copy session on tablet a9a74e04ce3745b2b479e96a3f3dc86e from peer 2d0fd64aaa6548e09e1ebde581c54658 at {username='slave'} at 127.0.99.137:43265: session id = 2d0fd64aaa6548e09e1ebde581c54658-a9a74e04ce3745b2b479e96a3f3dc86e
I20240503 01:59:45.587754 6647 tablet_copy_source_session.cc:205] T a9a74e04ce3745b2b479e96a3f3dc86e P 79163c69f6f04a9abd0208d74e1c61be: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:45.588670 11607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9a74e04ce3745b2b479e96a3f3dc86e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:45.589165 11617 ts_tablet_manager.cc:902] T 21d27cca81e8455e9934e1cc604fc2b2 P 9264f24d028347d98f7e4e2d1aedeb79: Initiating tablet copy from peer 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131:44005)
I20240503 01:59:45.590044 11617 tablet_copy_client.cc:308] T 21d27cca81e8455e9934e1cc604fc2b2 P 9264f24d028347d98f7e4e2d1aedeb79: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:44005
I20240503 01:59:45.590452 6647 tablet_copy_service.cc:140] P 79163c69f6f04a9abd0208d74e1c61be: Received BeginTabletCopySession request for tablet 21d27cca81e8455e9934e1cc604fc2b2 from peer 9264f24d028347d98f7e4e2d1aedeb79 ({username='slave'} at 127.0.99.133:50661)
I20240503 01:59:45.590535 6647 tablet_copy_service.cc:161] P 79163c69f6f04a9abd0208d74e1c61be: Beginning new tablet copy session on tablet 21d27cca81e8455e9934e1cc604fc2b2 from peer 9264f24d028347d98f7e4e2d1aedeb79 at {username='slave'} at 127.0.99.133:50661: session id = 9264f24d028347d98f7e4e2d1aedeb79-21d27cca81e8455e9934e1cc604fc2b2
I20240503 01:59:45.591107 11607 tablet_copy_client.cc:783] T a9a74e04ce3745b2b479e96a3f3dc86e P 2d0fd64aaa6548e09e1ebde581c54658: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:45.591261 11607 tablet_copy_client.cc:647] T a9a74e04ce3745b2b479e96a3f3dc86e P 2d0fd64aaa6548e09e1ebde581c54658: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:45.591434 6647 tablet_copy_source_session.cc:205] T 21d27cca81e8455e9934e1cc604fc2b2 P 79163c69f6f04a9abd0208d74e1c61be: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:45.592358 11615 tablet_copy_client.cc:1002] T a9a74e04ce3745b2b479e96a3f3dc86e P 2d0fd64aaa6548e09e1ebde581c54658: tablet copy: Injecting latency into file download: 1500
I20240503 01:59:45.592463 11617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21d27cca81e8455e9934e1cc604fc2b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:45.594625 11617 tablet_copy_client.cc:783] T 21d27cca81e8455e9934e1cc604fc2b2 P 9264f24d028347d98f7e4e2d1aedeb79: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:45.594791 11617 tablet_copy_client.cc:647] T 21d27cca81e8455e9934e1cc604fc2b2 P 9264f24d028347d98f7e4e2d1aedeb79: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:45.596151 11619 tablet_copy_client.cc:1002] T 21d27cca81e8455e9934e1cc604fc2b2 P 9264f24d028347d98f7e4e2d1aedeb79: tablet copy: Injecting latency into file download: 1500
I20240503 01:59:45.624699 11652 ts_tablet_manager.cc:902] T d6ee8175874c4063986033b0a80ccb82 P 65d4632bcdd54ba596acdbbe1b5ef8af: Initiating tablet copy from peer 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131:44005)
I20240503 01:59:45.625279 11652 tablet_copy_client.cc:308] T d6ee8175874c4063986033b0a80ccb82 P 65d4632bcdd54ba596acdbbe1b5ef8af: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:44005
I20240503 01:59:45.625694 6647 tablet_copy_service.cc:140] P 79163c69f6f04a9abd0208d74e1c61be: Received BeginTabletCopySession request for tablet d6ee8175874c4063986033b0a80ccb82 from peer 65d4632bcdd54ba596acdbbe1b5ef8af ({username='slave'} at 127.0.99.134:43049)
I20240503 01:59:45.625787 6647 tablet_copy_service.cc:161] P 79163c69f6f04a9abd0208d74e1c61be: Beginning new tablet copy session on tablet d6ee8175874c4063986033b0a80ccb82 from peer 65d4632bcdd54ba596acdbbe1b5ef8af at {username='slave'} at 127.0.99.134:43049: session id = 65d4632bcdd54ba596acdbbe1b5ef8af-d6ee8175874c4063986033b0a80ccb82
I20240503 01:59:45.626645 6647 tablet_copy_source_session.cc:205] T d6ee8175874c4063986033b0a80ccb82 P 79163c69f6f04a9abd0208d74e1c61be: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:45.627534 11652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6ee8175874c4063986033b0a80ccb82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:45.629606 11652 tablet_copy_client.cc:783] T d6ee8175874c4063986033b0a80ccb82 P 65d4632bcdd54ba596acdbbe1b5ef8af: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:45.629756 11652 tablet_copy_client.cc:647] T d6ee8175874c4063986033b0a80ccb82 P 65d4632bcdd54ba596acdbbe1b5ef8af: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:45.630725 11656 tablet_copy_client.cc:1002] T d6ee8175874c4063986033b0a80ccb82 P 65d4632bcdd54ba596acdbbe1b5ef8af: tablet copy: Injecting latency into file download: 1500
I20240503 01:59:45.646361 11657 ts_tablet_manager.cc:902] T 2ac8004211e64c3481e75377092fa93b P 2d0fd64aaa6548e09e1ebde581c54658: Initiating tablet copy from peer 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130:34189)
I20240503 01:59:45.646852 11657 tablet_copy_client.cc:308] T 2ac8004211e64c3481e75377092fa93b P 2d0fd64aaa6548e09e1ebde581c54658: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:34189
I20240503 01:59:45.647373 6516 tablet_copy_service.cc:140] P 0c4821d9898d42799c5789d4d2da3f63: Received BeginTabletCopySession request for tablet 2ac8004211e64c3481e75377092fa93b from peer 2d0fd64aaa6548e09e1ebde581c54658 ({username='slave'} at 127.0.99.137:56177)
I20240503 01:59:45.647490 6516 tablet_copy_service.cc:161] P 0c4821d9898d42799c5789d4d2da3f63: Beginning new tablet copy session on tablet 2ac8004211e64c3481e75377092fa93b from peer 2d0fd64aaa6548e09e1ebde581c54658 at {username='slave'} at 127.0.99.137:56177: session id = 2d0fd64aaa6548e09e1ebde581c54658-2ac8004211e64c3481e75377092fa93b
I20240503 01:59:45.648377 6516 tablet_copy_source_session.cc:205] T 2ac8004211e64c3481e75377092fa93b P 0c4821d9898d42799c5789d4d2da3f63: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:45.649312 11657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ac8004211e64c3481e75377092fa93b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:45.651054 11657 tablet_copy_client.cc:783] T 2ac8004211e64c3481e75377092fa93b P 2d0fd64aaa6548e09e1ebde581c54658: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:45.651211 11657 tablet_copy_client.cc:647] T 2ac8004211e64c3481e75377092fa93b P 2d0fd64aaa6548e09e1ebde581c54658: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:45.652206 11661 tablet_copy_client.cc:1002] T 2ac8004211e64c3481e75377092fa93b P 2d0fd64aaa6548e09e1ebde581c54658: tablet copy: Injecting latency into file download: 1500
I20240503 01:59:45.660584 11662 ts_tablet_manager.cc:902] T 82cfe99cd6fe4a82a7e35298ec634b8b P 65d4632bcdd54ba596acdbbe1b5ef8af: Initiating tablet copy from peer 4127249ba42644388d225d614c935a25 (127.0.99.129:35613)
I20240503 01:59:45.661141 11662 tablet_copy_client.cc:308] T 82cfe99cd6fe4a82a7e35298ec634b8b P 65d4632bcdd54ba596acdbbe1b5ef8af: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:35613
I20240503 01:59:45.661577 6385 tablet_copy_service.cc:140] P 4127249ba42644388d225d614c935a25: Received BeginTabletCopySession request for tablet 82cfe99cd6fe4a82a7e35298ec634b8b from peer 65d4632bcdd54ba596acdbbe1b5ef8af ({username='slave'} at 127.0.99.134:49429)
I20240503 01:59:45.661677 6385 tablet_copy_service.cc:161] P 4127249ba42644388d225d614c935a25: Beginning new tablet copy session on tablet 82cfe99cd6fe4a82a7e35298ec634b8b from peer 65d4632bcdd54ba596acdbbe1b5ef8af at {username='slave'} at 127.0.99.134:49429: session id = 65d4632bcdd54ba596acdbbe1b5ef8af-82cfe99cd6fe4a82a7e35298ec634b8b
I20240503 01:59:45.662367 6385 tablet_copy_source_session.cc:205] T 82cfe99cd6fe4a82a7e35298ec634b8b P 4127249ba42644388d225d614c935a25: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:45.663146 11662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82cfe99cd6fe4a82a7e35298ec634b8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:45.665279 11662 tablet_copy_client.cc:783] T 82cfe99cd6fe4a82a7e35298ec634b8b P 65d4632bcdd54ba596acdbbe1b5ef8af: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:45.665417 11662 tablet_copy_client.cc:647] T 82cfe99cd6fe4a82a7e35298ec634b8b P 65d4632bcdd54ba596acdbbe1b5ef8af: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:45.666407 11666 tablet_copy_client.cc:1002] T 82cfe99cd6fe4a82a7e35298ec634b8b P 65d4632bcdd54ba596acdbbe1b5ef8af: tablet copy: Injecting latency into file download: 1500
I20240503 01:59:45.701486 11667 ts_tablet_manager.cc:902] T 5b599bd1fea6409fbe485c7a35c0891c P 65d4632bcdd54ba596acdbbe1b5ef8af: Initiating tablet copy from peer 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131:44005)
I20240503 01:59:45.702005 11667 tablet_copy_client.cc:308] T 5b599bd1fea6409fbe485c7a35c0891c P 65d4632bcdd54ba596acdbbe1b5ef8af: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:44005
I20240503 01:59:45.702491 6647 tablet_copy_service.cc:140] P 79163c69f6f04a9abd0208d74e1c61be: Received BeginTabletCopySession request for tablet 5b599bd1fea6409fbe485c7a35c0891c from peer 65d4632bcdd54ba596acdbbe1b5ef8af ({username='slave'} at 127.0.99.134:43049)
I20240503 01:59:45.702584 6647 tablet_copy_service.cc:161] P 79163c69f6f04a9abd0208d74e1c61be: Beginning new tablet copy session on tablet 5b599bd1fea6409fbe485c7a35c0891c from peer 65d4632bcdd54ba596acdbbe1b5ef8af at {username='slave'} at 127.0.99.134:43049: session id = 65d4632bcdd54ba596acdbbe1b5ef8af-5b599bd1fea6409fbe485c7a35c0891c
I20240503 01:59:45.703307 6647 tablet_copy_source_session.cc:205] T 5b599bd1fea6409fbe485c7a35c0891c P 79163c69f6f04a9abd0208d74e1c61be: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:45.704057 11667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b599bd1fea6409fbe485c7a35c0891c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:45.705834 11667 tablet_copy_client.cc:783] T 5b599bd1fea6409fbe485c7a35c0891c P 65d4632bcdd54ba596acdbbe1b5ef8af: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:45.705979 11667 tablet_copy_client.cc:647] T 5b599bd1fea6409fbe485c7a35c0891c P 65d4632bcdd54ba596acdbbe1b5ef8af: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:45.706975 11671 tablet_copy_client.cc:1002] T 5b599bd1fea6409fbe485c7a35c0891c P 65d4632bcdd54ba596acdbbe1b5ef8af: tablet copy: Injecting latency into file download: 1500
I20240503 01:59:45.729379 11673 ts_tablet_manager.cc:902] T a5926dae08184952bc693342c2c61ebf P 2d0fd64aaa6548e09e1ebde581c54658: Initiating tablet copy from peer 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131:44005)
I20240503 01:59:45.729947 11673 tablet_copy_client.cc:308] T a5926dae08184952bc693342c2c61ebf P 2d0fd64aaa6548e09e1ebde581c54658: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:44005
I20240503 01:59:45.730420 6647 tablet_copy_service.cc:140] P 79163c69f6f04a9abd0208d74e1c61be: Received BeginTabletCopySession request for tablet a5926dae08184952bc693342c2c61ebf from peer 2d0fd64aaa6548e09e1ebde581c54658 ({username='slave'} at 127.0.99.137:43265)
I20240503 01:59:45.730520 6647 tablet_copy_service.cc:161] P 79163c69f6f04a9abd0208d74e1c61be: Beginning new tablet copy session on tablet a5926dae08184952bc693342c2c61ebf from peer 2d0fd64aaa6548e09e1ebde581c54658 at {username='slave'} at 127.0.99.137:43265: session id = 2d0fd64aaa6548e09e1ebde581c54658-a5926dae08184952bc693342c2c61ebf
I20240503 01:59:45.731232 6647 tablet_copy_source_session.cc:205] T a5926dae08184952bc693342c2c61ebf P 79163c69f6f04a9abd0208d74e1c61be: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:45.732090 11673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5926dae08184952bc693342c2c61ebf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:45.734005 11673 tablet_copy_client.cc:783] T a5926dae08184952bc693342c2c61ebf P 2d0fd64aaa6548e09e1ebde581c54658: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:45.734136 11673 tablet_copy_client.cc:647] T a5926dae08184952bc693342c2c61ebf P 2d0fd64aaa6548e09e1ebde581c54658: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:45.735149 11677 tablet_copy_client.cc:1002] T a5926dae08184952bc693342c2c61ebf P 2d0fd64aaa6548e09e1ebde581c54658: tablet copy: Injecting latency into file download: 1500
I20240503 01:59:45.748780 11678 ts_tablet_manager.cc:902] T 1d392e08c15a4d4098a350287b3163ef P 65d4632bcdd54ba596acdbbe1b5ef8af: Initiating tablet copy from peer 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130:34189)
I20240503 01:59:45.749316 11678 tablet_copy_client.cc:308] T 1d392e08c15a4d4098a350287b3163ef P 65d4632bcdd54ba596acdbbe1b5ef8af: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:34189
I20240503 01:59:45.749722 6516 tablet_copy_service.cc:140] P 0c4821d9898d42799c5789d4d2da3f63: Received BeginTabletCopySession request for tablet 1d392e08c15a4d4098a350287b3163ef from peer 65d4632bcdd54ba596acdbbe1b5ef8af ({username='slave'} at 127.0.99.134:55903)
I20240503 01:59:45.749817 6516 tablet_copy_service.cc:161] P 0c4821d9898d42799c5789d4d2da3f63: Beginning new tablet copy session on tablet 1d392e08c15a4d4098a350287b3163ef from peer 65d4632bcdd54ba596acdbbe1b5ef8af at {username='slave'} at 127.0.99.134:55903: session id = 65d4632bcdd54ba596acdbbe1b5ef8af-1d392e08c15a4d4098a350287b3163ef
I20240503 01:59:45.750506 6516 tablet_copy_source_session.cc:205] T 1d392e08c15a4d4098a350287b3163ef P 0c4821d9898d42799c5789d4d2da3f63: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:45.751189 11678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d392e08c15a4d4098a350287b3163ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:45.753082 11678 tablet_copy_client.cc:783] T 1d392e08c15a4d4098a350287b3163ef P 65d4632bcdd54ba596acdbbe1b5ef8af: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:45.753217 11678 tablet_copy_client.cc:647] T 1d392e08c15a4d4098a350287b3163ef P 65d4632bcdd54ba596acdbbe1b5ef8af: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:45.754235 11682 tablet_copy_client.cc:1002] T 1d392e08c15a4d4098a350287b3163ef P 65d4632bcdd54ba596acdbbe1b5ef8af: tablet copy: Injecting latency into file download: 1500
I20240503 01:59:45.794346 11683 ts_tablet_manager.cc:902] T 4e7ceb1453804246a41bd1dda0e39d02 P 7a1404fe1c1b4b388b821c8261b53104: Initiating tablet copy from peer 65d4632bcdd54ba596acdbbe1b5ef8af (127.0.99.134:35795)
I20240503 01:59:45.794871 11683 tablet_copy_client.cc:308] T 4e7ceb1453804246a41bd1dda0e39d02 P 7a1404fe1c1b4b388b821c8261b53104: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.134:35795
I20240503 01:59:45.799202 8244 tablet_copy_service.cc:140] P 65d4632bcdd54ba596acdbbe1b5ef8af: Received BeginTabletCopySession request for tablet 4e7ceb1453804246a41bd1dda0e39d02 from peer 7a1404fe1c1b4b388b821c8261b53104 ({username='slave'} at 127.0.99.136:35703)
I20240503 01:59:45.799335 8244 tablet_copy_service.cc:161] P 65d4632bcdd54ba596acdbbe1b5ef8af: Beginning new tablet copy session on tablet 4e7ceb1453804246a41bd1dda0e39d02 from peer 7a1404fe1c1b4b388b821c8261b53104 at {username='slave'} at 127.0.99.136:35703: session id = 7a1404fe1c1b4b388b821c8261b53104-4e7ceb1453804246a41bd1dda0e39d02
I20240503 01:59:45.800695 8244 tablet_copy_source_session.cc:205] T 4e7ceb1453804246a41bd1dda0e39d02 P 65d4632bcdd54ba596acdbbe1b5ef8af: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:45.801796 11683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e7ceb1453804246a41bd1dda0e39d02. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:45.803792 11683 tablet_copy_client.cc:783] T 4e7ceb1453804246a41bd1dda0e39d02 P 7a1404fe1c1b4b388b821c8261b53104: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:45.803921 11683 tablet_copy_client.cc:647] T 4e7ceb1453804246a41bd1dda0e39d02 P 7a1404fe1c1b4b388b821c8261b53104: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:45.805194 11684 tablet_copy_client.cc:1002] T 4e7ceb1453804246a41bd1dda0e39d02 P 7a1404fe1c1b4b388b821c8261b53104: tablet copy: Injecting latency into file download: 1500
I20240503 01:59:45.808625 11689 ts_tablet_manager.cc:902] T ccb34a4b52244d90850ab04e21299f55 P 9264f24d028347d98f7e4e2d1aedeb79: Initiating tablet copy from peer 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130:34189)
I20240503 01:59:45.809248 11689 tablet_copy_client.cc:308] T ccb34a4b52244d90850ab04e21299f55 P 9264f24d028347d98f7e4e2d1aedeb79: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:34189
I20240503 01:59:45.809795 6516 tablet_copy_service.cc:140] P 0c4821d9898d42799c5789d4d2da3f63: Received BeginTabletCopySession request for tablet ccb34a4b52244d90850ab04e21299f55 from peer 9264f24d028347d98f7e4e2d1aedeb79 ({username='slave'} at 127.0.99.133:35409)
I20240503 01:59:45.809913 6516 tablet_copy_service.cc:161] P 0c4821d9898d42799c5789d4d2da3f63: Beginning new tablet copy session on tablet ccb34a4b52244d90850ab04e21299f55 from peer 9264f24d028347d98f7e4e2d1aedeb79 at {username='slave'} at 127.0.99.133:35409: session id = 9264f24d028347d98f7e4e2d1aedeb79-ccb34a4b52244d90850ab04e21299f55
I20240503 01:59:45.810824 6516 tablet_copy_source_session.cc:205] T ccb34a4b52244d90850ab04e21299f55 P 0c4821d9898d42799c5789d4d2da3f63: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:45.811645 11689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccb34a4b52244d90850ab04e21299f55. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:45.813797 11689 tablet_copy_client.cc:783] T ccb34a4b52244d90850ab04e21299f55 P 9264f24d028347d98f7e4e2d1aedeb79: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:45.813947 11689 tablet_copy_client.cc:647] T ccb34a4b52244d90850ab04e21299f55 P 9264f24d028347d98f7e4e2d1aedeb79: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:45.814882 11693 tablet_copy_client.cc:1002] T ccb34a4b52244d90850ab04e21299f55 P 9264f24d028347d98f7e4e2d1aedeb79: tablet copy: Injecting latency into file download: 1500
I20240503 01:59:45.819897 11694 ts_tablet_manager.cc:902] T f11a8034585a4d57be0e9b1e5e8bcd85 P 2d0fd64aaa6548e09e1ebde581c54658: Initiating tablet copy from peer b6b1d5677b8d47788b7968f64b3dac3c (127.0.99.132:46067)
I20240503 01:59:45.820402 11694 tablet_copy_client.cc:308] T f11a8034585a4d57be0e9b1e5e8bcd85 P 2d0fd64aaa6548e09e1ebde581c54658: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:46067
I20240503 01:59:45.823794 6778 tablet_copy_service.cc:140] P b6b1d5677b8d47788b7968f64b3dac3c: Received BeginTabletCopySession request for tablet f11a8034585a4d57be0e9b1e5e8bcd85 from peer 2d0fd64aaa6548e09e1ebde581c54658 ({username='slave'} at 127.0.99.137:43373)
I20240503 01:59:45.823931 6778 tablet_copy_service.cc:161] P b6b1d5677b8d47788b7968f64b3dac3c: Beginning new tablet copy session on tablet f11a8034585a4d57be0e9b1e5e8bcd85 from peer 2d0fd64aaa6548e09e1ebde581c54658 at {username='slave'} at 127.0.99.137:43373: session id = 2d0fd64aaa6548e09e1ebde581c54658-f11a8034585a4d57be0e9b1e5e8bcd85
I20240503 01:59:45.824903 6778 tablet_copy_source_session.cc:205] T f11a8034585a4d57be0e9b1e5e8bcd85 P b6b1d5677b8d47788b7968f64b3dac3c: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:45.825858 11694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f11a8034585a4d57be0e9b1e5e8bcd85. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:45.828009 11694 tablet_copy_client.cc:783] T f11a8034585a4d57be0e9b1e5e8bcd85 P 2d0fd64aaa6548e09e1ebde581c54658: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:45.828155 11694 tablet_copy_client.cc:647] T f11a8034585a4d57be0e9b1e5e8bcd85 P 2d0fd64aaa6548e09e1ebde581c54658: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:45.829365 11698 tablet_copy_client.cc:1002] T f11a8034585a4d57be0e9b1e5e8bcd85 P 2d0fd64aaa6548e09e1ebde581c54658: tablet copy: Injecting latency into file download: 1500
I20240503 01:59:46.984304 11556 tablet_copy_client.cc:523] T ff8f06c5ad8442ee942bf900212fe858 P 9264f24d028347d98f7e4e2d1aedeb79: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:46.986016 11556 tablet_bootstrap.cc:492] T ff8f06c5ad8442ee942bf900212fe858 P 9264f24d028347d98f7e4e2d1aedeb79: Bootstrap starting.
I20240503 01:59:47.010308 11556 tablet_bootstrap.cc:492] T ff8f06c5ad8442ee942bf900212fe858 P 9264f24d028347d98f7e4e2d1aedeb79: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:47.010645 11556 tablet_bootstrap.cc:492] T ff8f06c5ad8442ee942bf900212fe858 P 9264f24d028347d98f7e4e2d1aedeb79: Bootstrap complete.
I20240503 01:59:47.010838 11556 ts_tablet_manager.cc:1367] T ff8f06c5ad8442ee942bf900212fe858 P 9264f24d028347d98f7e4e2d1aedeb79: Time spent bootstrapping tablet: real 0.025s user 0.022s sys 0.000s
I20240503 01:59:47.011260 11556 raft_consensus.cc:348] T ff8f06c5ad8442ee942bf900212fe858 P 9264f24d028347d98f7e4e2d1aedeb79 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: true } }
I20240503 01:59:47.011374 11556 raft_consensus.cc:729] T ff8f06c5ad8442ee942bf900212fe858 P 9264f24d028347d98f7e4e2d1aedeb79 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9264f24d028347d98f7e4e2d1aedeb79, State: Initialized, Role: LEARNER
I20240503 01:59:47.011456 11556 consensus_queue.cc:260] T ff8f06c5ad8442ee942bf900212fe858 P 9264f24d028347d98f7e4e2d1aedeb79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: true } }
I20240503 01:59:47.011793 11556 ts_tablet_manager.cc:1398] T ff8f06c5ad8442ee942bf900212fe858 P 9264f24d028347d98f7e4e2d1aedeb79: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240503 01:59:47.012802 6516 tablet_copy_service.cc:342] P 0c4821d9898d42799c5789d4d2da3f63: Request end of tablet copy session 9264f24d028347d98f7e4e2d1aedeb79-ff8f06c5ad8442ee942bf900212fe858 received from {username='slave'} at 127.0.99.133:35409
I20240503 01:59:47.012881 6516 tablet_copy_service.cc:433] P 0c4821d9898d42799c5789d4d2da3f63: ending tablet copy session 9264f24d028347d98f7e4e2d1aedeb79-ff8f06c5ad8442ee942bf900212fe858 on tablet ff8f06c5ad8442ee942bf900212fe858 with peer 9264f24d028347d98f7e4e2d1aedeb79
I20240503 01:59:47.016402 11567 tablet_copy_client.cc:523] T 897e88ac93484a11829189e027de8bae P 7a1404fe1c1b4b388b821c8261b53104: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:47.017928 11567 tablet_bootstrap.cc:492] T 897e88ac93484a11829189e027de8bae P 7a1404fe1c1b4b388b821c8261b53104: Bootstrap starting.
I20240503 01:59:47.023308 11561 tablet_copy_client.cc:523] T 120d5dd4f26c42399702141094ce7982 P 2d0fd64aaa6548e09e1ebde581c54658: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:47.025426 11561 tablet_bootstrap.cc:492] T 120d5dd4f26c42399702141094ce7982 P 2d0fd64aaa6548e09e1ebde581c54658: Bootstrap starting.
I20240503 01:59:47.032076 11561 log.cc:824] T 120d5dd4f26c42399702141094ce7982 P 2d0fd64aaa6548e09e1ebde581c54658: Log is configured to *not* fsync() on all Append() calls
I20240503 01:59:47.042817 11567 tablet_bootstrap.cc:492] T 897e88ac93484a11829189e027de8bae P 7a1404fe1c1b4b388b821c8261b53104: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:47.043140 11567 tablet_bootstrap.cc:492] T 897e88ac93484a11829189e027de8bae P 7a1404fe1c1b4b388b821c8261b53104: Bootstrap complete.
I20240503 01:59:47.043321 11567 ts_tablet_manager.cc:1367] T 897e88ac93484a11829189e027de8bae P 7a1404fe1c1b4b388b821c8261b53104: Time spent bootstrapping tablet: real 0.025s user 0.023s sys 0.000s
I20240503 01:59:47.043738 11567 raft_consensus.cc:348] T 897e88ac93484a11829189e027de8bae P 7a1404fe1c1b4b388b821c8261b53104 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: true } }
I20240503 01:59:47.043877 11567 raft_consensus.cc:729] T 897e88ac93484a11829189e027de8bae P 7a1404fe1c1b4b388b821c8261b53104 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7a1404fe1c1b4b388b821c8261b53104, State: Initialized, Role: LEARNER
I20240503 01:59:47.043994 11567 consensus_queue.cc:260] T 897e88ac93484a11829189e027de8bae P 7a1404fe1c1b4b388b821c8261b53104 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: true } }
I20240503 01:59:47.044355 11567 ts_tablet_manager.cc:1398] T 897e88ac93484a11829189e027de8bae P 7a1404fe1c1b4b388b821c8261b53104: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240503 01:59:47.044888 6516 tablet_copy_service.cc:342] P 0c4821d9898d42799c5789d4d2da3f63: Request end of tablet copy session 7a1404fe1c1b4b388b821c8261b53104-897e88ac93484a11829189e027de8bae received from {username='slave'} at 127.0.99.136:32823
I20240503 01:59:47.044966 6516 tablet_copy_service.cc:433] P 0c4821d9898d42799c5789d4d2da3f63: ending tablet copy session 7a1404fe1c1b4b388b821c8261b53104-897e88ac93484a11829189e027de8bae on tablet 897e88ac93484a11829189e027de8bae with peer 7a1404fe1c1b4b388b821c8261b53104
I20240503 01:59:47.056563 11561 tablet_bootstrap.cc:492] T 120d5dd4f26c42399702141094ce7982 P 2d0fd64aaa6548e09e1ebde581c54658: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:47.056880 11561 tablet_bootstrap.cc:492] T 120d5dd4f26c42399702141094ce7982 P 2d0fd64aaa6548e09e1ebde581c54658: Bootstrap complete.
I20240503 01:59:47.057130 11561 ts_tablet_manager.cc:1367] T 120d5dd4f26c42399702141094ce7982 P 2d0fd64aaa6548e09e1ebde581c54658: Time spent bootstrapping tablet: real 0.032s user 0.022s sys 0.009s
I20240503 01:59:47.058943 11561 raft_consensus.cc:348] T 120d5dd4f26c42399702141094ce7982 P 2d0fd64aaa6548e09e1ebde581c54658 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: true } }
I20240503 01:59:47.059139 11561 raft_consensus.cc:729] T 120d5dd4f26c42399702141094ce7982 P 2d0fd64aaa6548e09e1ebde581c54658 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2d0fd64aaa6548e09e1ebde581c54658, State: Initialized, Role: LEARNER
I20240503 01:59:47.059391 11561 consensus_queue.cc:260] T 120d5dd4f26c42399702141094ce7982 P 2d0fd64aaa6548e09e1ebde581c54658 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: true } }
I20240503 01:59:47.060081 11561 ts_tablet_manager.cc:1398] T 120d5dd4f26c42399702141094ce7982 P 2d0fd64aaa6548e09e1ebde581c54658: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20240503 01:59:47.060693 6516 tablet_copy_service.cc:342] P 0c4821d9898d42799c5789d4d2da3f63: Request end of tablet copy session 2d0fd64aaa6548e09e1ebde581c54658-120d5dd4f26c42399702141094ce7982 received from {username='slave'} at 127.0.99.137:56177
I20240503 01:59:47.060786 6516 tablet_copy_service.cc:433] P 0c4821d9898d42799c5789d4d2da3f63: ending tablet copy session 2d0fd64aaa6548e09e1ebde581c54658-120d5dd4f26c42399702141094ce7982 on tablet 120d5dd4f26c42399702141094ce7982 with peer 2d0fd64aaa6548e09e1ebde581c54658
I20240503 01:59:47.066428 8486 raft_consensus.cc:1203] T 897e88ac93484a11829189e027de8bae P 7a1404fe1c1b4b388b821c8261b53104 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 01:59:47.092697 11607 tablet_copy_client.cc:523] T a9a74e04ce3745b2b479e96a3f3dc86e P 2d0fd64aaa6548e09e1ebde581c54658: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:47.094154 11607 tablet_bootstrap.cc:492] T a9a74e04ce3745b2b479e96a3f3dc86e P 2d0fd64aaa6548e09e1ebde581c54658: Bootstrap starting.
I20240503 01:59:47.096449 11617 tablet_copy_client.cc:523] T 21d27cca81e8455e9934e1cc604fc2b2 P 9264f24d028347d98f7e4e2d1aedeb79: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:47.097939 11617 tablet_bootstrap.cc:492] T 21d27cca81e8455e9934e1cc604fc2b2 P 9264f24d028347d98f7e4e2d1aedeb79: Bootstrap starting.
I20240503 01:59:47.120076 11607 tablet_bootstrap.cc:492] T a9a74e04ce3745b2b479e96a3f3dc86e P 2d0fd64aaa6548e09e1ebde581c54658: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:47.120378 11607 tablet_bootstrap.cc:492] T a9a74e04ce3745b2b479e96a3f3dc86e P 2d0fd64aaa6548e09e1ebde581c54658: Bootstrap complete.
I20240503 01:59:47.120548 11607 ts_tablet_manager.cc:1367] T a9a74e04ce3745b2b479e96a3f3dc86e P 2d0fd64aaa6548e09e1ebde581c54658: Time spent bootstrapping tablet: real 0.026s user 0.023s sys 0.000s
I20240503 01:59:47.121126 11607 raft_consensus.cc:348] T a9a74e04ce3745b2b479e96a3f3dc86e P 2d0fd64aaa6548e09e1ebde581c54658 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: true } }
I20240503 01:59:47.121258 11607 raft_consensus.cc:729] T a9a74e04ce3745b2b479e96a3f3dc86e P 2d0fd64aaa6548e09e1ebde581c54658 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2d0fd64aaa6548e09e1ebde581c54658, State: Initialized, Role: LEARNER
I20240503 01:59:47.121352 11607 consensus_queue.cc:260] T a9a74e04ce3745b2b479e96a3f3dc86e P 2d0fd64aaa6548e09e1ebde581c54658 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: true } }
I20240503 01:59:47.121635 11607 ts_tablet_manager.cc:1398] T a9a74e04ce3745b2b479e96a3f3dc86e P 2d0fd64aaa6548e09e1ebde581c54658: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240503 01:59:47.122144 6647 tablet_copy_service.cc:342] P 79163c69f6f04a9abd0208d74e1c61be: Request end of tablet copy session 2d0fd64aaa6548e09e1ebde581c54658-a9a74e04ce3745b2b479e96a3f3dc86e received from {username='slave'} at 127.0.99.137:43265
I20240503 01:59:47.122236 6647 tablet_copy_service.cc:433] P 79163c69f6f04a9abd0208d74e1c61be: ending tablet copy session 2d0fd64aaa6548e09e1ebde581c54658-a9a74e04ce3745b2b479e96a3f3dc86e on tablet a9a74e04ce3745b2b479e96a3f3dc86e with peer 2d0fd64aaa6548e09e1ebde581c54658
I20240503 01:59:47.124816 11617 tablet_bootstrap.cc:492] T 21d27cca81e8455e9934e1cc604fc2b2 P 9264f24d028347d98f7e4e2d1aedeb79: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:47.125027 11617 tablet_bootstrap.cc:492] T 21d27cca81e8455e9934e1cc604fc2b2 P 9264f24d028347d98f7e4e2d1aedeb79: Bootstrap complete.
I20240503 01:59:47.125151 11617 ts_tablet_manager.cc:1367] T 21d27cca81e8455e9934e1cc604fc2b2 P 9264f24d028347d98f7e4e2d1aedeb79: Time spent bootstrapping tablet: real 0.027s user 0.028s sys 0.000s
I20240503 01:59:47.125494 11617 raft_consensus.cc:348] T 21d27cca81e8455e9934e1cc604fc2b2 P 9264f24d028347d98f7e4e2d1aedeb79 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: true } }
I20240503 01:59:47.125591 11617 raft_consensus.cc:729] T 21d27cca81e8455e9934e1cc604fc2b2 P 9264f24d028347d98f7e4e2d1aedeb79 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9264f24d028347d98f7e4e2d1aedeb79, State: Initialized, Role: LEARNER
I20240503 01:59:47.125656 11617 consensus_queue.cc:260] T 21d27cca81e8455e9934e1cc604fc2b2 P 9264f24d028347d98f7e4e2d1aedeb79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: true } }
I20240503 01:59:47.125895 11617 ts_tablet_manager.cc:1398] T 21d27cca81e8455e9934e1cc604fc2b2 P 9264f24d028347d98f7e4e2d1aedeb79: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:47.126366 6647 tablet_copy_service.cc:342] P 79163c69f6f04a9abd0208d74e1c61be: Request end of tablet copy session 9264f24d028347d98f7e4e2d1aedeb79-21d27cca81e8455e9934e1cc604fc2b2 received from {username='slave'} at 127.0.99.133:50661
I20240503 01:59:47.126441 6647 tablet_copy_service.cc:433] P 79163c69f6f04a9abd0208d74e1c61be: ending tablet copy session 9264f24d028347d98f7e4e2d1aedeb79-21d27cca81e8455e9934e1cc604fc2b2 on tablet 21d27cca81e8455e9934e1cc604fc2b2 with peer 9264f24d028347d98f7e4e2d1aedeb79
I20240503 01:59:47.128544 9410 raft_consensus.cc:1203] T a9a74e04ce3745b2b479e96a3f3dc86e P 2d0fd64aaa6548e09e1ebde581c54658 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 01:59:47.131011 11652 tablet_copy_client.cc:523] T d6ee8175874c4063986033b0a80ccb82 P 65d4632bcdd54ba596acdbbe1b5ef8af: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:47.132292 11652 tablet_bootstrap.cc:492] T d6ee8175874c4063986033b0a80ccb82 P 65d4632bcdd54ba596acdbbe1b5ef8af: Bootstrap starting.
I20240503 01:59:47.151849 11652 tablet_bootstrap.cc:492] T d6ee8175874c4063986033b0a80ccb82 P 65d4632bcdd54ba596acdbbe1b5ef8af: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:47.152177 11652 tablet_bootstrap.cc:492] T d6ee8175874c4063986033b0a80ccb82 P 65d4632bcdd54ba596acdbbe1b5ef8af: Bootstrap complete.
I20240503 01:59:47.152344 11652 ts_tablet_manager.cc:1367] T d6ee8175874c4063986033b0a80ccb82 P 65d4632bcdd54ba596acdbbe1b5ef8af: Time spent bootstrapping tablet: real 0.020s user 0.016s sys 0.005s
I20240503 01:59:47.152508 11657 tablet_copy_client.cc:523] T 2ac8004211e64c3481e75377092fa93b P 2d0fd64aaa6548e09e1ebde581c54658: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:47.152817 11652 raft_consensus.cc:348] T d6ee8175874c4063986033b0a80ccb82 P 65d4632bcdd54ba596acdbbe1b5ef8af [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: true } }
I20240503 01:59:47.152932 11652 raft_consensus.cc:729] T d6ee8175874c4063986033b0a80ccb82 P 65d4632bcdd54ba596acdbbe1b5ef8af [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 65d4632bcdd54ba596acdbbe1b5ef8af, State: Initialized, Role: LEARNER
I20240503 01:59:47.153025 11652 consensus_queue.cc:260] T d6ee8175874c4063986033b0a80ccb82 P 65d4632bcdd54ba596acdbbe1b5ef8af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: true } }
I20240503 01:59:47.153384 11652 ts_tablet_manager.cc:1398] T d6ee8175874c4063986033b0a80ccb82 P 65d4632bcdd54ba596acdbbe1b5ef8af: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:47.153820 6647 tablet_copy_service.cc:342] P 79163c69f6f04a9abd0208d74e1c61be: Request end of tablet copy session 65d4632bcdd54ba596acdbbe1b5ef8af-d6ee8175874c4063986033b0a80ccb82 received from {username='slave'} at 127.0.99.134:43049
I20240503 01:59:47.153906 6647 tablet_copy_service.cc:433] P 79163c69f6f04a9abd0208d74e1c61be: ending tablet copy session 65d4632bcdd54ba596acdbbe1b5ef8af-d6ee8175874c4063986033b0a80ccb82 on tablet d6ee8175874c4063986033b0a80ccb82 with peer 65d4632bcdd54ba596acdbbe1b5ef8af
I20240503 01:59:47.154033 11657 tablet_bootstrap.cc:492] T 2ac8004211e64c3481e75377092fa93b P 2d0fd64aaa6548e09e1ebde581c54658: Bootstrap starting.
I20240503 01:59:47.166680 11662 tablet_copy_client.cc:523] T 82cfe99cd6fe4a82a7e35298ec634b8b P 65d4632bcdd54ba596acdbbe1b5ef8af: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:47.167882 11662 tablet_bootstrap.cc:492] T 82cfe99cd6fe4a82a7e35298ec634b8b P 65d4632bcdd54ba596acdbbe1b5ef8af: Bootstrap starting.
I20240503 01:59:47.172273 11657 tablet_bootstrap.cc:492] T 2ac8004211e64c3481e75377092fa93b P 2d0fd64aaa6548e09e1ebde581c54658: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:47.172521 11657 tablet_bootstrap.cc:492] T 2ac8004211e64c3481e75377092fa93b P 2d0fd64aaa6548e09e1ebde581c54658: Bootstrap complete.
I20240503 01:59:47.172703 11657 ts_tablet_manager.cc:1367] T 2ac8004211e64c3481e75377092fa93b P 2d0fd64aaa6548e09e1ebde581c54658: Time spent bootstrapping tablet: real 0.019s user 0.015s sys 0.005s
I20240503 01:59:47.173096 11657 raft_consensus.cc:348] T 2ac8004211e64c3481e75377092fa93b P 2d0fd64aaa6548e09e1ebde581c54658 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: true } }
I20240503 01:59:47.173199 11657 raft_consensus.cc:729] T 2ac8004211e64c3481e75377092fa93b P 2d0fd64aaa6548e09e1ebde581c54658 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2d0fd64aaa6548e09e1ebde581c54658, State: Initialized, Role: LEARNER
I20240503 01:59:47.173307 11657 consensus_queue.cc:260] T 2ac8004211e64c3481e75377092fa93b P 2d0fd64aaa6548e09e1ebde581c54658 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: true } }
I20240503 01:59:47.173621 11657 ts_tablet_manager.cc:1398] T 2ac8004211e64c3481e75377092fa93b P 2d0fd64aaa6548e09e1ebde581c54658: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:47.174139 6516 tablet_copy_service.cc:342] P 0c4821d9898d42799c5789d4d2da3f63: Request end of tablet copy session 2d0fd64aaa6548e09e1ebde581c54658-2ac8004211e64c3481e75377092fa93b received from {username='slave'} at 127.0.99.137:56177
I20240503 01:59:47.174232 6516 tablet_copy_service.cc:433] P 0c4821d9898d42799c5789d4d2da3f63: ending tablet copy session 2d0fd64aaa6548e09e1ebde581c54658-2ac8004211e64c3481e75377092fa93b on tablet 2ac8004211e64c3481e75377092fa93b with peer 2d0fd64aaa6548e09e1ebde581c54658
I20240503 01:59:47.187283 11662 tablet_bootstrap.cc:492] T 82cfe99cd6fe4a82a7e35298ec634b8b P 65d4632bcdd54ba596acdbbe1b5ef8af: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:47.187546 11662 tablet_bootstrap.cc:492] T 82cfe99cd6fe4a82a7e35298ec634b8b P 65d4632bcdd54ba596acdbbe1b5ef8af: Bootstrap complete.
I20240503 01:59:47.187697 11662 ts_tablet_manager.cc:1367] T 82cfe99cd6fe4a82a7e35298ec634b8b P 65d4632bcdd54ba596acdbbe1b5ef8af: Time spent bootstrapping tablet: real 0.020s user 0.013s sys 0.004s
I20240503 01:59:47.188066 11662 raft_consensus.cc:348] T 82cfe99cd6fe4a82a7e35298ec634b8b P 65d4632bcdd54ba596acdbbe1b5ef8af [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: true } }
I20240503 01:59:47.188167 11662 raft_consensus.cc:729] T 82cfe99cd6fe4a82a7e35298ec634b8b P 65d4632bcdd54ba596acdbbe1b5ef8af [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 65d4632bcdd54ba596acdbbe1b5ef8af, State: Initialized, Role: LEARNER
I20240503 01:59:47.188263 11662 consensus_queue.cc:260] T 82cfe99cd6fe4a82a7e35298ec634b8b P 65d4632bcdd54ba596acdbbe1b5ef8af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: true } }
I20240503 01:59:47.188499 11662 ts_tablet_manager.cc:1398] T 82cfe99cd6fe4a82a7e35298ec634b8b P 65d4632bcdd54ba596acdbbe1b5ef8af: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:47.189069 6385 tablet_copy_service.cc:342] P 4127249ba42644388d225d614c935a25: Request end of tablet copy session 65d4632bcdd54ba596acdbbe1b5ef8af-82cfe99cd6fe4a82a7e35298ec634b8b received from {username='slave'} at 127.0.99.134:49429
I20240503 01:59:47.189168 6385 tablet_copy_service.cc:433] P 4127249ba42644388d225d614c935a25: ending tablet copy session 65d4632bcdd54ba596acdbbe1b5ef8af-82cfe99cd6fe4a82a7e35298ec634b8b on tablet 82cfe99cd6fe4a82a7e35298ec634b8b with peer 65d4632bcdd54ba596acdbbe1b5ef8af
I20240503 01:59:47.197538 9410 raft_consensus.cc:1203] T 2ac8004211e64c3481e75377092fa93b P 2d0fd64aaa6548e09e1ebde581c54658 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 01:59:47.207257 11667 tablet_copy_client.cc:523] T 5b599bd1fea6409fbe485c7a35c0891c P 65d4632bcdd54ba596acdbbe1b5ef8af: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:47.208634 11667 tablet_bootstrap.cc:492] T 5b599bd1fea6409fbe485c7a35c0891c P 65d4632bcdd54ba596acdbbe1b5ef8af: Bootstrap starting.
I20240503 01:59:47.227370 11667 tablet_bootstrap.cc:492] T 5b599bd1fea6409fbe485c7a35c0891c P 65d4632bcdd54ba596acdbbe1b5ef8af: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:47.227625 11667 tablet_bootstrap.cc:492] T 5b599bd1fea6409fbe485c7a35c0891c P 65d4632bcdd54ba596acdbbe1b5ef8af: Bootstrap complete.
I20240503 01:59:47.227779 11667 ts_tablet_manager.cc:1367] T 5b599bd1fea6409fbe485c7a35c0891c P 65d4632bcdd54ba596acdbbe1b5ef8af: Time spent bootstrapping tablet: real 0.019s user 0.017s sys 0.000s
I20240503 01:59:47.228127 11667 raft_consensus.cc:348] T 5b599bd1fea6409fbe485c7a35c0891c P 65d4632bcdd54ba596acdbbe1b5ef8af [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: true } }
I20240503 01:59:47.228231 11667 raft_consensus.cc:729] T 5b599bd1fea6409fbe485c7a35c0891c P 65d4632bcdd54ba596acdbbe1b5ef8af [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 65d4632bcdd54ba596acdbbe1b5ef8af, State: Initialized, Role: LEARNER
I20240503 01:59:47.228305 11667 consensus_queue.cc:260] T 5b599bd1fea6409fbe485c7a35c0891c P 65d4632bcdd54ba596acdbbe1b5ef8af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: true } }
I20240503 01:59:47.228544 11667 ts_tablet_manager.cc:1398] T 5b599bd1fea6409fbe485c7a35c0891c P 65d4632bcdd54ba596acdbbe1b5ef8af: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:47.229049 6647 tablet_copy_service.cc:342] P 79163c69f6f04a9abd0208d74e1c61be: Request end of tablet copy session 65d4632bcdd54ba596acdbbe1b5ef8af-5b599bd1fea6409fbe485c7a35c0891c received from {username='slave'} at 127.0.99.134:43049
I20240503 01:59:47.229146 6647 tablet_copy_service.cc:433] P 79163c69f6f04a9abd0208d74e1c61be: ending tablet copy session 65d4632bcdd54ba596acdbbe1b5ef8af-5b599bd1fea6409fbe485c7a35c0891c on tablet 5b599bd1fea6409fbe485c7a35c0891c with peer 65d4632bcdd54ba596acdbbe1b5ef8af
I20240503 01:59:47.235541 11673 tablet_copy_client.cc:523] T a5926dae08184952bc693342c2c61ebf P 2d0fd64aaa6548e09e1ebde581c54658: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:47.236881 11673 tablet_bootstrap.cc:492] T a5926dae08184952bc693342c2c61ebf P 2d0fd64aaa6548e09e1ebde581c54658: Bootstrap starting.
I20240503 01:59:47.238905 8223 raft_consensus.cc:1203] T d6ee8175874c4063986033b0a80ccb82 P 65d4632bcdd54ba596acdbbe1b5ef8af [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 01:59:47.254516 11678 tablet_copy_client.cc:523] T 1d392e08c15a4d4098a350287b3163ef P 65d4632bcdd54ba596acdbbe1b5ef8af: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:47.255964 11678 tablet_bootstrap.cc:492] T 1d392e08c15a4d4098a350287b3163ef P 65d4632bcdd54ba596acdbbe1b5ef8af: Bootstrap starting.
I20240503 01:59:47.258304 11673 tablet_bootstrap.cc:492] T a5926dae08184952bc693342c2c61ebf P 2d0fd64aaa6548e09e1ebde581c54658: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:47.258576 11673 tablet_bootstrap.cc:492] T a5926dae08184952bc693342c2c61ebf P 2d0fd64aaa6548e09e1ebde581c54658: Bootstrap complete.
I20240503 01:59:47.258746 11673 ts_tablet_manager.cc:1367] T a5926dae08184952bc693342c2c61ebf P 2d0fd64aaa6548e09e1ebde581c54658: Time spent bootstrapping tablet: real 0.022s user 0.020s sys 0.000s
I20240503 01:59:47.259307 11673 raft_consensus.cc:348] T a5926dae08184952bc693342c2c61ebf P 2d0fd64aaa6548e09e1ebde581c54658 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: true } }
I20240503 01:59:47.259438 11673 raft_consensus.cc:729] T a5926dae08184952bc693342c2c61ebf P 2d0fd64aaa6548e09e1ebde581c54658 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2d0fd64aaa6548e09e1ebde581c54658, State: Initialized, Role: LEARNER
I20240503 01:59:47.259589 11673 consensus_queue.cc:260] T a5926dae08184952bc693342c2c61ebf P 2d0fd64aaa6548e09e1ebde581c54658 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: true } }
I20240503 01:59:47.259995 11673 ts_tablet_manager.cc:1398] T a5926dae08184952bc693342c2c61ebf P 2d0fd64aaa6548e09e1ebde581c54658: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:47.261251 6647 tablet_copy_service.cc:342] P 79163c69f6f04a9abd0208d74e1c61be: Request end of tablet copy session 2d0fd64aaa6548e09e1ebde581c54658-a5926dae08184952bc693342c2c61ebf received from {username='slave'} at 127.0.99.137:43265
I20240503 01:59:47.261350 6647 tablet_copy_service.cc:433] P 79163c69f6f04a9abd0208d74e1c61be: ending tablet copy session 2d0fd64aaa6548e09e1ebde581c54658-a5926dae08184952bc693342c2c61ebf on tablet a5926dae08184952bc693342c2c61ebf with peer 2d0fd64aaa6548e09e1ebde581c54658
I20240503 01:59:47.265782 8223 raft_consensus.cc:1203] T 82cfe99cd6fe4a82a7e35298ec634b8b P 65d4632bcdd54ba596acdbbe1b5ef8af [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 01:59:47.282001 11678 tablet_bootstrap.cc:492] T 1d392e08c15a4d4098a350287b3163ef P 65d4632bcdd54ba596acdbbe1b5ef8af: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:47.296873 11678 tablet_bootstrap.cc:492] T 1d392e08c15a4d4098a350287b3163ef P 65d4632bcdd54ba596acdbbe1b5ef8af: Bootstrap complete.
I20240503 01:59:47.297137 11678 ts_tablet_manager.cc:1367] T 1d392e08c15a4d4098a350287b3163ef P 65d4632bcdd54ba596acdbbe1b5ef8af: Time spent bootstrapping tablet: real 0.041s user 0.024s sys 0.000s
I20240503 01:59:47.297616 11678 raft_consensus.cc:348] T 1d392e08c15a4d4098a350287b3163ef P 65d4632bcdd54ba596acdbbe1b5ef8af [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: true } }
I20240503 01:59:47.297794 11678 raft_consensus.cc:729] T 1d392e08c15a4d4098a350287b3163ef P 65d4632bcdd54ba596acdbbe1b5ef8af [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 65d4632bcdd54ba596acdbbe1b5ef8af, State: Initialized, Role: LEARNER
I20240503 01:59:47.297910 11678 consensus_queue.cc:260] T 1d392e08c15a4d4098a350287b3163ef P 65d4632bcdd54ba596acdbbe1b5ef8af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: true } }
I20240503 01:59:47.298216 11678 ts_tablet_manager.cc:1398] T 1d392e08c15a4d4098a350287b3163ef P 65d4632bcdd54ba596acdbbe1b5ef8af: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:47.298679 6516 tablet_copy_service.cc:342] P 0c4821d9898d42799c5789d4d2da3f63: Request end of tablet copy session 65d4632bcdd54ba596acdbbe1b5ef8af-1d392e08c15a4d4098a350287b3163ef received from {username='slave'} at 127.0.99.134:55903
I20240503 01:59:47.298759 6516 tablet_copy_service.cc:433] P 0c4821d9898d42799c5789d4d2da3f63: ending tablet copy session 65d4632bcdd54ba596acdbbe1b5ef8af-1d392e08c15a4d4098a350287b3163ef on tablet 1d392e08c15a4d4098a350287b3163ef with peer 65d4632bcdd54ba596acdbbe1b5ef8af
I20240503 01:59:47.305495 11683 tablet_copy_client.cc:523] T 4e7ceb1453804246a41bd1dda0e39d02 P 7a1404fe1c1b4b388b821c8261b53104: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:47.307116 11683 tablet_bootstrap.cc:492] T 4e7ceb1453804246a41bd1dda0e39d02 P 7a1404fe1c1b4b388b821c8261b53104: Bootstrap starting.
I20240503 01:59:47.315227 11689 tablet_copy_client.cc:523] T ccb34a4b52244d90850ab04e21299f55 P 9264f24d028347d98f7e4e2d1aedeb79: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:47.316529 11689 tablet_bootstrap.cc:492] T ccb34a4b52244d90850ab04e21299f55 P 9264f24d028347d98f7e4e2d1aedeb79: Bootstrap starting.
I20240503 01:59:47.329668 11694 tablet_copy_client.cc:523] T f11a8034585a4d57be0e9b1e5e8bcd85 P 2d0fd64aaa6548e09e1ebde581c54658: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:47.331274 11694 tablet_bootstrap.cc:492] T f11a8034585a4d57be0e9b1e5e8bcd85 P 2d0fd64aaa6548e09e1ebde581c54658: Bootstrap starting.
I20240503 01:59:47.331715 11683 tablet_bootstrap.cc:492] T 4e7ceb1453804246a41bd1dda0e39d02 P 7a1404fe1c1b4b388b821c8261b53104: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:47.332031 11683 tablet_bootstrap.cc:492] T 4e7ceb1453804246a41bd1dda0e39d02 P 7a1404fe1c1b4b388b821c8261b53104: Bootstrap complete.
I20240503 01:59:47.332201 11683 ts_tablet_manager.cc:1367] T 4e7ceb1453804246a41bd1dda0e39d02 P 7a1404fe1c1b4b388b821c8261b53104: Time spent bootstrapping tablet: real 0.025s user 0.025s sys 0.000s
I20240503 01:59:47.332718 11683 raft_consensus.cc:348] T 4e7ceb1453804246a41bd1dda0e39d02 P 7a1404fe1c1b4b388b821c8261b53104 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: true } }
I20240503 01:59:47.332841 11683 raft_consensus.cc:729] T 4e7ceb1453804246a41bd1dda0e39d02 P 7a1404fe1c1b4b388b821c8261b53104 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 7a1404fe1c1b4b388b821c8261b53104, State: Initialized, Role: LEARNER
I20240503 01:59:47.332944 11683 consensus_queue.cc:260] T 4e7ceb1453804246a41bd1dda0e39d02 P 7a1404fe1c1b4b388b821c8261b53104 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: true } }
I20240503 01:59:47.333250 11683 ts_tablet_manager.cc:1398] T 4e7ceb1453804246a41bd1dda0e39d02 P 7a1404fe1c1b4b388b821c8261b53104: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 01:59:47.334034 8244 tablet_copy_service.cc:342] P 65d4632bcdd54ba596acdbbe1b5ef8af: Request end of tablet copy session 7a1404fe1c1b4b388b821c8261b53104-4e7ceb1453804246a41bd1dda0e39d02 received from {username='slave'} at 127.0.99.136:35703
I20240503 01:59:47.334146 8244 tablet_copy_service.cc:433] P 65d4632bcdd54ba596acdbbe1b5ef8af: ending tablet copy session 7a1404fe1c1b4b388b821c8261b53104-4e7ceb1453804246a41bd1dda0e39d02 on tablet 4e7ceb1453804246a41bd1dda0e39d02 with peer 7a1404fe1c1b4b388b821c8261b53104
I20240503 01:59:47.342558 11689 tablet_bootstrap.cc:492] T ccb34a4b52244d90850ab04e21299f55 P 9264f24d028347d98f7e4e2d1aedeb79: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:47.342954 11689 tablet_bootstrap.cc:492] T ccb34a4b52244d90850ab04e21299f55 P 9264f24d028347d98f7e4e2d1aedeb79: Bootstrap complete.
I20240503 01:59:47.343192 11689 ts_tablet_manager.cc:1367] T ccb34a4b52244d90850ab04e21299f55 P 9264f24d028347d98f7e4e2d1aedeb79: Time spent bootstrapping tablet: real 0.027s user 0.020s sys 0.005s
I20240503 01:59:47.343609 11689 raft_consensus.cc:348] T ccb34a4b52244d90850ab04e21299f55 P 9264f24d028347d98f7e4e2d1aedeb79 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: true } }
I20240503 01:59:47.343751 11689 raft_consensus.cc:729] T ccb34a4b52244d90850ab04e21299f55 P 9264f24d028347d98f7e4e2d1aedeb79 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9264f24d028347d98f7e4e2d1aedeb79, State: Initialized, Role: LEARNER
I20240503 01:59:47.343859 11689 consensus_queue.cc:260] T ccb34a4b52244d90850ab04e21299f55 P 9264f24d028347d98f7e4e2d1aedeb79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: true } }
I20240503 01:59:47.344137 11689 ts_tablet_manager.cc:1398] T ccb34a4b52244d90850ab04e21299f55 P 9264f24d028347d98f7e4e2d1aedeb79: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:47.344581 6516 tablet_copy_service.cc:342] P 0c4821d9898d42799c5789d4d2da3f63: Request end of tablet copy session 9264f24d028347d98f7e4e2d1aedeb79-ccb34a4b52244d90850ab04e21299f55 received from {username='slave'} at 127.0.99.133:35409
I20240503 01:59:47.344684 6516 tablet_copy_service.cc:433] P 0c4821d9898d42799c5789d4d2da3f63: ending tablet copy session 9264f24d028347d98f7e4e2d1aedeb79-ccb34a4b52244d90850ab04e21299f55 on tablet ccb34a4b52244d90850ab04e21299f55 with peer 9264f24d028347d98f7e4e2d1aedeb79
I20240503 01:59:47.359850 11694 tablet_bootstrap.cc:492] T f11a8034585a4d57be0e9b1e5e8bcd85 P 2d0fd64aaa6548e09e1ebde581c54658: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:47.360136 11694 tablet_bootstrap.cc:492] T f11a8034585a4d57be0e9b1e5e8bcd85 P 2d0fd64aaa6548e09e1ebde581c54658: Bootstrap complete.
I20240503 01:59:47.360307 11694 ts_tablet_manager.cc:1367] T f11a8034585a4d57be0e9b1e5e8bcd85 P 2d0fd64aaa6548e09e1ebde581c54658: Time spent bootstrapping tablet: real 0.029s user 0.023s sys 0.000s
I20240503 01:59:47.360747 11694 raft_consensus.cc:348] T f11a8034585a4d57be0e9b1e5e8bcd85 P 2d0fd64aaa6548e09e1ebde581c54658 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: true } }
I20240503 01:59:47.360855 11694 raft_consensus.cc:729] T f11a8034585a4d57be0e9b1e5e8bcd85 P 2d0fd64aaa6548e09e1ebde581c54658 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2d0fd64aaa6548e09e1ebde581c54658, State: Initialized, Role: LEARNER
I20240503 01:59:47.360955 11694 consensus_queue.cc:260] T f11a8034585a4d57be0e9b1e5e8bcd85 P 2d0fd64aaa6548e09e1ebde581c54658 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: true } }
I20240503 01:59:47.361189 11694 ts_tablet_manager.cc:1398] T f11a8034585a4d57be0e9b1e5e8bcd85 P 2d0fd64aaa6548e09e1ebde581c54658: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240503 01:59:47.361709 6778 tablet_copy_service.cc:342] P b6b1d5677b8d47788b7968f64b3dac3c: Request end of tablet copy session 2d0fd64aaa6548e09e1ebde581c54658-f11a8034585a4d57be0e9b1e5e8bcd85 received from {username='slave'} at 127.0.99.137:43373
I20240503 01:59:47.361785 6778 tablet_copy_service.cc:433] P b6b1d5677b8d47788b7968f64b3dac3c: ending tablet copy session 2d0fd64aaa6548e09e1ebde581c54658-f11a8034585a4d57be0e9b1e5e8bcd85 on tablet f11a8034585a4d57be0e9b1e5e8bcd85 with peer 2d0fd64aaa6548e09e1ebde581c54658
I20240503 01:59:47.370348 8080 raft_consensus.cc:1203] T ccb34a4b52244d90850ab04e21299f55 P 9264f24d028347d98f7e4e2d1aedeb79 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 01:59:47.380324 8080 raft_consensus.cc:1203] T ff8f06c5ad8442ee942bf900212fe858 P 9264f24d028347d98f7e4e2d1aedeb79 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 01:59:47.410257 8486 raft_consensus.cc:1203] T 4e7ceb1453804246a41bd1dda0e39d02 P 7a1404fe1c1b4b388b821c8261b53104 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
I20240503 01:59:47.446333 9410 raft_consensus.cc:1203] T a5926dae08184952bc693342c2c61ebf P 2d0fd64aaa6548e09e1ebde581c54658 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 01:59:47.469365 8223 raft_consensus.cc:1203] T 5b599bd1fea6409fbe485c7a35c0891c P 65d4632bcdd54ba596acdbbe1b5ef8af [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 01:59:47.484927 12149 raft_consensus.cc:1050] T 897e88ac93484a11829189e027de8bae P 0c4821d9898d42799c5789d4d2da3f63: attempting to promote NON_VOTER 7a1404fe1c1b4b388b821c8261b53104 to VOTER
I20240503 01:59:47.485397 12149 consensus_queue.cc:237] T 897e88ac93484a11829189e027de8bae P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } }
I20240503 01:59:47.486780 8486 raft_consensus.cc:1261] T 897e88ac93484a11829189e027de8bae P 7a1404fe1c1b4b388b821c8261b53104 [term 1 LEARNER]: Refusing update from remote peer 0c4821d9898d42799c5789d4d2da3f63: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:47.486891 8355 raft_consensus.cc:1261] T 897e88ac93484a11829189e027de8bae P de2f076398ee44008622fb5db2a75952 [term 1 FOLLOWER]: Refusing update from remote peer 0c4821d9898d42799c5789d4d2da3f63: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:47.487025 6757 raft_consensus.cc:1261] T 897e88ac93484a11829189e027de8bae P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Refusing update from remote peer 0c4821d9898d42799c5789d4d2da3f63: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:47.487296 12151 consensus_queue.cc:1035] T 897e88ac93484a11829189e027de8bae P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:47.487635 11379 consensus_queue.cc:1035] T 897e88ac93484a11829189e027de8bae P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:47.487941 11379 consensus_queue.cc:1035] T 897e88ac93484a11829189e027de8bae P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:47.489892 11379 raft_consensus.cc:2932] T 897e88ac93484a11829189e027de8bae P 0c4821d9898d42799c5789d4d2da3f63 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 7a1404fe1c1b4b388b821c8261b53104 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } }
I20240503 01:59:47.490360 6757 raft_consensus.cc:2932] T 897e88ac93484a11829189e027de8bae P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 7a1404fe1c1b4b388b821c8261b53104 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } }
I20240503 01:59:47.491546 8486 raft_consensus.cc:2932] T 897e88ac93484a11829189e027de8bae P 7a1404fe1c1b4b388b821c8261b53104 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 7a1404fe1c1b4b388b821c8261b53104 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } }
I20240503 01:59:47.493026 6222 catalog_manager.cc:5484] T 897e88ac93484a11829189e027de8bae P b6b1d5677b8d47788b7968f64b3dac3c reported cstate change: config changed from index 7 to 8, 7a1404fe1c1b4b388b821c8261b53104 (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0c4821d9898d42799c5789d4d2da3f63" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } }
I20240503 01:59:47.493345 8355 raft_consensus.cc:2932] T 897e88ac93484a11829189e027de8bae P de2f076398ee44008622fb5db2a75952 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 7a1404fe1c1b4b388b821c8261b53104 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } }
I20240503 01:59:47.497571 6494 consensus_queue.cc:237] T 897e88ac93484a11829189e027de8bae P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } }
I20240503 01:59:47.498507 8080 raft_consensus.cc:1203] T 21d27cca81e8455e9934e1cc604fc2b2 P 9264f24d028347d98f7e4e2d1aedeb79 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 01:59:47.498785 8355 raft_consensus.cc:1261] T 897e88ac93484a11829189e027de8bae P de2f076398ee44008622fb5db2a75952 [term 1 FOLLOWER]: Refusing update from remote peer 0c4821d9898d42799c5789d4d2da3f63: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 01:59:47.499003 8486 raft_consensus.cc:1261] T 897e88ac93484a11829189e027de8bae P 7a1404fe1c1b4b388b821c8261b53104 [term 1 FOLLOWER]: Refusing update from remote peer 0c4821d9898d42799c5789d4d2da3f63: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 01:59:47.499191 12149 consensus_queue.cc:1035] T 897e88ac93484a11829189e027de8bae P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 01:59:47.499936 12149 consensus_queue.cc:1035] T 897e88ac93484a11829189e027de8bae P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 01:59:47.501292 11379 raft_consensus.cc:2932] T 897e88ac93484a11829189e027de8bae P 0c4821d9898d42799c5789d4d2da3f63 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER b6b1d5677b8d47788b7968f64b3dac3c (127.0.99.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } }
I20240503 01:59:47.501762 8355 raft_consensus.cc:2932] T 897e88ac93484a11829189e027de8bae P de2f076398ee44008622fb5db2a75952 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER b6b1d5677b8d47788b7968f64b3dac3c (127.0.99.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } }
I20240503 01:59:47.502985 6210 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 897e88ac93484a11829189e027de8bae with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:47.504020 9410 raft_consensus.cc:1203] T 120d5dd4f26c42399702141094ce7982 P 2d0fd64aaa6548e09e1ebde581c54658 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 01:59:47.504473 6221 catalog_manager.cc:5484] T 897e88ac93484a11829189e027de8bae P de2f076398ee44008622fb5db2a75952 reported cstate change: config changed from index 8 to 9, VOTER b6b1d5677b8d47788b7968f64b3dac3c (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "0c4821d9898d42799c5789d4d2da3f63" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } }
I20240503 01:59:47.504971 8486 raft_consensus.cc:2932] T 897e88ac93484a11829189e027de8bae P 7a1404fe1c1b4b388b821c8261b53104 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER b6b1d5677b8d47788b7968f64b3dac3c (127.0.99.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } }
I20240503 01:59:47.507261 6738 tablet_service.cc:1514] Processing DeleteTablet for tablet 897e88ac93484a11829189e027de8bae with delete_type TABLET_DATA_TOMBSTONED (TS b6b1d5677b8d47788b7968f64b3dac3c not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:51980
I20240503 01:59:47.507637 12160 tablet_replica.cc:330] T 897e88ac93484a11829189e027de8bae P b6b1d5677b8d47788b7968f64b3dac3c: stopping tablet replica
I20240503 01:59:47.507792 12160 raft_consensus.cc:2229] T 897e88ac93484a11829189e027de8bae P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:47.507995 12160 raft_consensus.cc:2258] T 897e88ac93484a11829189e027de8bae P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:47.508870 12160 ts_tablet_manager.cc:1875] T 897e88ac93484a11829189e027de8bae P b6b1d5677b8d47788b7968f64b3dac3c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:47.511121 12160 ts_tablet_manager.cc:1888] T 897e88ac93484a11829189e027de8bae P b6b1d5677b8d47788b7968f64b3dac3c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 01:59:47.511186 12160 log.cc:1196] T 897e88ac93484a11829189e027de8bae P b6b1d5677b8d47788b7968f64b3dac3c: Deleting WAL directory at /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/897e88ac93484a11829189e027de8bae
I20240503 01:59:47.511690 6209 catalog_manager.cc:4830] TS b6b1d5677b8d47788b7968f64b3dac3c (127.0.99.132:46067): tablet 897e88ac93484a11829189e027de8bae (table rebalance_test_table_9 [id=27f301d87e884061bddb2605833c29a6]) successfully deleted
I20240503 01:59:47.513682 12163 raft_consensus.cc:1050] T a9a74e04ce3745b2b479e96a3f3dc86e P 79163c69f6f04a9abd0208d74e1c61be: attempting to promote NON_VOTER 2d0fd64aaa6548e09e1ebde581c54658 to VOTER
I20240503 01:59:47.514138 12163 consensus_queue.cc:237] T a9a74e04ce3745b2b479e96a3f3dc86e P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } }
I20240503 01:59:47.515538 8486 raft_consensus.cc:1261] T a9a74e04ce3745b2b479e96a3f3dc86e P 7a1404fe1c1b4b388b821c8261b53104 [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:47.515561 6365 raft_consensus.cc:1261] T a9a74e04ce3745b2b479e96a3f3dc86e P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:47.515802 9410 raft_consensus.cc:1261] T a9a74e04ce3745b2b479e96a3f3dc86e P 2d0fd64aaa6548e09e1ebde581c54658 [term 1 LEARNER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:47.515944 12165 consensus_queue.cc:1035] T a9a74e04ce3745b2b479e96a3f3dc86e P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:47.516108 12163 consensus_queue.cc:1035] T a9a74e04ce3745b2b479e96a3f3dc86e P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:47.516252 11781 consensus_queue.cc:1035] T a9a74e04ce3745b2b479e96a3f3dc86e P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:47.518777 12163 raft_consensus.cc:2932] T a9a74e04ce3745b2b479e96a3f3dc86e P 79163c69f6f04a9abd0208d74e1c61be [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2d0fd64aaa6548e09e1ebde581c54658 (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } } }
I20240503 01:59:47.519162 6365 raft_consensus.cc:2932] T a9a74e04ce3745b2b479e96a3f3dc86e P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2d0fd64aaa6548e09e1ebde581c54658 (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } } }
I20240503 01:59:47.520857 8486 raft_consensus.cc:2932] T a9a74e04ce3745b2b479e96a3f3dc86e P 7a1404fe1c1b4b388b821c8261b53104 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2d0fd64aaa6548e09e1ebde581c54658 (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } } }
I20240503 01:59:47.520741 9410 raft_consensus.cc:2932] T a9a74e04ce3745b2b479e96a3f3dc86e P 2d0fd64aaa6548e09e1ebde581c54658 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2d0fd64aaa6548e09e1ebde581c54658 (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } } }
I20240503 01:59:47.521179 6219 catalog_manager.cc:5484] T a9a74e04ce3745b2b479e96a3f3dc86e P 79163c69f6f04a9abd0208d74e1c61be reported cstate change: config changed from index 7 to 8, 2d0fd64aaa6548e09e1ebde581c54658 (127.0.99.137) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "79163c69f6f04a9abd0208d74e1c61be" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:47.524760 6625 consensus_queue.cc:237] T a9a74e04ce3745b2b479e96a3f3dc86e P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } }
I20240503 01:59:47.525903 9410 raft_consensus.cc:1261] T a9a74e04ce3745b2b479e96a3f3dc86e P 2d0fd64aaa6548e09e1ebde581c54658 [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 01:59:47.525933 8486 raft_consensus.cc:1261] T a9a74e04ce3745b2b479e96a3f3dc86e P 7a1404fe1c1b4b388b821c8261b53104 [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 01:59:47.526211 12162 consensus_queue.cc:1035] T a9a74e04ce3745b2b479e96a3f3dc86e P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 01:59:47.526386 12165 consensus_queue.cc:1035] T a9a74e04ce3745b2b479e96a3f3dc86e P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 01:59:47.527905 12162 raft_consensus.cc:2932] T a9a74e04ce3745b2b479e96a3f3dc86e P 79163c69f6f04a9abd0208d74e1c61be [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 4127249ba42644388d225d614c935a25 (127.0.99.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } } }
I20240503 01:59:47.528235 8486 raft_consensus.cc:2932] T a9a74e04ce3745b2b479e96a3f3dc86e P 7a1404fe1c1b4b388b821c8261b53104 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 4127249ba42644388d225d614c935a25 (127.0.99.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } } }
I20240503 01:59:47.528455 9410 raft_consensus.cc:2932] T a9a74e04ce3745b2b479e96a3f3dc86e P 2d0fd64aaa6548e09e1ebde581c54658 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 4127249ba42644388d225d614c935a25 (127.0.99.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } } }
I20240503 01:59:47.529595 6208 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a9a74e04ce3745b2b479e96a3f3dc86e with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:47.530475 6222 catalog_manager.cc:5484] T a9a74e04ce3745b2b479e96a3f3dc86e P 79163c69f6f04a9abd0208d74e1c61be reported cstate change: config changed from index 8 to 9, VOTER 4127249ba42644388d225d614c935a25 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "79163c69f6f04a9abd0208d74e1c61be" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:47.532817 6344 tablet_service.cc:1514] Processing DeleteTablet for tablet a9a74e04ce3745b2b479e96a3f3dc86e with delete_type TABLET_DATA_TOMBSTONED (TS 4127249ba42644388d225d614c935a25 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:57398
I20240503 01:59:47.533274 12169 tablet_replica.cc:330] T a9a74e04ce3745b2b479e96a3f3dc86e P 4127249ba42644388d225d614c935a25: stopping tablet replica
I20240503 01:59:47.533432 12169 raft_consensus.cc:2229] T a9a74e04ce3745b2b479e96a3f3dc86e P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:47.533665 12169 raft_consensus.cc:2258] T a9a74e04ce3745b2b479e96a3f3dc86e P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:47.534297 12169 ts_tablet_manager.cc:1875] T a9a74e04ce3745b2b479e96a3f3dc86e P 4127249ba42644388d225d614c935a25: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:47.536116 12169 ts_tablet_manager.cc:1888] T a9a74e04ce3745b2b479e96a3f3dc86e P 4127249ba42644388d225d614c935a25: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 01:59:47.536181 12169 log.cc:1196] T a9a74e04ce3745b2b479e96a3f3dc86e P 4127249ba42644388d225d614c935a25: Deleting WAL directory at /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/a9a74e04ce3745b2b479e96a3f3dc86e
I20240503 01:59:47.536660 6208 catalog_manager.cc:4830] TS 4127249ba42644388d225d614c935a25 (127.0.99.129:35613): tablet a9a74e04ce3745b2b479e96a3f3dc86e (table rebalance_test_table_2 [id=4a248d80763944458620bf40775e2098]) successfully deleted
I20240503 01:59:47.595693 11379 raft_consensus.cc:1050] T 2ac8004211e64c3481e75377092fa93b P 0c4821d9898d42799c5789d4d2da3f63: attempting to promote NON_VOTER 2d0fd64aaa6548e09e1ebde581c54658 to VOTER
I20240503 01:59:47.596159 11379 consensus_queue.cc:237] T 2ac8004211e64c3481e75377092fa93b P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } }
I20240503 01:59:47.597649 9410 raft_consensus.cc:1261] T 2ac8004211e64c3481e75377092fa93b P 2d0fd64aaa6548e09e1ebde581c54658 [term 1 LEARNER]: Refusing update from remote peer 0c4821d9898d42799c5789d4d2da3f63: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:47.597854 8080 raft_consensus.cc:1261] T 2ac8004211e64c3481e75377092fa93b P 9264f24d028347d98f7e4e2d1aedeb79 [term 1 FOLLOWER]: Refusing update from remote peer 0c4821d9898d42799c5789d4d2da3f63: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:47.598021 6365 raft_consensus.cc:1261] T 2ac8004211e64c3481e75377092fa93b P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Refusing update from remote peer 0c4821d9898d42799c5789d4d2da3f63: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:47.598438 12151 consensus_queue.cc:1035] T 2ac8004211e64c3481e75377092fa93b P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:47.598634 12148 consensus_queue.cc:1035] T 2ac8004211e64c3481e75377092fa93b P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:47.599162 12148 consensus_queue.cc:1035] T 2ac8004211e64c3481e75377092fa93b P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:47.601387 12149 raft_consensus.cc:2932] T 2ac8004211e64c3481e75377092fa93b P 0c4821d9898d42799c5789d4d2da3f63 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2d0fd64aaa6548e09e1ebde581c54658 (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } } }
I20240503 01:59:47.601759 6365 raft_consensus.cc:2932] T 2ac8004211e64c3481e75377092fa93b P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2d0fd64aaa6548e09e1ebde581c54658 (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } } }
I20240503 01:59:47.603070 9410 raft_consensus.cc:2932] T 2ac8004211e64c3481e75377092fa93b P 2d0fd64aaa6548e09e1ebde581c54658 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2d0fd64aaa6548e09e1ebde581c54658 (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } } }
I20240503 01:59:47.603540 8080 raft_consensus.cc:2932] T 2ac8004211e64c3481e75377092fa93b P 9264f24d028347d98f7e4e2d1aedeb79 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2d0fd64aaa6548e09e1ebde581c54658 (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } } }
I20240503 01:59:47.604251 6220 catalog_manager.cc:5484] T 2ac8004211e64c3481e75377092fa93b P 4127249ba42644388d225d614c935a25 reported cstate change: config changed from index 7 to 8, 2d0fd64aaa6548e09e1ebde581c54658 (127.0.99.137) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0c4821d9898d42799c5789d4d2da3f63" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } } }
I20240503 01:59:47.607993 6494 consensus_queue.cc:237] T 2ac8004211e64c3481e75377092fa93b P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } }
I20240503 01:59:47.609248 8080 raft_consensus.cc:1261] T 2ac8004211e64c3481e75377092fa93b P 9264f24d028347d98f7e4e2d1aedeb79 [term 1 FOLLOWER]: Refusing update from remote peer 0c4821d9898d42799c5789d4d2da3f63: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 01:59:47.609490 9410 raft_consensus.cc:1261] T 2ac8004211e64c3481e75377092fa93b P 2d0fd64aaa6548e09e1ebde581c54658 [term 1 FOLLOWER]: Refusing update from remote peer 0c4821d9898d42799c5789d4d2da3f63: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 01:59:47.609742 12149 consensus_queue.cc:1035] T 2ac8004211e64c3481e75377092fa93b P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 01:59:47.609900 12151 consensus_queue.cc:1035] T 2ac8004211e64c3481e75377092fa93b P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 01:59:47.611373 12151 raft_consensus.cc:2932] T 2ac8004211e64c3481e75377092fa93b P 0c4821d9898d42799c5789d4d2da3f63 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 4127249ba42644388d225d614c935a25 (127.0.99.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } } }
I20240503 01:59:47.611817 8080 raft_consensus.cc:2932] T 2ac8004211e64c3481e75377092fa93b P 9264f24d028347d98f7e4e2d1aedeb79 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 4127249ba42644388d225d614c935a25 (127.0.99.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } } }
I20240503 01:59:47.612015 9410 raft_consensus.cc:2932] T 2ac8004211e64c3481e75377092fa93b P 2d0fd64aaa6548e09e1ebde581c54658 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 4127249ba42644388d225d614c935a25 (127.0.99.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } } }
I20240503 01:59:47.613518 6210 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2ac8004211e64c3481e75377092fa93b with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:47.614117 6219 catalog_manager.cc:5484] T 2ac8004211e64c3481e75377092fa93b P 9264f24d028347d98f7e4e2d1aedeb79 reported cstate change: config changed from index 8 to 9, VOTER 4127249ba42644388d225d614c935a25 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "0c4821d9898d42799c5789d4d2da3f63" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } } }
I20240503 01:59:47.617406 6344 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ac8004211e64c3481e75377092fa93b with delete_type TABLET_DATA_TOMBSTONED (TS 4127249ba42644388d225d614c935a25 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:57398
I20240503 01:59:47.617774 12169 tablet_replica.cc:330] T 2ac8004211e64c3481e75377092fa93b P 4127249ba42644388d225d614c935a25: stopping tablet replica
I20240503 01:59:47.617941 12169 raft_consensus.cc:2229] T 2ac8004211e64c3481e75377092fa93b P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:47.618147 12169 raft_consensus.cc:2258] T 2ac8004211e64c3481e75377092fa93b P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:47.618849 12169 ts_tablet_manager.cc:1875] T 2ac8004211e64c3481e75377092fa93b P 4127249ba42644388d225d614c935a25: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:47.620556 12169 ts_tablet_manager.cc:1888] T 2ac8004211e64c3481e75377092fa93b P 4127249ba42644388d225d614c935a25: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 01:59:47.620666 12169 log.cc:1196] T 2ac8004211e64c3481e75377092fa93b P 4127249ba42644388d225d614c935a25: Deleting WAL directory at /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/2ac8004211e64c3481e75377092fa93b
I20240503 01:59:47.621162 6208 catalog_manager.cc:4830] TS 4127249ba42644388d225d614c935a25 (127.0.99.129:35613): tablet 2ac8004211e64c3481e75377092fa93b (table rebalance_test_table_4 [id=d52e5f2a44ec4f12b7ef6d85bd14d4ac]) successfully deleted
I20240503 01:59:47.659298 6625 consensus_queue.cc:237] T 0bdcea4e59ec4e2da63b9316f9d718be P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: true } }
I20240503 01:59:47.660796 6757 raft_consensus.cc:1261] T 0bdcea4e59ec4e2da63b9316f9d718be P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:47.660877 8355 raft_consensus.cc:1261] T 0bdcea4e59ec4e2da63b9316f9d718be P de2f076398ee44008622fb5db2a75952 [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 01:59:47.660929 6562 consensus_peers.cc:485] T 0bdcea4e59ec4e2da63b9316f9d718be P 79163c69f6f04a9abd0208d74e1c61be -> Peer 2d0fd64aaa6548e09e1ebde581c54658 (127.0.99.137:40613): Couldn't send request to peer 2d0fd64aaa6548e09e1ebde581c54658. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0bdcea4e59ec4e2da63b9316f9d718be. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:47.661108 11781 consensus_queue.cc:1035] T 0bdcea4e59ec4e2da63b9316f9d718be P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:47.661264 12162 consensus_queue.cc:1035] T 0bdcea4e59ec4e2da63b9316f9d718be P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:47.662936 12165 raft_consensus.cc:2932] T 0bdcea4e59ec4e2da63b9316f9d718be P 79163c69f6f04a9abd0208d74e1c61be [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2d0fd64aaa6548e09e1ebde581c54658 (127.0.99.137) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: true } } }
I20240503 01:59:47.663506 6757 raft_consensus.cc:2932] T 0bdcea4e59ec4e2da63b9316f9d718be P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2d0fd64aaa6548e09e1ebde581c54658 (127.0.99.137) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: true } } }
I20240503 01:59:47.663553 8355 raft_consensus.cc:2932] T 0bdcea4e59ec4e2da63b9316f9d718be P de2f076398ee44008622fb5db2a75952 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2d0fd64aaa6548e09e1ebde581c54658 (127.0.99.137) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: true } } }
I20240503 01:59:47.666049 6220 catalog_manager.cc:5484] T 0bdcea4e59ec4e2da63b9316f9d718be P 79163c69f6f04a9abd0208d74e1c61be reported cstate change: config changed from index 6 to 7, NON_VOTER 2d0fd64aaa6548e09e1ebde581c54658 (127.0.99.137) added. New cstate: current_term: 1 leader_uuid: "79163c69f6f04a9abd0208d74e1c61be" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:47.670974 11781 raft_consensus.cc:1050] T d6ee8175874c4063986033b0a80ccb82 P 79163c69f6f04a9abd0208d74e1c61be: attempting to promote NON_VOTER 65d4632bcdd54ba596acdbbe1b5ef8af to VOTER
I20240503 01:59:47.671295 11781 consensus_queue.cc:237] T d6ee8175874c4063986033b0a80ccb82 P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } }
I20240503 01:59:47.672817 6757 raft_consensus.cc:1261] T d6ee8175874c4063986033b0a80ccb82 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:47.672892 8486 raft_consensus.cc:1261] T d6ee8175874c4063986033b0a80ccb82 P 7a1404fe1c1b4b388b821c8261b53104 [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:47.673274 8223 raft_consensus.cc:1261] T d6ee8175874c4063986033b0a80ccb82 P 65d4632bcdd54ba596acdbbe1b5ef8af [term 1 LEARNER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:47.673689 12163 consensus_queue.cc:1035] T d6ee8175874c4063986033b0a80ccb82 P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:47.673883 12162 consensus_queue.cc:1035] T d6ee8175874c4063986033b0a80ccb82 P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:47.674103 12165 consensus_queue.cc:1035] T d6ee8175874c4063986033b0a80ccb82 P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:47.676263 11781 raft_consensus.cc:2932] T d6ee8175874c4063986033b0a80ccb82 P 79163c69f6f04a9abd0208d74e1c61be [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 65d4632bcdd54ba596acdbbe1b5ef8af (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } }
I20240503 01:59:47.677029 8223 raft_consensus.cc:2932] T d6ee8175874c4063986033b0a80ccb82 P 65d4632bcdd54ba596acdbbe1b5ef8af [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 65d4632bcdd54ba596acdbbe1b5ef8af (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } }
I20240503 01:59:47.678025 8486 raft_consensus.cc:2932] T d6ee8175874c4063986033b0a80ccb82 P 7a1404fe1c1b4b388b821c8261b53104 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 65d4632bcdd54ba596acdbbe1b5ef8af (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } }
I20240503 01:59:47.679570 6757 raft_consensus.cc:2932] T d6ee8175874c4063986033b0a80ccb82 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 65d4632bcdd54ba596acdbbe1b5ef8af (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } }
I20240503 01:59:47.679987 6221 catalog_manager.cc:5484] T d6ee8175874c4063986033b0a80ccb82 P 65d4632bcdd54ba596acdbbe1b5ef8af reported cstate change: config changed from index 7 to 8, 65d4632bcdd54ba596acdbbe1b5ef8af (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "79163c69f6f04a9abd0208d74e1c61be" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } }
I20240503 01:59:47.682868 6625 consensus_queue.cc:237] T d6ee8175874c4063986033b0a80ccb82 P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } }
I20240503 01:59:47.683879 8486 raft_consensus.cc:1261] T d6ee8175874c4063986033b0a80ccb82 P 7a1404fe1c1b4b388b821c8261b53104 [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 01:59:47.683879 8223 raft_consensus.cc:1261] T d6ee8175874c4063986033b0a80ccb82 P 65d4632bcdd54ba596acdbbe1b5ef8af [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 01:59:47.684240 11781 consensus_queue.cc:1035] T d6ee8175874c4063986033b0a80ccb82 P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 01:59:47.684533 12165 consensus_queue.cc:1035] T d6ee8175874c4063986033b0a80ccb82 P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 01:59:47.685763 12162 raft_consensus.cc:2932] T d6ee8175874c4063986033b0a80ccb82 P 79163c69f6f04a9abd0208d74e1c61be [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER b6b1d5677b8d47788b7968f64b3dac3c (127.0.99.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } }
I20240503 01:59:47.686846 8223 raft_consensus.cc:2932] T d6ee8175874c4063986033b0a80ccb82 P 65d4632bcdd54ba596acdbbe1b5ef8af [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER b6b1d5677b8d47788b7968f64b3dac3c (127.0.99.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } }
I20240503 01:59:47.687044 6208 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d6ee8175874c4063986033b0a80ccb82 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:47.687379 8486 raft_consensus.cc:2932] T d6ee8175874c4063986033b0a80ccb82 P 7a1404fe1c1b4b388b821c8261b53104 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER b6b1d5677b8d47788b7968f64b3dac3c (127.0.99.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } }
I20240503 01:59:47.690457 6219 catalog_manager.cc:5484] T d6ee8175874c4063986033b0a80ccb82 P 79163c69f6f04a9abd0208d74e1c61be reported cstate change: config changed from index 8 to 9, VOTER b6b1d5677b8d47788b7968f64b3dac3c (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "79163c69f6f04a9abd0208d74e1c61be" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:47.692871 6738 tablet_service.cc:1514] Processing DeleteTablet for tablet d6ee8175874c4063986033b0a80ccb82 with delete_type TABLET_DATA_TOMBSTONED (TS b6b1d5677b8d47788b7968f64b3dac3c not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:51980
I20240503 01:59:47.693102 12160 tablet_replica.cc:330] T d6ee8175874c4063986033b0a80ccb82 P b6b1d5677b8d47788b7968f64b3dac3c: stopping tablet replica
I20240503 01:59:47.693241 12160 raft_consensus.cc:2229] T d6ee8175874c4063986033b0a80ccb82 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:47.693383 12160 raft_consensus.cc:2258] T d6ee8175874c4063986033b0a80ccb82 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:47.695290 12160 ts_tablet_manager.cc:1875] T d6ee8175874c4063986033b0a80ccb82 P b6b1d5677b8d47788b7968f64b3dac3c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:47.696955 12160 ts_tablet_manager.cc:1888] T d6ee8175874c4063986033b0a80ccb82 P b6b1d5677b8d47788b7968f64b3dac3c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 01:59:47.697016 12160 log.cc:1196] T d6ee8175874c4063986033b0a80ccb82 P b6b1d5677b8d47788b7968f64b3dac3c: Deleting WAL directory at /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/d6ee8175874c4063986033b0a80ccb82
I20240503 01:59:47.697453 6209 catalog_manager.cc:4830] TS b6b1d5677b8d47788b7968f64b3dac3c (127.0.99.132:46067): tablet d6ee8175874c4063986033b0a80ccb82 (table rebalance_test_table_1 [id=91bdb83bc45b4b61b094a0180eacf8e3]) successfully deleted
I20240503 01:59:47.744632 11843 raft_consensus.cc:1050] T 82cfe99cd6fe4a82a7e35298ec634b8b P 4127249ba42644388d225d614c935a25: attempting to promote NON_VOTER 65d4632bcdd54ba596acdbbe1b5ef8af to VOTER
I20240503 01:59:47.745090 11843 consensus_queue.cc:237] T 82cfe99cd6fe4a82a7e35298ec634b8b P 4127249ba42644388d225d614c935a25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } }
I20240503 01:59:47.746573 8223 raft_consensus.cc:1261] T 82cfe99cd6fe4a82a7e35298ec634b8b P 65d4632bcdd54ba596acdbbe1b5ef8af [term 1 LEARNER]: Refusing update from remote peer 4127249ba42644388d225d614c935a25: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:47.746683 6494 raft_consensus.cc:1261] T 82cfe99cd6fe4a82a7e35298ec634b8b P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Refusing update from remote peer 4127249ba42644388d225d614c935a25: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:47.746779 8080 raft_consensus.cc:1261] T 82cfe99cd6fe4a82a7e35298ec634b8b P 9264f24d028347d98f7e4e2d1aedeb79 [term 1 FOLLOWER]: Refusing update from remote peer 4127249ba42644388d225d614c935a25: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:47.747056 12318 consensus_queue.cc:1035] T 82cfe99cd6fe4a82a7e35298ec634b8b P 4127249ba42644388d225d614c935a25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:47.747241 12161 consensus_queue.cc:1035] T 82cfe99cd6fe4a82a7e35298ec634b8b P 4127249ba42644388d225d614c935a25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:47.747609 12315 consensus_queue.cc:1035] T 82cfe99cd6fe4a82a7e35298ec634b8b P 4127249ba42644388d225d614c935a25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:47.750329 12161 raft_consensus.cc:2932] T 82cfe99cd6fe4a82a7e35298ec634b8b P 4127249ba42644388d225d614c935a25 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 65d4632bcdd54ba596acdbbe1b5ef8af (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } }
I20240503 01:59:47.750881 8223 raft_consensus.cc:2932] T 82cfe99cd6fe4a82a7e35298ec634b8b P 65d4632bcdd54ba596acdbbe1b5ef8af [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 65d4632bcdd54ba596acdbbe1b5ef8af (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } }
I20240503 01:59:47.751482 8080 raft_consensus.cc:2932] T 82cfe99cd6fe4a82a7e35298ec634b8b P 9264f24d028347d98f7e4e2d1aedeb79 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 65d4632bcdd54ba596acdbbe1b5ef8af (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } }
I20240503 01:59:47.752334 6494 raft_consensus.cc:2932] T 82cfe99cd6fe4a82a7e35298ec634b8b P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 65d4632bcdd54ba596acdbbe1b5ef8af (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } }
I20240503 01:59:47.753912 6221 catalog_manager.cc:5484] T 82cfe99cd6fe4a82a7e35298ec634b8b P 4127249ba42644388d225d614c935a25 reported cstate change: config changed from index 7 to 8, 65d4632bcdd54ba596acdbbe1b5ef8af (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4127249ba42644388d225d614c935a25" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:47.755751 9410 raft_consensus.cc:1203] T f11a8034585a4d57be0e9b1e5e8bcd85 P 2d0fd64aaa6548e09e1ebde581c54658 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 01:59:47.757663 6365 consensus_queue.cc:237] T 82cfe99cd6fe4a82a7e35298ec634b8b P 4127249ba42644388d225d614c935a25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } }
I20240503 01:59:47.758873 8223 raft_consensus.cc:1261] T 82cfe99cd6fe4a82a7e35298ec634b8b P 65d4632bcdd54ba596acdbbe1b5ef8af [term 1 FOLLOWER]: Refusing update from remote peer 4127249ba42644388d225d614c935a25: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 01:59:47.759151 8080 raft_consensus.cc:1261] T 82cfe99cd6fe4a82a7e35298ec634b8b P 9264f24d028347d98f7e4e2d1aedeb79 [term 1 FOLLOWER]: Refusing update from remote peer 4127249ba42644388d225d614c935a25: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 01:59:47.759253 12315 consensus_queue.cc:1035] T 82cfe99cd6fe4a82a7e35298ec634b8b P 4127249ba42644388d225d614c935a25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 01:59:47.759562 12318 consensus_queue.cc:1035] T 82cfe99cd6fe4a82a7e35298ec634b8b P 4127249ba42644388d225d614c935a25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 01:59:47.761328 12318 raft_consensus.cc:2932] T 82cfe99cd6fe4a82a7e35298ec634b8b P 4127249ba42644388d225d614c935a25 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } }
I20240503 01:59:47.762699 6208 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 82cfe99cd6fe4a82a7e35298ec634b8b with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:47.763608 8080 raft_consensus.cc:2932] T 82cfe99cd6fe4a82a7e35298ec634b8b P 9264f24d028347d98f7e4e2d1aedeb79 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } }
I20240503 01:59:47.763873 6219 catalog_manager.cc:5484] T 82cfe99cd6fe4a82a7e35298ec634b8b P 4127249ba42644388d225d614c935a25 reported cstate change: config changed from index 8 to 9, VOTER 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "4127249ba42644388d225d614c935a25" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:47.765442 8223 raft_consensus.cc:2932] T 82cfe99cd6fe4a82a7e35298ec634b8b P 65d4632bcdd54ba596acdbbe1b5ef8af [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } }
I20240503 01:59:47.767096 6476 tablet_service.cc:1514] Processing DeleteTablet for tablet 82cfe99cd6fe4a82a7e35298ec634b8b with delete_type TABLET_DATA_TOMBSTONED (TS 0c4821d9898d42799c5789d4d2da3f63 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:39850
I20240503 01:59:47.767506 12332 tablet_replica.cc:330] T 82cfe99cd6fe4a82a7e35298ec634b8b P 0c4821d9898d42799c5789d4d2da3f63: stopping tablet replica
I20240503 01:59:47.767666 12332 raft_consensus.cc:2229] T 82cfe99cd6fe4a82a7e35298ec634b8b P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:47.767902 12332 raft_consensus.cc:2258] T 82cfe99cd6fe4a82a7e35298ec634b8b P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:47.770059 12332 ts_tablet_manager.cc:1875] T 82cfe99cd6fe4a82a7e35298ec634b8b P 0c4821d9898d42799c5789d4d2da3f63: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:47.772150 12332 ts_tablet_manager.cc:1888] T 82cfe99cd6fe4a82a7e35298ec634b8b P 0c4821d9898d42799c5789d4d2da3f63: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 01:59:47.772233 12332 log.cc:1196] T 82cfe99cd6fe4a82a7e35298ec634b8b P 0c4821d9898d42799c5789d4d2da3f63: Deleting WAL directory at /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/82cfe99cd6fe4a82a7e35298ec634b8b
I20240503 01:59:47.772599 6625 consensus_queue.cc:237] T c3337c95b354418faf37684e7440b1c7 P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: true } }
I20240503 01:59:47.772766 6210 catalog_manager.cc:4830] TS 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130:34189): tablet 82cfe99cd6fe4a82a7e35298ec634b8b (table rebalance_test_table_6 [id=80cb1486f22e4bee930434fdfc3463d2]) successfully deleted
I20240503 01:59:47.774106 8355 raft_consensus.cc:1261] T c3337c95b354418faf37684e7440b1c7 P de2f076398ee44008622fb5db2a75952 [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 01:59:47.774262 6562 consensus_peers.cc:485] T c3337c95b354418faf37684e7440b1c7 P 79163c69f6f04a9abd0208d74e1c61be -> Peer 2d0fd64aaa6548e09e1ebde581c54658 (127.0.99.137:40613): Couldn't send request to peer 2d0fd64aaa6548e09e1ebde581c54658. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c3337c95b354418faf37684e7440b1c7. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:47.774442 6757 raft_consensus.cc:1261] T c3337c95b354418faf37684e7440b1c7 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:47.774544 12162 consensus_queue.cc:1035] T c3337c95b354418faf37684e7440b1c7 P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:47.774830 11781 consensus_queue.cc:1035] T c3337c95b354418faf37684e7440b1c7 P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:47.776345 12162 raft_consensus.cc:2932] T c3337c95b354418faf37684e7440b1c7 P 79163c69f6f04a9abd0208d74e1c61be [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2d0fd64aaa6548e09e1ebde581c54658 (127.0.99.137) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: true } } }
I20240503 01:59:47.777225 6757 raft_consensus.cc:2932] T c3337c95b354418faf37684e7440b1c7 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2d0fd64aaa6548e09e1ebde581c54658 (127.0.99.137) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: true } } }
I20240503 01:59:47.777814 8355 raft_consensus.cc:2932] T c3337c95b354418faf37684e7440b1c7 P de2f076398ee44008622fb5db2a75952 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2d0fd64aaa6548e09e1ebde581c54658 (127.0.99.137) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: true } } }
I20240503 01:59:47.780625 6214 catalog_manager.cc:5484] T c3337c95b354418faf37684e7440b1c7 P de2f076398ee44008622fb5db2a75952 reported cstate change: config changed from index 6 to 7, NON_VOTER 2d0fd64aaa6548e09e1ebde581c54658 (127.0.99.137) added. New cstate: current_term: 1 leader_uuid: "79163c69f6f04a9abd0208d74e1c61be" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: true } } }
I20240503 01:59:47.784894 8223 raft_consensus.cc:1203] T 1d392e08c15a4d4098a350287b3163ef P 65d4632bcdd54ba596acdbbe1b5ef8af [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 01:59:47.912041 12165 raft_consensus.cc:1050] T 5b599bd1fea6409fbe485c7a35c0891c P 79163c69f6f04a9abd0208d74e1c61be: attempting to promote NON_VOTER 65d4632bcdd54ba596acdbbe1b5ef8af to VOTER
I20240503 01:59:47.912492 12165 consensus_queue.cc:237] T 5b599bd1fea6409fbe485c7a35c0891c P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } }
I20240503 01:59:47.913918 8223 raft_consensus.cc:1261] T 5b599bd1fea6409fbe485c7a35c0891c P 65d4632bcdd54ba596acdbbe1b5ef8af [term 1 LEARNER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:47.913918 6494 raft_consensus.cc:1261] T 5b599bd1fea6409fbe485c7a35c0891c P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:47.914013 8486 raft_consensus.cc:1261] T 5b599bd1fea6409fbe485c7a35c0891c P 7a1404fe1c1b4b388b821c8261b53104 [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:47.914443 12163 consensus_queue.cc:1035] T 5b599bd1fea6409fbe485c7a35c0891c P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:47.914633 11781 consensus_queue.cc:1035] T 5b599bd1fea6409fbe485c7a35c0891c P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:47.914883 12165 consensus_queue.cc:1035] T 5b599bd1fea6409fbe485c7a35c0891c P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:47.917217 12163 raft_consensus.cc:2932] T 5b599bd1fea6409fbe485c7a35c0891c P 79163c69f6f04a9abd0208d74e1c61be [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 65d4632bcdd54ba596acdbbe1b5ef8af (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } }
I20240503 01:59:47.917734 8223 raft_consensus.cc:2932] T 5b599bd1fea6409fbe485c7a35c0891c P 65d4632bcdd54ba596acdbbe1b5ef8af [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 65d4632bcdd54ba596acdbbe1b5ef8af (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } }
I20240503 01:59:47.917793 6494 raft_consensus.cc:2932] T 5b599bd1fea6409fbe485c7a35c0891c P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 65d4632bcdd54ba596acdbbe1b5ef8af (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } }
I20240503 01:59:47.919963 8486 raft_consensus.cc:2932] T 5b599bd1fea6409fbe485c7a35c0891c P 7a1404fe1c1b4b388b821c8261b53104 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 65d4632bcdd54ba596acdbbe1b5ef8af (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } }
I20240503 01:59:47.920433 6214 catalog_manager.cc:5484] T 5b599bd1fea6409fbe485c7a35c0891c P 79163c69f6f04a9abd0208d74e1c61be reported cstate change: config changed from index 7 to 8, 65d4632bcdd54ba596acdbbe1b5ef8af (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "79163c69f6f04a9abd0208d74e1c61be" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:47.923326 6625 consensus_queue.cc:237] T 5b599bd1fea6409fbe485c7a35c0891c P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } }
I20240503 01:59:47.924401 8486 raft_consensus.cc:1261] T 5b599bd1fea6409fbe485c7a35c0891c P 7a1404fe1c1b4b388b821c8261b53104 [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 01:59:47.924520 8223 raft_consensus.cc:1261] T 5b599bd1fea6409fbe485c7a35c0891c P 65d4632bcdd54ba596acdbbe1b5ef8af [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 01:59:47.924841 12163 consensus_queue.cc:1035] T 5b599bd1fea6409fbe485c7a35c0891c P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 01:59:47.925139 12163 consensus_queue.cc:1035] T 5b599bd1fea6409fbe485c7a35c0891c P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 01:59:47.926573 12165 raft_consensus.cc:2932] T 5b599bd1fea6409fbe485c7a35c0891c P 79163c69f6f04a9abd0208d74e1c61be [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } }
I20240503 01:59:47.927162 8486 raft_consensus.cc:2932] T 5b599bd1fea6409fbe485c7a35c0891c P 7a1404fe1c1b4b388b821c8261b53104 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } }
I20240503 01:59:47.928499 6208 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5b599bd1fea6409fbe485c7a35c0891c with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:47.928915 8223 raft_consensus.cc:2932] T 5b599bd1fea6409fbe485c7a35c0891c P 65d4632bcdd54ba596acdbbe1b5ef8af [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } }
I20240503 01:59:47.929526 6220 catalog_manager.cc:5484] T 5b599bd1fea6409fbe485c7a35c0891c P 79163c69f6f04a9abd0208d74e1c61be reported cstate change: config changed from index 8 to 9, VOTER 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "79163c69f6f04a9abd0208d74e1c61be" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:47.930881 12162 raft_consensus.cc:1050] T 21d27cca81e8455e9934e1cc604fc2b2 P 79163c69f6f04a9abd0208d74e1c61be: attempting to promote NON_VOTER 9264f24d028347d98f7e4e2d1aedeb79 to VOTER
I20240503 01:59:47.931229 12162 consensus_queue.cc:237] T 21d27cca81e8455e9934e1cc604fc2b2 P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } }
I20240503 01:59:47.932371 6476 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b599bd1fea6409fbe485c7a35c0891c with delete_type TABLET_DATA_TOMBSTONED (TS 0c4821d9898d42799c5789d4d2da3f63 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:39850
I20240503 01:59:47.932591 12332 tablet_replica.cc:330] T 5b599bd1fea6409fbe485c7a35c0891c P 0c4821d9898d42799c5789d4d2da3f63: stopping tablet replica
I20240503 01:59:47.932739 6365 raft_consensus.cc:1261] T 21d27cca81e8455e9934e1cc604fc2b2 P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:47.932786 12332 raft_consensus.cc:2229] T 5b599bd1fea6409fbe485c7a35c0891c P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:47.933012 12332 raft_consensus.cc:2258] T 5b599bd1fea6409fbe485c7a35c0891c P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:47.933234 11781 consensus_queue.cc:1035] T 21d27cca81e8455e9934e1cc604fc2b2 P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:47.934180 12332 ts_tablet_manager.cc:1875] T 5b599bd1fea6409fbe485c7a35c0891c P 0c4821d9898d42799c5789d4d2da3f63: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:47.935168 12148 raft_consensus.cc:1050] T ccb34a4b52244d90850ab04e21299f55 P 0c4821d9898d42799c5789d4d2da3f63: attempting to promote NON_VOTER 9264f24d028347d98f7e4e2d1aedeb79 to VOTER
I20240503 01:59:47.935590 12148 consensus_queue.cc:237] T ccb34a4b52244d90850ab04e21299f55 P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } }
I20240503 01:59:47.935721 8080 raft_consensus.cc:1261] T 21d27cca81e8455e9934e1cc604fc2b2 P 9264f24d028347d98f7e4e2d1aedeb79 [term 1 LEARNER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:47.936167 11781 consensus_queue.cc:1035] T 21d27cca81e8455e9934e1cc604fc2b2 P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:47.937104 8080 raft_consensus.cc:1261] T ccb34a4b52244d90850ab04e21299f55 P 9264f24d028347d98f7e4e2d1aedeb79 [term 1 LEARNER]: Refusing update from remote peer 0c4821d9898d42799c5789d4d2da3f63: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:47.937391 8485 raft_consensus.cc:1261] T ccb34a4b52244d90850ab04e21299f55 P 7a1404fe1c1b4b388b821c8261b53104 [term 1 FOLLOWER]: Refusing update from remote peer 0c4821d9898d42799c5789d4d2da3f63: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:47.937475 12151 consensus_queue.cc:1035] T ccb34a4b52244d90850ab04e21299f55 P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:47.937634 6625 raft_consensus.cc:1261] T ccb34a4b52244d90850ab04e21299f55 P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Refusing update from remote peer 0c4821d9898d42799c5789d4d2da3f63: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:47.937897 12332 ts_tablet_manager.cc:1888] T 5b599bd1fea6409fbe485c7a35c0891c P 0c4821d9898d42799c5789d4d2da3f63: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 01:59:47.937984 12151 consensus_queue.cc:1035] T ccb34a4b52244d90850ab04e21299f55 P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:47.938177 12332 log.cc:1196] T 5b599bd1fea6409fbe485c7a35c0891c P 0c4821d9898d42799c5789d4d2da3f63: Deleting WAL directory at /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/5b599bd1fea6409fbe485c7a35c0891c
I20240503 01:59:47.938773 8486 raft_consensus.cc:1261] T 21d27cca81e8455e9934e1cc604fc2b2 P 7a1404fe1c1b4b388b821c8261b53104 [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:47.938798 12149 consensus_queue.cc:1035] T ccb34a4b52244d90850ab04e21299f55 P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:47.939122 12163 consensus_queue.cc:1035] T 21d27cca81e8455e9934e1cc604fc2b2 P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:47.939563 6210 catalog_manager.cc:4830] TS 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130:34189): tablet 5b599bd1fea6409fbe485c7a35c0891c (table rebalance_test_table_4 [id=d52e5f2a44ec4f12b7ef6d85bd14d4ac]) successfully deleted
I20240503 01:59:47.939646 12163 raft_consensus.cc:2932] T 21d27cca81e8455e9934e1cc604fc2b2 P 79163c69f6f04a9abd0208d74e1c61be [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9264f24d028347d98f7e4e2d1aedeb79 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } }
I20240503 01:59:47.939985 8080 raft_consensus.cc:2932] T 21d27cca81e8455e9934e1cc604fc2b2 P 9264f24d028347d98f7e4e2d1aedeb79 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9264f24d028347d98f7e4e2d1aedeb79 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } }
I20240503 01:59:47.941565 12149 raft_consensus.cc:2932] T ccb34a4b52244d90850ab04e21299f55 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9264f24d028347d98f7e4e2d1aedeb79 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } }
I20240503 01:59:47.943003 6214 catalog_manager.cc:5484] T 21d27cca81e8455e9934e1cc604fc2b2 P 79163c69f6f04a9abd0208d74e1c61be reported cstate change: config changed from index 7 to 8, 9264f24d028347d98f7e4e2d1aedeb79 (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "79163c69f6f04a9abd0208d74e1c61be" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:47.943349 6365 raft_consensus.cc:2932] T 21d27cca81e8455e9934e1cc604fc2b2 P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9264f24d028347d98f7e4e2d1aedeb79 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } }
I20240503 01:59:47.943660 8080 raft_consensus.cc:2932] T ccb34a4b52244d90850ab04e21299f55 P 9264f24d028347d98f7e4e2d1aedeb79 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9264f24d028347d98f7e4e2d1aedeb79 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } }
I20240503 01:59:47.945108 6625 raft_consensus.cc:2932] T ccb34a4b52244d90850ab04e21299f55 P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9264f24d028347d98f7e4e2d1aedeb79 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } }
I20240503 01:59:47.946465 8486 raft_consensus.cc:2932] T ccb34a4b52244d90850ab04e21299f55 P 7a1404fe1c1b4b388b821c8261b53104 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9264f24d028347d98f7e4e2d1aedeb79 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } }
I20240503 01:59:47.947024 6222 catalog_manager.cc:5484] T ccb34a4b52244d90850ab04e21299f55 P 0c4821d9898d42799c5789d4d2da3f63 reported cstate change: config changed from index 7 to 8, 9264f24d028347d98f7e4e2d1aedeb79 (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0c4821d9898d42799c5789d4d2da3f63" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:47.949152 8485 raft_consensus.cc:2932] T 21d27cca81e8455e9934e1cc604fc2b2 P 7a1404fe1c1b4b388b821c8261b53104 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9264f24d028347d98f7e4e2d1aedeb79 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } }
I20240503 01:59:47.951570 6494 consensus_queue.cc:237] T ccb34a4b52244d90850ab04e21299f55 P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } }
I20240503 01:59:47.952849 8080 raft_consensus.cc:1261] T ccb34a4b52244d90850ab04e21299f55 P 9264f24d028347d98f7e4e2d1aedeb79 [term 1 FOLLOWER]: Refusing update from remote peer 0c4821d9898d42799c5789d4d2da3f63: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 01:59:47.952895 8485 raft_consensus.cc:1261] T ccb34a4b52244d90850ab04e21299f55 P 7a1404fe1c1b4b388b821c8261b53104 [term 1 FOLLOWER]: Refusing update from remote peer 0c4821d9898d42799c5789d4d2da3f63: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 01:59:47.953226 12148 consensus_queue.cc:1035] T ccb34a4b52244d90850ab04e21299f55 P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 01:59:47.954736 11379 raft_consensus.cc:2932] T ccb34a4b52244d90850ab04e21299f55 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } }
I20240503 01:59:47.956013 8080 raft_consensus.cc:2932] T ccb34a4b52244d90850ab04e21299f55 P 9264f24d028347d98f7e4e2d1aedeb79 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } }
I20240503 01:59:47.956312 6210 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ccb34a4b52244d90850ab04e21299f55 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:47.957525 12151 consensus_queue.cc:1035] T ccb34a4b52244d90850ab04e21299f55 P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 01:59:47.957933 6214 catalog_manager.cc:5484] T ccb34a4b52244d90850ab04e21299f55 P 9264f24d028347d98f7e4e2d1aedeb79 reported cstate change: config changed from index 8 to 9, VOTER 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "0c4821d9898d42799c5789d4d2da3f63" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } }
I20240503 01:59:47.958441 8485 raft_consensus.cc:2932] T ccb34a4b52244d90850ab04e21299f55 P 7a1404fe1c1b4b388b821c8261b53104 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } }
I20240503 01:59:47.960191 6607 tablet_service.cc:1514] Processing DeleteTablet for tablet ccb34a4b52244d90850ab04e21299f55 with delete_type TABLET_DATA_TOMBSTONED (TS 79163c69f6f04a9abd0208d74e1c61be not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:47560
I20240503 01:59:47.960638 12464 tablet_replica.cc:330] T ccb34a4b52244d90850ab04e21299f55 P 79163c69f6f04a9abd0208d74e1c61be: stopping tablet replica
I20240503 01:59:47.960806 12464 raft_consensus.cc:2229] T ccb34a4b52244d90850ab04e21299f55 P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:47.961052 12464 raft_consensus.cc:2258] T ccb34a4b52244d90850ab04e21299f55 P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:47.961839 12464 ts_tablet_manager.cc:1875] T ccb34a4b52244d90850ab04e21299f55 P 79163c69f6f04a9abd0208d74e1c61be: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:47.963728 12464 ts_tablet_manager.cc:1888] T ccb34a4b52244d90850ab04e21299f55 P 79163c69f6f04a9abd0208d74e1c61be: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 01:59:47.963789 12464 log.cc:1196] T ccb34a4b52244d90850ab04e21299f55 P 79163c69f6f04a9abd0208d74e1c61be: Deleting WAL directory at /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/ccb34a4b52244d90850ab04e21299f55
I20240503 01:59:47.964237 6208 catalog_manager.cc:4830] TS 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131:44005): tablet ccb34a4b52244d90850ab04e21299f55 (table rebalance_test_table_8 [id=85186d17f9464a408a3790b0978eb502]) successfully deleted
I20240503 01:59:47.973942 12148 raft_consensus.cc:1050] T ff8f06c5ad8442ee942bf900212fe858 P 0c4821d9898d42799c5789d4d2da3f63: attempting to promote NON_VOTER 9264f24d028347d98f7e4e2d1aedeb79 to VOTER
I20240503 01:59:47.974359 12148 consensus_queue.cc:237] T ff8f06c5ad8442ee942bf900212fe858 P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } }
I20240503 01:59:47.975900 6625 raft_consensus.cc:1261] T ff8f06c5ad8442ee942bf900212fe858 P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Refusing update from remote peer 0c4821d9898d42799c5789d4d2da3f63: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:47.976095 8223 raft_consensus.cc:1261] T ff8f06c5ad8442ee942bf900212fe858 P 65d4632bcdd54ba596acdbbe1b5ef8af [term 1 FOLLOWER]: Refusing update from remote peer 0c4821d9898d42799c5789d4d2da3f63: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:47.976177 8080 raft_consensus.cc:1261] T ff8f06c5ad8442ee942bf900212fe858 P 9264f24d028347d98f7e4e2d1aedeb79 [term 1 LEARNER]: Refusing update from remote peer 0c4821d9898d42799c5789d4d2da3f63: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:47.976393 12148 consensus_queue.cc:1035] T ff8f06c5ad8442ee942bf900212fe858 P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:47.976596 12151 consensus_queue.cc:1035] T ff8f06c5ad8442ee942bf900212fe858 P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:47.976784 11379 consensus_queue.cc:1035] T ff8f06c5ad8442ee942bf900212fe858 P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:47.978960 12149 raft_consensus.cc:2932] T ff8f06c5ad8442ee942bf900212fe858 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9264f24d028347d98f7e4e2d1aedeb79 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } }
I20240503 01:59:47.979316 6625 raft_consensus.cc:2932] T ff8f06c5ad8442ee942bf900212fe858 P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9264f24d028347d98f7e4e2d1aedeb79 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } }
I20240503 01:59:47.981374 8080 raft_consensus.cc:2932] T ff8f06c5ad8442ee942bf900212fe858 P 9264f24d028347d98f7e4e2d1aedeb79 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9264f24d028347d98f7e4e2d1aedeb79 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } }
I20240503 01:59:47.981494 8223 raft_consensus.cc:2932] T ff8f06c5ad8442ee942bf900212fe858 P 65d4632bcdd54ba596acdbbe1b5ef8af [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9264f24d028347d98f7e4e2d1aedeb79 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } }
I20240503 01:59:47.983325 6214 catalog_manager.cc:5484] T ff8f06c5ad8442ee942bf900212fe858 P 0c4821d9898d42799c5789d4d2da3f63 reported cstate change: config changed from index 7 to 8, 9264f24d028347d98f7e4e2d1aedeb79 (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0c4821d9898d42799c5789d4d2da3f63" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:47.984699 11379 raft_consensus.cc:1050] T 120d5dd4f26c42399702141094ce7982 P 0c4821d9898d42799c5789d4d2da3f63: attempting to promote NON_VOTER 2d0fd64aaa6548e09e1ebde581c54658 to VOTER
I20240503 01:59:47.985037 11379 consensus_queue.cc:237] T 120d5dd4f26c42399702141094ce7982 P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } }
I20240503 01:59:47.986914 9410 raft_consensus.cc:1261] T 120d5dd4f26c42399702141094ce7982 P 2d0fd64aaa6548e09e1ebde581c54658 [term 1 LEARNER]: Refusing update from remote peer 0c4821d9898d42799c5789d4d2da3f63: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:47.987349 12148 consensus_queue.cc:1035] T 120d5dd4f26c42399702141094ce7982 P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:47.987504 6494 consensus_queue.cc:237] T ff8f06c5ad8442ee942bf900212fe858 P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } }
I20240503 01:59:47.988699 8080 raft_consensus.cc:1261] T ff8f06c5ad8442ee942bf900212fe858 P 9264f24d028347d98f7e4e2d1aedeb79 [term 1 FOLLOWER]: Refusing update from remote peer 0c4821d9898d42799c5789d4d2da3f63: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 01:59:47.988720 8223 raft_consensus.cc:1261] T ff8f06c5ad8442ee942bf900212fe858 P 65d4632bcdd54ba596acdbbe1b5ef8af [term 1 FOLLOWER]: Refusing update from remote peer 0c4821d9898d42799c5789d4d2da3f63: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 01:59:47.989112 8485 raft_consensus.cc:1261] T 120d5dd4f26c42399702141094ce7982 P 7a1404fe1c1b4b388b821c8261b53104 [term 1 FOLLOWER]: Refusing update from remote peer 0c4821d9898d42799c5789d4d2da3f63: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:47.989198 12149 consensus_queue.cc:1035] T ff8f06c5ad8442ee942bf900212fe858 P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 01:59:47.989514 12151 consensus_queue.cc:1035] T 120d5dd4f26c42399702141094ce7982 P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:47.989807 12149 consensus_queue.cc:1035] T ff8f06c5ad8442ee942bf900212fe858 P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 01:59:47.990669 6365 raft_consensus.cc:1261] T 120d5dd4f26c42399702141094ce7982 P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Refusing update from remote peer 0c4821d9898d42799c5789d4d2da3f63: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:47.991134 11379 consensus_queue.cc:1035] T 120d5dd4f26c42399702141094ce7982 P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:47.991442 12151 raft_consensus.cc:2932] T 120d5dd4f26c42399702141094ce7982 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2d0fd64aaa6548e09e1ebde581c54658 (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } } }
I20240503 01:59:47.991904 8486 raft_consensus.cc:2932] T 120d5dd4f26c42399702141094ce7982 P 7a1404fe1c1b4b388b821c8261b53104 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2d0fd64aaa6548e09e1ebde581c54658 (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } } }
I20240503 01:59:47.993402 12149 raft_consensus.cc:2932] T ff8f06c5ad8442ee942bf900212fe858 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } }
I20240503 01:59:47.993343 9410 raft_consensus.cc:2932] T 120d5dd4f26c42399702141094ce7982 P 2d0fd64aaa6548e09e1ebde581c54658 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2d0fd64aaa6548e09e1ebde581c54658 (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } } }
I20240503 01:59:47.994385 8223 raft_consensus.cc:2932] T ff8f06c5ad8442ee942bf900212fe858 P 65d4632bcdd54ba596acdbbe1b5ef8af [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } }
I20240503 01:59:47.995333 8080 raft_consensus.cc:2932] T ff8f06c5ad8442ee942bf900212fe858 P 9264f24d028347d98f7e4e2d1aedeb79 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } }
I20240503 01:59:47.995472 6221 catalog_manager.cc:5484] T 120d5dd4f26c42399702141094ce7982 P 7a1404fe1c1b4b388b821c8261b53104 reported cstate change: config changed from index 7 to 8, 2d0fd64aaa6548e09e1ebde581c54658 (127.0.99.137) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0c4821d9898d42799c5789d4d2da3f63" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } } }
I20240503 01:59:47.996520 6365 raft_consensus.cc:2932] T 120d5dd4f26c42399702141094ce7982 P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2d0fd64aaa6548e09e1ebde581c54658 (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } } }
I20240503 01:59:47.996915 6210 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ff8f06c5ad8442ee942bf900212fe858 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:47.998853 6219 catalog_manager.cc:5484] T ff8f06c5ad8442ee942bf900212fe858 P 0c4821d9898d42799c5789d4d2da3f63 reported cstate change: config changed from index 8 to 9, VOTER 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "0c4821d9898d42799c5789d4d2da3f63" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:48.001639 6494 consensus_queue.cc:237] T 120d5dd4f26c42399702141094ce7982 P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } }
I20240503 01:59:48.002278 6607 tablet_service.cc:1514] Processing DeleteTablet for tablet ff8f06c5ad8442ee942bf900212fe858 with delete_type TABLET_DATA_TOMBSTONED (TS 79163c69f6f04a9abd0208d74e1c61be not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:47560
I20240503 01:59:48.002528 12464 tablet_replica.cc:330] T ff8f06c5ad8442ee942bf900212fe858 P 79163c69f6f04a9abd0208d74e1c61be: stopping tablet replica
I20240503 01:59:48.002699 12464 raft_consensus.cc:2229] T ff8f06c5ad8442ee942bf900212fe858 P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:48.002882 12464 raft_consensus.cc:2258] T ff8f06c5ad8442ee942bf900212fe858 P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:48.002972 8486 raft_consensus.cc:1261] T 120d5dd4f26c42399702141094ce7982 P 7a1404fe1c1b4b388b821c8261b53104 [term 1 FOLLOWER]: Refusing update from remote peer 0c4821d9898d42799c5789d4d2da3f63: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 01:59:48.002972 9410 raft_consensus.cc:1261] T 120d5dd4f26c42399702141094ce7982 P 2d0fd64aaa6548e09e1ebde581c54658 [term 1 FOLLOWER]: Refusing update from remote peer 0c4821d9898d42799c5789d4d2da3f63: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 01:59:48.003367 12151 consensus_queue.cc:1035] T 120d5dd4f26c42399702141094ce7982 P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 01:59:48.003530 12148 consensus_queue.cc:1035] T 120d5dd4f26c42399702141094ce7982 P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 01:59:48.003587 12464 ts_tablet_manager.cc:1875] T ff8f06c5ad8442ee942bf900212fe858 P 79163c69f6f04a9abd0208d74e1c61be: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:48.005354 12464 ts_tablet_manager.cc:1888] T ff8f06c5ad8442ee942bf900212fe858 P 79163c69f6f04a9abd0208d74e1c61be: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 01:59:48.005438 12464 log.cc:1196] T ff8f06c5ad8442ee942bf900212fe858 P 79163c69f6f04a9abd0208d74e1c61be: Deleting WAL directory at /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/ff8f06c5ad8442ee942bf900212fe858
I20240503 01:59:48.005311 12148 raft_consensus.cc:2932] T 120d5dd4f26c42399702141094ce7982 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 4127249ba42644388d225d614c935a25 (127.0.99.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } } }
I20240503 01:59:48.005926 6208 catalog_manager.cc:4830] TS 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131:44005): tablet ff8f06c5ad8442ee942bf900212fe858 (table rebalance_test_table_0 [id=2e2540d8e10e43848a2e486f767d1576]) successfully deleted
I20240503 01:59:48.005828 8486 raft_consensus.cc:2932] T 120d5dd4f26c42399702141094ce7982 P 7a1404fe1c1b4b388b821c8261b53104 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 4127249ba42644388d225d614c935a25 (127.0.99.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } } }
I20240503 01:59:48.006980 6210 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 120d5dd4f26c42399702141094ce7982 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:48.008002 9410 raft_consensus.cc:2932] T 120d5dd4f26c42399702141094ce7982 P 2d0fd64aaa6548e09e1ebde581c54658 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 4127249ba42644388d225d614c935a25 (127.0.99.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } } }
I20240503 01:59:48.008688 6220 catalog_manager.cc:5484] T 120d5dd4f26c42399702141094ce7982 P 0c4821d9898d42799c5789d4d2da3f63 reported cstate change: config changed from index 8 to 9, VOTER 4127249ba42644388d225d614c935a25 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "0c4821d9898d42799c5789d4d2da3f63" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:48.011090 6344 tablet_service.cc:1514] Processing DeleteTablet for tablet 120d5dd4f26c42399702141094ce7982 with delete_type TABLET_DATA_TOMBSTONED (TS 4127249ba42644388d225d614c935a25 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:57398
I20240503 01:59:48.011690 12169 tablet_replica.cc:330] T 120d5dd4f26c42399702141094ce7982 P 4127249ba42644388d225d614c935a25: stopping tablet replica
I20240503 01:59:48.011833 12169 raft_consensus.cc:2229] T 120d5dd4f26c42399702141094ce7982 P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:48.012027 12169 raft_consensus.cc:2258] T 120d5dd4f26c42399702141094ce7982 P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:48.012751 12169 ts_tablet_manager.cc:1875] T 120d5dd4f26c42399702141094ce7982 P 4127249ba42644388d225d614c935a25: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:48.014299 12169 ts_tablet_manager.cc:1888] T 120d5dd4f26c42399702141094ce7982 P 4127249ba42644388d225d614c935a25: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 01:59:48.014365 12169 log.cc:1196] T 120d5dd4f26c42399702141094ce7982 P 4127249ba42644388d225d614c935a25: Deleting WAL directory at /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/120d5dd4f26c42399702141094ce7982
I20240503 01:59:48.014834 6208 catalog_manager.cc:4830] TS 4127249ba42644388d225d614c935a25 (127.0.99.129:35613): tablet 120d5dd4f26c42399702141094ce7982 (table rebalance_test_table_6 [id=80cb1486f22e4bee930434fdfc3463d2]) successfully deleted
I20240503 01:59:48.029515 12475 raft_consensus.cc:1050] T 4e7ceb1453804246a41bd1dda0e39d02 P 65d4632bcdd54ba596acdbbe1b5ef8af: attempting to promote NON_VOTER 7a1404fe1c1b4b388b821c8261b53104 to VOTER
I20240503 01:59:48.029966 12475 consensus_queue.cc:237] T 4e7ceb1453804246a41bd1dda0e39d02 P 65d4632bcdd54ba596acdbbe1b5ef8af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } }
I20240503 01:59:48.031306 8486 raft_consensus.cc:1261] T 4e7ceb1453804246a41bd1dda0e39d02 P 7a1404fe1c1b4b388b821c8261b53104 [term 2 LEARNER]: Refusing update from remote peer 65d4632bcdd54ba596acdbbe1b5ef8af: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 01:59:48.031423 6625 raft_consensus.cc:1261] T 4e7ceb1453804246a41bd1dda0e39d02 P 79163c69f6f04a9abd0208d74e1c61be [term 2 FOLLOWER]: Refusing update from remote peer 65d4632bcdd54ba596acdbbe1b5ef8af: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 01:59:48.031423 6494 raft_consensus.cc:1261] T 4e7ceb1453804246a41bd1dda0e39d02 P 0c4821d9898d42799c5789d4d2da3f63 [term 2 FOLLOWER]: Refusing update from remote peer 65d4632bcdd54ba596acdbbe1b5ef8af: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 01:59:48.031859 12474 consensus_queue.cc:1035] T 4e7ceb1453804246a41bd1dda0e39d02 P 65d4632bcdd54ba596acdbbe1b5ef8af [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 01:59:48.032037 12475 consensus_queue.cc:1035] T 4e7ceb1453804246a41bd1dda0e39d02 P 65d4632bcdd54ba596acdbbe1b5ef8af [LEADER]: Connected to new peer: Peer: permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 01:59:48.032222 12477 consensus_queue.cc:1035] T 4e7ceb1453804246a41bd1dda0e39d02 P 65d4632bcdd54ba596acdbbe1b5ef8af [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 01:59:48.034246 12475 raft_consensus.cc:2932] T 4e7ceb1453804246a41bd1dda0e39d02 P 65d4632bcdd54ba596acdbbe1b5ef8af [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 7a1404fe1c1b4b388b821c8261b53104 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } }
I20240503 01:59:48.034730 8486 raft_consensus.cc:2932] T 4e7ceb1453804246a41bd1dda0e39d02 P 7a1404fe1c1b4b388b821c8261b53104 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 7a1404fe1c1b4b388b821c8261b53104 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } }
I20240503 01:59:48.036089 6494 raft_consensus.cc:2932] T 4e7ceb1453804246a41bd1dda0e39d02 P 0c4821d9898d42799c5789d4d2da3f63 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 7a1404fe1c1b4b388b821c8261b53104 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } }
I20240503 01:59:48.036842 6625 raft_consensus.cc:2932] T 4e7ceb1453804246a41bd1dda0e39d02 P 79163c69f6f04a9abd0208d74e1c61be [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 7a1404fe1c1b4b388b821c8261b53104 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } }
I20240503 01:59:48.036986 6219 catalog_manager.cc:5484] T 4e7ceb1453804246a41bd1dda0e39d02 P 65d4632bcdd54ba596acdbbe1b5ef8af reported cstate change: config changed from index 8 to 9, 7a1404fe1c1b4b388b821c8261b53104 (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:48.040683 12162 raft_consensus.cc:1050] T a5926dae08184952bc693342c2c61ebf P 79163c69f6f04a9abd0208d74e1c61be: attempting to promote NON_VOTER 2d0fd64aaa6548e09e1ebde581c54658 to VOTER
I20240503 01:59:48.041074 12162 consensus_queue.cc:237] T a5926dae08184952bc693342c2c61ebf P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } }
I20240503 01:59:48.041086 8223 consensus_queue.cc:237] T 4e7ceb1453804246a41bd1dda0e39d02 P 65d4632bcdd54ba596acdbbe1b5ef8af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } }
I20240503 01:59:48.042146 6494 raft_consensus.cc:1261] T 4e7ceb1453804246a41bd1dda0e39d02 P 0c4821d9898d42799c5789d4d2da3f63 [term 2 FOLLOWER]: Refusing update from remote peer 65d4632bcdd54ba596acdbbe1b5ef8af: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 01:59:48.042464 8355 raft_consensus.cc:1261] T a5926dae08184952bc693342c2c61ebf P de2f076398ee44008622fb5db2a75952 [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:48.042515 8486 raft_consensus.cc:1261] T 4e7ceb1453804246a41bd1dda0e39d02 P 7a1404fe1c1b4b388b821c8261b53104 [term 2 FOLLOWER]: Refusing update from remote peer 65d4632bcdd54ba596acdbbe1b5ef8af: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 01:59:48.042491 12474 consensus_queue.cc:1035] T 4e7ceb1453804246a41bd1dda0e39d02 P 65d4632bcdd54ba596acdbbe1b5ef8af [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 01:59:48.043078 12474 consensus_queue.cc:1035] T 4e7ceb1453804246a41bd1dda0e39d02 P 65d4632bcdd54ba596acdbbe1b5ef8af [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 01:59:48.043376 6757 raft_consensus.cc:1261] T a5926dae08184952bc693342c2c61ebf P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:48.043632 12165 consensus_queue.cc:1035] T a5926dae08184952bc693342c2c61ebf P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:48.044145 9410 raft_consensus.cc:1261] T a5926dae08184952bc693342c2c61ebf P 2d0fd64aaa6548e09e1ebde581c54658 [term 1 LEARNER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:48.044715 12163 consensus_queue.cc:1035] T a5926dae08184952bc693342c2c61ebf P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:48.045039 12163 consensus_queue.cc:1035] T a5926dae08184952bc693342c2c61ebf P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:48.045331 11847 raft_consensus.cc:2932] T 4e7ceb1453804246a41bd1dda0e39d02 P 65d4632bcdd54ba596acdbbe1b5ef8af [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } }
I20240503 01:59:48.045812 8485 raft_consensus.cc:2932] T 4e7ceb1453804246a41bd1dda0e39d02 P 7a1404fe1c1b4b388b821c8261b53104 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } }
I20240503 01:59:48.047093 6207 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4e7ceb1453804246a41bd1dda0e39d02 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:48.047753 6494 raft_consensus.cc:2932] T 4e7ceb1453804246a41bd1dda0e39d02 P 0c4821d9898d42799c5789d4d2da3f63 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } }
I20240503 01:59:48.048064 6220 catalog_manager.cc:5484] T 4e7ceb1453804246a41bd1dda0e39d02 P 65d4632bcdd54ba596acdbbe1b5ef8af reported cstate change: config changed from index 9 to 10, VOTER 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131) evicted. New cstate: current_term: 2 leader_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:48.048523 12163 raft_consensus.cc:2932] T a5926dae08184952bc693342c2c61ebf P 79163c69f6f04a9abd0208d74e1c61be [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2d0fd64aaa6548e09e1ebde581c54658 (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } } }
I20240503 01:59:48.049021 8355 raft_consensus.cc:2932] T a5926dae08184952bc693342c2c61ebf P de2f076398ee44008622fb5db2a75952 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2d0fd64aaa6548e09e1ebde581c54658 (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } } }
I20240503 01:59:48.051185 6757 raft_consensus.cc:2932] T a5926dae08184952bc693342c2c61ebf P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2d0fd64aaa6548e09e1ebde581c54658 (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } } }
I20240503 01:59:48.051973 6214 catalog_manager.cc:5484] T a5926dae08184952bc693342c2c61ebf P 79163c69f6f04a9abd0208d74e1c61be reported cstate change: config changed from index 7 to 8, 2d0fd64aaa6548e09e1ebde581c54658 (127.0.99.137) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "79163c69f6f04a9abd0208d74e1c61be" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:48.053845 6607 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e7ceb1453804246a41bd1dda0e39d02 with delete_type TABLET_DATA_TOMBSTONED (TS 79163c69f6f04a9abd0208d74e1c61be not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:47560
I20240503 01:59:48.054051 12464 tablet_replica.cc:330] T 4e7ceb1453804246a41bd1dda0e39d02 P 79163c69f6f04a9abd0208d74e1c61be: stopping tablet replica
I20240503 01:59:48.054219 12464 raft_consensus.cc:2229] T 4e7ceb1453804246a41bd1dda0e39d02 P 79163c69f6f04a9abd0208d74e1c61be [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:48.054385 12464 raft_consensus.cc:2258] T 4e7ceb1453804246a41bd1dda0e39d02 P 79163c69f6f04a9abd0208d74e1c61be [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:48.055003 12464 ts_tablet_manager.cc:1875] T 4e7ceb1453804246a41bd1dda0e39d02 P 79163c69f6f04a9abd0208d74e1c61be: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:48.056231 9410 raft_consensus.cc:2932] T a5926dae08184952bc693342c2c61ebf P 2d0fd64aaa6548e09e1ebde581c54658 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2d0fd64aaa6548e09e1ebde581c54658 (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } } }
I20240503 01:59:48.057217 12464 ts_tablet_manager.cc:1888] T 4e7ceb1453804246a41bd1dda0e39d02 P 79163c69f6f04a9abd0208d74e1c61be: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240503 01:59:48.057294 12464 log.cc:1196] T 4e7ceb1453804246a41bd1dda0e39d02 P 79163c69f6f04a9abd0208d74e1c61be: Deleting WAL directory at /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/4e7ceb1453804246a41bd1dda0e39d02
I20240503 01:59:48.057907 6208 catalog_manager.cc:4830] TS 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131:44005): tablet 4e7ceb1453804246a41bd1dda0e39d02 (table rebalance_test_table_7 [id=dbd6f15c28634e3dac01249aa7cc2e25]) successfully deleted
I20240503 01:59:48.058867 6625 consensus_queue.cc:237] T a5926dae08184952bc693342c2c61ebf P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } }
I20240503 01:59:48.059919 9410 raft_consensus.cc:1261] T a5926dae08184952bc693342c2c61ebf P 2d0fd64aaa6548e09e1ebde581c54658 [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 01:59:48.059986 8355 raft_consensus.cc:1261] T a5926dae08184952bc693342c2c61ebf P de2f076398ee44008622fb5db2a75952 [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 01:59:48.060269 11781 consensus_queue.cc:1035] T a5926dae08184952bc693342c2c61ebf P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 01:59:48.060456 12162 consensus_queue.cc:1035] T a5926dae08184952bc693342c2c61ebf P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 01:59:48.061920 11781 raft_consensus.cc:2932] T a5926dae08184952bc693342c2c61ebf P 79163c69f6f04a9abd0208d74e1c61be [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER b6b1d5677b8d47788b7968f64b3dac3c (127.0.99.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } } }
I20240503 01:59:48.062371 9410 raft_consensus.cc:2932] T a5926dae08184952bc693342c2c61ebf P 2d0fd64aaa6548e09e1ebde581c54658 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER b6b1d5677b8d47788b7968f64b3dac3c (127.0.99.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } } }
I20240503 01:59:48.062487 8355 raft_consensus.cc:2932] T a5926dae08184952bc693342c2c61ebf P de2f076398ee44008622fb5db2a75952 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER b6b1d5677b8d47788b7968f64b3dac3c (127.0.99.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } } }
I20240503 01:59:48.063345 6208 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a5926dae08184952bc693342c2c61ebf with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:48.064561 6220 catalog_manager.cc:5484] T a5926dae08184952bc693342c2c61ebf P 2d0fd64aaa6548e09e1ebde581c54658 reported cstate change: config changed from index 8 to 9, VOTER b6b1d5677b8d47788b7968f64b3dac3c (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "79163c69f6f04a9abd0208d74e1c61be" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } } }
I20240503 01:59:48.066754 6738 tablet_service.cc:1514] Processing DeleteTablet for tablet a5926dae08184952bc693342c2c61ebf with delete_type TABLET_DATA_TOMBSTONED (TS b6b1d5677b8d47788b7968f64b3dac3c not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:51980
I20240503 01:59:48.066962 12160 tablet_replica.cc:330] T a5926dae08184952bc693342c2c61ebf P b6b1d5677b8d47788b7968f64b3dac3c: stopping tablet replica
I20240503 01:59:48.067104 12160 raft_consensus.cc:2229] T a5926dae08184952bc693342c2c61ebf P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:48.067262 12160 raft_consensus.cc:2258] T a5926dae08184952bc693342c2c61ebf P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:48.067865 12160 ts_tablet_manager.cc:1875] T a5926dae08184952bc693342c2c61ebf P b6b1d5677b8d47788b7968f64b3dac3c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:48.069526 12160 ts_tablet_manager.cc:1888] T a5926dae08184952bc693342c2c61ebf P b6b1d5677b8d47788b7968f64b3dac3c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 01:59:48.069593 12160 log.cc:1196] T a5926dae08184952bc693342c2c61ebf P b6b1d5677b8d47788b7968f64b3dac3c: Deleting WAL directory at /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/a5926dae08184952bc693342c2c61ebf
I20240503 01:59:48.070269 6209 catalog_manager.cc:4830] TS b6b1d5677b8d47788b7968f64b3dac3c (127.0.99.132:46067): tablet a5926dae08184952bc693342c2c61ebf (table rebalance_test_table_3 [id=2c07d364168a409a98aed9a7c3c1015c]) successfully deleted
I20240503 01:59:48.155076 12525 ts_tablet_manager.cc:902] T 0bdcea4e59ec4e2da63b9316f9d718be P 2d0fd64aaa6548e09e1ebde581c54658: Initiating tablet copy from peer 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131:44005)
I20240503 01:59:48.155649 12525 tablet_copy_client.cc:308] T 0bdcea4e59ec4e2da63b9316f9d718be P 2d0fd64aaa6548e09e1ebde581c54658: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:44005
I20240503 01:59:48.156131 6647 tablet_copy_service.cc:140] P 79163c69f6f04a9abd0208d74e1c61be: Received BeginTabletCopySession request for tablet 0bdcea4e59ec4e2da63b9316f9d718be from peer 2d0fd64aaa6548e09e1ebde581c54658 ({username='slave'} at 127.0.99.137:43265)
I20240503 01:59:48.156244 6647 tablet_copy_service.cc:161] P 79163c69f6f04a9abd0208d74e1c61be: Beginning new tablet copy session on tablet 0bdcea4e59ec4e2da63b9316f9d718be from peer 2d0fd64aaa6548e09e1ebde581c54658 at {username='slave'} at 127.0.99.137:43265: session id = 2d0fd64aaa6548e09e1ebde581c54658-0bdcea4e59ec4e2da63b9316f9d718be
I20240503 01:59:48.157244 6647 tablet_copy_source_session.cc:205] T 0bdcea4e59ec4e2da63b9316f9d718be P 79163c69f6f04a9abd0208d74e1c61be: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:48.158206 12525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bdcea4e59ec4e2da63b9316f9d718be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:48.160230 12525 tablet_copy_client.cc:783] T 0bdcea4e59ec4e2da63b9316f9d718be P 2d0fd64aaa6548e09e1ebde581c54658: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:48.160418 12525 tablet_copy_client.cc:647] T 0bdcea4e59ec4e2da63b9316f9d718be P 2d0fd64aaa6548e09e1ebde581c54658: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:48.161757 12529 tablet_copy_client.cc:1002] T 0bdcea4e59ec4e2da63b9316f9d718be P 2d0fd64aaa6548e09e1ebde581c54658: tablet copy: Injecting latency into file download: 1500
I20240503 01:59:48.180538 6625 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "21d27cca81e8455e9934e1cc604fc2b2"
dest_uuid: "79163c69f6f04a9abd0208d74e1c61be"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:55540
I20240503 01:59:48.180684 6625 raft_consensus.cc:595] T 21d27cca81e8455e9934e1cc604fc2b2 P 79163c69f6f04a9abd0208d74e1c61be [term 1 LEADER]: Received request to transfer leadership
I20240503 01:59:48.187667 6625 consensus_queue.cc:237] T a388d59f6ba94e91958f55b340343fa6 P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: true } }
I20240503 01:59:48.189165 6365 raft_consensus.cc:1261] T a388d59f6ba94e91958f55b340343fa6 P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 01:59:48.189244 6562 consensus_peers.cc:485] T a388d59f6ba94e91958f55b340343fa6 P 79163c69f6f04a9abd0208d74e1c61be -> Peer 2d0fd64aaa6548e09e1ebde581c54658 (127.0.99.137:40613): Couldn't send request to peer 2d0fd64aaa6548e09e1ebde581c54658. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a388d59f6ba94e91958f55b340343fa6. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:48.189502 8080 raft_consensus.cc:1261] T a388d59f6ba94e91958f55b340343fa6 P 9264f24d028347d98f7e4e2d1aedeb79 [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:48.189600 12163 consensus_queue.cc:1035] T a388d59f6ba94e91958f55b340343fa6 P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:48.189999 11781 consensus_queue.cc:1035] T a388d59f6ba94e91958f55b340343fa6 P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:48.191306 12163 raft_consensus.cc:2932] T a388d59f6ba94e91958f55b340343fa6 P 79163c69f6f04a9abd0208d74e1c61be [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2d0fd64aaa6548e09e1ebde581c54658 (127.0.99.137) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: true } } }
I20240503 01:59:48.191718 6365 raft_consensus.cc:2932] T a388d59f6ba94e91958f55b340343fa6 P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2d0fd64aaa6548e09e1ebde581c54658 (127.0.99.137) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: true } } }
I20240503 01:59:48.192098 8080 raft_consensus.cc:2932] T a388d59f6ba94e91958f55b340343fa6 P 9264f24d028347d98f7e4e2d1aedeb79 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2d0fd64aaa6548e09e1ebde581c54658 (127.0.99.137) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: true } } }
I20240503 01:59:48.195763 6220 catalog_manager.cc:5484] T a388d59f6ba94e91958f55b340343fa6 P 9264f24d028347d98f7e4e2d1aedeb79 reported cstate change: config changed from index 6 to 7, NON_VOTER 2d0fd64aaa6548e09e1ebde581c54658 (127.0.99.137) added. New cstate: current_term: 1 leader_uuid: "79163c69f6f04a9abd0208d74e1c61be" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: true } } }
I20240503 01:59:48.202750 6494 consensus_queue.cc:237] T e5ed3e729bc64ebeac216ce5076b9fd9 P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: true } }
I20240503 01:59:48.204156 8355 raft_consensus.cc:1261] T e5ed3e729bc64ebeac216ce5076b9fd9 P de2f076398ee44008622fb5db2a75952 [term 1 FOLLOWER]: Refusing update from remote peer 0c4821d9898d42799c5789d4d2da3f63: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:48.204448 6757 raft_consensus.cc:1261] T e5ed3e729bc64ebeac216ce5076b9fd9 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Refusing update from remote peer 0c4821d9898d42799c5789d4d2da3f63: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 01:59:48.204571 6431 consensus_peers.cc:485] T e5ed3e729bc64ebeac216ce5076b9fd9 P 0c4821d9898d42799c5789d4d2da3f63 -> Peer 2d0fd64aaa6548e09e1ebde581c54658 (127.0.99.137:40613): Couldn't send request to peer 2d0fd64aaa6548e09e1ebde581c54658. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e5ed3e729bc64ebeac216ce5076b9fd9. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:48.204932 11379 consensus_queue.cc:1035] T e5ed3e729bc64ebeac216ce5076b9fd9 P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:48.205143 12151 consensus_queue.cc:1035] T e5ed3e729bc64ebeac216ce5076b9fd9 P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:48.205464 12356 raft_consensus.cc:1050] T f11a8034585a4d57be0e9b1e5e8bcd85 P b6b1d5677b8d47788b7968f64b3dac3c: attempting to promote NON_VOTER 2d0fd64aaa6548e09e1ebde581c54658 to VOTER
I20240503 01:59:48.205873 12356 consensus_queue.cc:237] T f11a8034585a4d57be0e9b1e5e8bcd85 P b6b1d5677b8d47788b7968f64b3dac3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } }
I20240503 01:59:48.207329 9410 raft_consensus.cc:1261] T f11a8034585a4d57be0e9b1e5e8bcd85 P 2d0fd64aaa6548e09e1ebde581c54658 [term 1 LEARNER]: Refusing update from remote peer b6b1d5677b8d47788b7968f64b3dac3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:48.207527 6494 raft_consensus.cc:1261] T f11a8034585a4d57be0e9b1e5e8bcd85 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Refusing update from remote peer b6b1d5677b8d47788b7968f64b3dac3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:48.207854 12564 consensus_queue.cc:1035] T f11a8034585a4d57be0e9b1e5e8bcd85 P b6b1d5677b8d47788b7968f64b3dac3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:48.207928 8080 raft_consensus.cc:1261] T f11a8034585a4d57be0e9b1e5e8bcd85 P 9264f24d028347d98f7e4e2d1aedeb79 [term 1 FOLLOWER]: Refusing update from remote peer b6b1d5677b8d47788b7968f64b3dac3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:48.208052 12151 raft_consensus.cc:2932] T e5ed3e729bc64ebeac216ce5076b9fd9 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2d0fd64aaa6548e09e1ebde581c54658 (127.0.99.137) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: true } } }
I20240503 01:59:48.208900 6757 raft_consensus.cc:2932] T e5ed3e729bc64ebeac216ce5076b9fd9 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2d0fd64aaa6548e09e1ebde581c54658 (127.0.99.137) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: true } } }
I20240503 01:59:48.209841 11990 consensus_queue.cc:1035] T f11a8034585a4d57be0e9b1e5e8bcd85 P b6b1d5677b8d47788b7968f64b3dac3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:48.210021 12564 consensus_queue.cc:1035] T f11a8034585a4d57be0e9b1e5e8bcd85 P b6b1d5677b8d47788b7968f64b3dac3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:48.210194 8355 raft_consensus.cc:2932] T e5ed3e729bc64ebeac216ce5076b9fd9 P de2f076398ee44008622fb5db2a75952 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2d0fd64aaa6548e09e1ebde581c54658 (127.0.99.137) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: true } } }
I20240503 01:59:48.211807 6219 catalog_manager.cc:5484] T e5ed3e729bc64ebeac216ce5076b9fd9 P 0c4821d9898d42799c5789d4d2da3f63 reported cstate change: config changed from index 6 to 7, NON_VOTER 2d0fd64aaa6548e09e1ebde581c54658 (127.0.99.137) added. New cstate: current_term: 1 leader_uuid: "0c4821d9898d42799c5789d4d2da3f63" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:48.213553 11990 raft_consensus.cc:2932] T f11a8034585a4d57be0e9b1e5e8bcd85 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2d0fd64aaa6548e09e1ebde581c54658 (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } } }
I20240503 01:59:48.215525 8080 raft_consensus.cc:2932] T f11a8034585a4d57be0e9b1e5e8bcd85 P 9264f24d028347d98f7e4e2d1aedeb79 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2d0fd64aaa6548e09e1ebde581c54658 (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } } }
I20240503 01:59:48.215600 9410 raft_consensus.cc:2932] T f11a8034585a4d57be0e9b1e5e8bcd85 P 2d0fd64aaa6548e09e1ebde581c54658 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2d0fd64aaa6548e09e1ebde581c54658 (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } } }
I20240503 01:59:48.217000 6494 raft_consensus.cc:2932] T f11a8034585a4d57be0e9b1e5e8bcd85 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2d0fd64aaa6548e09e1ebde581c54658 (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } } }
I20240503 01:59:48.217818 6220 catalog_manager.cc:5484] T f11a8034585a4d57be0e9b1e5e8bcd85 P 9264f24d028347d98f7e4e2d1aedeb79 reported cstate change: config changed from index 7 to 8, 2d0fd64aaa6548e09e1ebde581c54658 (127.0.99.137) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } } }
I20240503 01:59:48.221235 6757 consensus_queue.cc:237] T f11a8034585a4d57be0e9b1e5e8bcd85 P b6b1d5677b8d47788b7968f64b3dac3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } }
I20240503 01:59:48.222328 9410 raft_consensus.cc:1261] T f11a8034585a4d57be0e9b1e5e8bcd85 P 2d0fd64aaa6548e09e1ebde581c54658 [term 1 FOLLOWER]: Refusing update from remote peer b6b1d5677b8d47788b7968f64b3dac3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 01:59:48.222941 8080 raft_consensus.cc:1261] T f11a8034585a4d57be0e9b1e5e8bcd85 P 9264f24d028347d98f7e4e2d1aedeb79 [term 1 FOLLOWER]: Refusing update from remote peer b6b1d5677b8d47788b7968f64b3dac3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 01:59:48.223299 12564 consensus_queue.cc:1035] T f11a8034585a4d57be0e9b1e5e8bcd85 P b6b1d5677b8d47788b7968f64b3dac3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 01:59:48.223619 12575 consensus_queue.cc:1035] T f11a8034585a4d57be0e9b1e5e8bcd85 P b6b1d5677b8d47788b7968f64b3dac3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 01:59:48.225049 12564 raft_consensus.cc:2932] T f11a8034585a4d57be0e9b1e5e8bcd85 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } } }
I20240503 01:59:48.225525 9410 raft_consensus.cc:2932] T f11a8034585a4d57be0e9b1e5e8bcd85 P 2d0fd64aaa6548e09e1ebde581c54658 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } } }
I20240503 01:59:48.225726 8080 raft_consensus.cc:2932] T f11a8034585a4d57be0e9b1e5e8bcd85 P 9264f24d028347d98f7e4e2d1aedeb79 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } } }
I20240503 01:59:48.227366 6209 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f11a8034585a4d57be0e9b1e5e8bcd85 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:48.227633 6220 catalog_manager.cc:5484] T f11a8034585a4d57be0e9b1e5e8bcd85 P b6b1d5677b8d47788b7968f64b3dac3c reported cstate change: config changed from index 8 to 9, VOTER 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:48.230155 6476 tablet_service.cc:1514] Processing DeleteTablet for tablet f11a8034585a4d57be0e9b1e5e8bcd85 with delete_type TABLET_DATA_TOMBSTONED (TS 0c4821d9898d42799c5789d4d2da3f63 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:39850
I20240503 01:59:48.230360 12332 tablet_replica.cc:330] T f11a8034585a4d57be0e9b1e5e8bcd85 P 0c4821d9898d42799c5789d4d2da3f63: stopping tablet replica
I20240503 01:59:48.230477 12332 raft_consensus.cc:2229] T f11a8034585a4d57be0e9b1e5e8bcd85 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:48.230650 12332 raft_consensus.cc:2258] T f11a8034585a4d57be0e9b1e5e8bcd85 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:48.231454 12332 ts_tablet_manager.cc:1875] T f11a8034585a4d57be0e9b1e5e8bcd85 P 0c4821d9898d42799c5789d4d2da3f63: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:48.233398 12332 ts_tablet_manager.cc:1888] T f11a8034585a4d57be0e9b1e5e8bcd85 P 0c4821d9898d42799c5789d4d2da3f63: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 01:59:48.233474 12332 log.cc:1196] T f11a8034585a4d57be0e9b1e5e8bcd85 P 0c4821d9898d42799c5789d4d2da3f63: Deleting WAL directory at /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/f11a8034585a4d57be0e9b1e5e8bcd85
I20240503 01:59:48.233968 6210 catalog_manager.cc:4830] TS 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130:34189): tablet f11a8034585a4d57be0e9b1e5e8bcd85 (table rebalance_test_table_9 [id=27f301d87e884061bddb2605833c29a6]) successfully deleted
I20240503 01:59:48.240893 12587 ts_tablet_manager.cc:902] T c3337c95b354418faf37684e7440b1c7 P 2d0fd64aaa6548e09e1ebde581c54658: Initiating tablet copy from peer 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131:44005)
I20240503 01:59:48.241463 12587 tablet_copy_client.cc:308] T c3337c95b354418faf37684e7440b1c7 P 2d0fd64aaa6548e09e1ebde581c54658: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:44005
I20240503 01:59:48.242101 6647 tablet_copy_service.cc:140] P 79163c69f6f04a9abd0208d74e1c61be: Received BeginTabletCopySession request for tablet c3337c95b354418faf37684e7440b1c7 from peer 2d0fd64aaa6548e09e1ebde581c54658 ({username='slave'} at 127.0.99.137:43265)
I20240503 01:59:48.242197 6647 tablet_copy_service.cc:161] P 79163c69f6f04a9abd0208d74e1c61be: Beginning new tablet copy session on tablet c3337c95b354418faf37684e7440b1c7 from peer 2d0fd64aaa6548e09e1ebde581c54658 at {username='slave'} at 127.0.99.137:43265: session id = 2d0fd64aaa6548e09e1ebde581c54658-c3337c95b354418faf37684e7440b1c7
I20240503 01:59:48.243006 6647 tablet_copy_source_session.cc:205] T c3337c95b354418faf37684e7440b1c7 P 79163c69f6f04a9abd0208d74e1c61be: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:48.243831 12587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3337c95b354418faf37684e7440b1c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:48.246063 12587 tablet_copy_client.cc:783] T c3337c95b354418faf37684e7440b1c7 P 2d0fd64aaa6548e09e1ebde581c54658: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:48.246232 12587 tablet_copy_client.cc:647] T c3337c95b354418faf37684e7440b1c7 P 2d0fd64aaa6548e09e1ebde581c54658: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:48.247202 12591 tablet_copy_client.cc:1002] T c3337c95b354418faf37684e7440b1c7 P 2d0fd64aaa6548e09e1ebde581c54658: tablet copy: Injecting latency into file download: 1500
I20240503 01:59:48.267654 6625 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "21d27cca81e8455e9934e1cc604fc2b2"
dest_uuid: "79163c69f6f04a9abd0208d74e1c61be"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:55614
I20240503 01:59:48.267777 6625 raft_consensus.cc:595] T 21d27cca81e8455e9934e1cc604fc2b2 P 79163c69f6f04a9abd0208d74e1c61be [term 1 LEADER]: Received request to transfer leadership
I20240503 01:59:48.275887 6365 consensus_queue.cc:237] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 4127249ba42644388d225d614c935a25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: true } }
I20240503 01:59:48.277313 6625 raft_consensus.cc:1261] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Refusing update from remote peer 4127249ba42644388d225d614c935a25: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:48.277549 8080 raft_consensus.cc:1261] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 9264f24d028347d98f7e4e2d1aedeb79 [term 1 FOLLOWER]: Refusing update from remote peer 4127249ba42644388d225d614c935a25: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:48.277720 11843 consensus_queue.cc:1035] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 4127249ba42644388d225d614c935a25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:48.277926 12161 consensus_queue.cc:1035] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 4127249ba42644388d225d614c935a25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:48.280047 11843 raft_consensus.cc:2932] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 4127249ba42644388d225d614c935a25 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2d0fd64aaa6548e09e1ebde581c54658 (127.0.99.137) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: true } } }
I20240503 01:59:48.282038 8080 raft_consensus.cc:2932] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 9264f24d028347d98f7e4e2d1aedeb79 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2d0fd64aaa6548e09e1ebde581c54658 (127.0.99.137) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: true } } }
I20240503 01:59:48.282135 6625 raft_consensus.cc:2932] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2d0fd64aaa6548e09e1ebde581c54658 (127.0.99.137) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: true } } }
I20240503 01:59:48.284128 6219 catalog_manager.cc:5484] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 4127249ba42644388d225d614c935a25 reported cstate change: config changed from index 6 to 7, NON_VOTER 2d0fd64aaa6548e09e1ebde581c54658 (127.0.99.137) added. New cstate: current_term: 1 leader_uuid: "4127249ba42644388d225d614c935a25" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 01:59:48.285265 6300 consensus_peers.cc:485] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 4127249ba42644388d225d614c935a25 -> Peer 2d0fd64aaa6548e09e1ebde581c54658 (127.0.99.137:40613): Couldn't send request to peer 2d0fd64aaa6548e09e1ebde581c54658. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2088c84e4eef4ccfbc70d7b668b8f2cf. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:48.294920 11379 raft_consensus.cc:1050] T 1d392e08c15a4d4098a350287b3163ef P 0c4821d9898d42799c5789d4d2da3f63: attempting to promote NON_VOTER 65d4632bcdd54ba596acdbbe1b5ef8af to VOTER
I20240503 01:59:48.295346 11379 consensus_queue.cc:237] T 1d392e08c15a4d4098a350287b3163ef P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } }
I20240503 01:59:48.296775 8355 raft_consensus.cc:1261] T 1d392e08c15a4d4098a350287b3163ef P de2f076398ee44008622fb5db2a75952 [term 1 FOLLOWER]: Refusing update from remote peer 0c4821d9898d42799c5789d4d2da3f63: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:48.296775 6757 raft_consensus.cc:1261] T 1d392e08c15a4d4098a350287b3163ef P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Refusing update from remote peer 0c4821d9898d42799c5789d4d2da3f63: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:48.296895 8223 raft_consensus.cc:1261] T 1d392e08c15a4d4098a350287b3163ef P 65d4632bcdd54ba596acdbbe1b5ef8af [term 1 LEARNER]: Refusing update from remote peer 0c4821d9898d42799c5789d4d2da3f63: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:48.297178 12149 consensus_queue.cc:1035] T 1d392e08c15a4d4098a350287b3163ef P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:48.297425 12148 consensus_queue.cc:1035] T 1d392e08c15a4d4098a350287b3163ef P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:48.297562 12151 consensus_queue.cc:1035] T 1d392e08c15a4d4098a350287b3163ef P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:48.299780 12151 raft_consensus.cc:2932] T 1d392e08c15a4d4098a350287b3163ef P 0c4821d9898d42799c5789d4d2da3f63 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 65d4632bcdd54ba596acdbbe1b5ef8af (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } }
I20240503 01:59:48.300191 8223 raft_consensus.cc:2932] T 1d392e08c15a4d4098a350287b3163ef P 65d4632bcdd54ba596acdbbe1b5ef8af [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 65d4632bcdd54ba596acdbbe1b5ef8af (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } }
I20240503 01:59:48.300706 8355 raft_consensus.cc:2932] T 1d392e08c15a4d4098a350287b3163ef P de2f076398ee44008622fb5db2a75952 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 65d4632bcdd54ba596acdbbe1b5ef8af (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } }
I20240503 01:59:48.301723 6757 raft_consensus.cc:2932] T 1d392e08c15a4d4098a350287b3163ef P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 65d4632bcdd54ba596acdbbe1b5ef8af (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } }
I20240503 01:59:48.303833 6219 catalog_manager.cc:5484] T 1d392e08c15a4d4098a350287b3163ef P de2f076398ee44008622fb5db2a75952 reported cstate change: config changed from index 7 to 8, 65d4632bcdd54ba596acdbbe1b5ef8af (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0c4821d9898d42799c5789d4d2da3f63" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } }
I20240503 01:59:48.330883 6494 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1d392e08c15a4d4098a350287b3163ef"
dest_uuid: "0c4821d9898d42799c5789d4d2da3f63"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:56670
I20240503 01:59:48.331075 6494 raft_consensus.cc:595] T 1d392e08c15a4d4098a350287b3163ef P 0c4821d9898d42799c5789d4d2da3f63 [term 1 LEADER]: Received request to transfer leadership
I20240503 01:59:48.341434 6625 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "21d27cca81e8455e9934e1cc604fc2b2"
dest_uuid: "79163c69f6f04a9abd0208d74e1c61be"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:55680
I20240503 01:59:48.341564 6625 raft_consensus.cc:595] T 21d27cca81e8455e9934e1cc604fc2b2 P 79163c69f6f04a9abd0208d74e1c61be [term 1 LEADER]: Received request to transfer leadership
I20240503 01:59:48.424860 12162 raft_consensus.cc:979] T 21d27cca81e8455e9934e1cc604fc2b2 P 79163c69f6f04a9abd0208d74e1c61be: : Instructing follower 9264f24d028347d98f7e4e2d1aedeb79 to start an election
I20240503 01:59:48.424973 12162 raft_consensus.cc:1067] T 21d27cca81e8455e9934e1cc604fc2b2 P 79163c69f6f04a9abd0208d74e1c61be [term 1 LEADER]: Signalling peer 9264f24d028347d98f7e4e2d1aedeb79 to start an election
I20240503 01:59:48.425467 8080 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "21d27cca81e8455e9934e1cc604fc2b2"
dest_uuid: "9264f24d028347d98f7e4e2d1aedeb79"
from {username='slave'} at 127.0.99.131:40425
I20240503 01:59:48.425670 8080 raft_consensus.cc:482] T 21d27cca81e8455e9934e1cc604fc2b2 P 9264f24d028347d98f7e4e2d1aedeb79 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 01:59:48.425793 8080 raft_consensus.cc:3037] T 21d27cca81e8455e9934e1cc604fc2b2 P 9264f24d028347d98f7e4e2d1aedeb79 [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:48.426936 8080 raft_consensus.cc:504] T 21d27cca81e8455e9934e1cc604fc2b2 P 9264f24d028347d98f7e4e2d1aedeb79 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } }
I20240503 01:59:48.427942 8080 leader_election.cc:288] T 21d27cca81e8455e9934e1cc604fc2b2 P 9264f24d028347d98f7e4e2d1aedeb79 [CANDIDATE]: Term 2 election: Requested vote from peers 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131:44005), 4127249ba42644388d225d614c935a25 (127.0.99.129:35613), 7a1404fe1c1b4b388b821c8261b53104 (127.0.99.136:46759)
I20240503 01:59:48.428339 6625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21d27cca81e8455e9934e1cc604fc2b2" candidate_uuid: "9264f24d028347d98f7e4e2d1aedeb79" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "79163c69f6f04a9abd0208d74e1c61be"
I20240503 01:59:48.428355 6365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21d27cca81e8455e9934e1cc604fc2b2" candidate_uuid: "9264f24d028347d98f7e4e2d1aedeb79" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "4127249ba42644388d225d614c935a25"
I20240503 01:59:48.428478 6625 raft_consensus.cc:3032] T 21d27cca81e8455e9934e1cc604fc2b2 P 79163c69f6f04a9abd0208d74e1c61be [term 1 LEADER]: Stepping down as leader of term 1
I20240503 01:59:48.428491 6365 raft_consensus.cc:3037] T 21d27cca81e8455e9934e1cc604fc2b2 P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:48.428526 6625 raft_consensus.cc:729] T 21d27cca81e8455e9934e1cc604fc2b2 P 79163c69f6f04a9abd0208d74e1c61be [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 79163c69f6f04a9abd0208d74e1c61be, State: Running, Role: LEADER
I20240503 01:59:48.428695 6625 consensus_queue.cc:260] T 21d27cca81e8455e9934e1cc604fc2b2 P 79163c69f6f04a9abd0208d74e1c61be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } }
I20240503 01:59:48.428905 6625 raft_consensus.cc:3037] T 21d27cca81e8455e9934e1cc604fc2b2 P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:48.429591 6365 raft_consensus.cc:2454] T 21d27cca81e8455e9934e1cc604fc2b2 P 4127249ba42644388d225d614c935a25 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9264f24d028347d98f7e4e2d1aedeb79 in term 2.
I20240503 01:59:48.429792 6625 raft_consensus.cc:2454] T 21d27cca81e8455e9934e1cc604fc2b2 P 79163c69f6f04a9abd0208d74e1c61be [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9264f24d028347d98f7e4e2d1aedeb79 in term 2.
I20240503 01:59:48.430083 8027 leader_election.cc:302] T 21d27cca81e8455e9934e1cc604fc2b2 P 9264f24d028347d98f7e4e2d1aedeb79 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 4127249ba42644388d225d614c935a25, 79163c69f6f04a9abd0208d74e1c61be, 9264f24d028347d98f7e4e2d1aedeb79; no voters:
I20240503 01:59:48.430303 12038 raft_consensus.cc:2781] T 21d27cca81e8455e9934e1cc604fc2b2 P 9264f24d028347d98f7e4e2d1aedeb79 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 01:59:48.430488 12038 raft_consensus.cc:686] T 21d27cca81e8455e9934e1cc604fc2b2 P 9264f24d028347d98f7e4e2d1aedeb79 [term 2 LEADER]: Becoming Leader. State: Replica: 9264f24d028347d98f7e4e2d1aedeb79, State: Running, Role: LEADER
I20240503 01:59:48.430770 12038 consensus_queue.cc:237] T 21d27cca81e8455e9934e1cc604fc2b2 P 9264f24d028347d98f7e4e2d1aedeb79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } }
I20240503 01:59:48.432258 8485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21d27cca81e8455e9934e1cc604fc2b2" candidate_uuid: "9264f24d028347d98f7e4e2d1aedeb79" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "7a1404fe1c1b4b388b821c8261b53104"
I20240503 01:59:48.432416 8485 raft_consensus.cc:3037] T 21d27cca81e8455e9934e1cc604fc2b2 P 7a1404fe1c1b4b388b821c8261b53104 [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:48.433398 8485 raft_consensus.cc:2454] T 21d27cca81e8455e9934e1cc604fc2b2 P 7a1404fe1c1b4b388b821c8261b53104 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9264f24d028347d98f7e4e2d1aedeb79 in term 2.
I20240503 01:59:48.433710 6220 catalog_manager.cc:5484] T 21d27cca81e8455e9934e1cc604fc2b2 P 9264f24d028347d98f7e4e2d1aedeb79 reported cstate change: term changed from 1 to 2, leader changed from 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131) to 9264f24d028347d98f7e4e2d1aedeb79 (127.0.99.133). New cstate: current_term: 2 leader_uuid: "9264f24d028347d98f7e4e2d1aedeb79" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:48.589026 6494 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1d392e08c15a4d4098a350287b3163ef"
dest_uuid: "0c4821d9898d42799c5789d4d2da3f63"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:56712
I20240503 01:59:48.589191 6494 raft_consensus.cc:595] T 1d392e08c15a4d4098a350287b3163ef P 0c4821d9898d42799c5789d4d2da3f63 [term 1 LEADER]: Received request to transfer leadership
I20240503 01:59:48.628340 12721 ts_tablet_manager.cc:902] T a388d59f6ba94e91958f55b340343fa6 P 2d0fd64aaa6548e09e1ebde581c54658: Initiating tablet copy from peer 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131:44005)
I20240503 01:59:48.628883 12721 tablet_copy_client.cc:308] T a388d59f6ba94e91958f55b340343fa6 P 2d0fd64aaa6548e09e1ebde581c54658: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:44005
I20240503 01:59:48.629351 6647 tablet_copy_service.cc:140] P 79163c69f6f04a9abd0208d74e1c61be: Received BeginTabletCopySession request for tablet a388d59f6ba94e91958f55b340343fa6 from peer 2d0fd64aaa6548e09e1ebde581c54658 ({username='slave'} at 127.0.99.137:43265)
I20240503 01:59:48.629468 6647 tablet_copy_service.cc:161] P 79163c69f6f04a9abd0208d74e1c61be: Beginning new tablet copy session on tablet a388d59f6ba94e91958f55b340343fa6 from peer 2d0fd64aaa6548e09e1ebde581c54658 at {username='slave'} at 127.0.99.137:43265: session id = 2d0fd64aaa6548e09e1ebde581c54658-a388d59f6ba94e91958f55b340343fa6
I20240503 01:59:48.630172 6647 tablet_copy_source_session.cc:205] T a388d59f6ba94e91958f55b340343fa6 P 79163c69f6f04a9abd0208d74e1c61be: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:48.630973 12721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a388d59f6ba94e91958f55b340343fa6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:48.632738 12721 tablet_copy_client.cc:783] T a388d59f6ba94e91958f55b340343fa6 P 2d0fd64aaa6548e09e1ebde581c54658: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:48.632899 12721 tablet_copy_client.cc:647] T a388d59f6ba94e91958f55b340343fa6 P 2d0fd64aaa6548e09e1ebde581c54658: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:48.633970 12725 tablet_copy_client.cc:1002] T a388d59f6ba94e91958f55b340343fa6 P 2d0fd64aaa6548e09e1ebde581c54658: tablet copy: Injecting latency into file download: 1500
I20240503 01:59:48.688943 12148 raft_consensus.cc:979] T 1d392e08c15a4d4098a350287b3163ef P 0c4821d9898d42799c5789d4d2da3f63: : Instructing follower 65d4632bcdd54ba596acdbbe1b5ef8af to start an election
I20240503 01:59:48.689090 12148 raft_consensus.cc:1067] T 1d392e08c15a4d4098a350287b3163ef P 0c4821d9898d42799c5789d4d2da3f63 [term 1 LEADER]: Signalling peer 65d4632bcdd54ba596acdbbe1b5ef8af to start an election
I20240503 01:59:48.689622 8223 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1d392e08c15a4d4098a350287b3163ef"
dest_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af"
from {username='slave'} at 127.0.99.130:36119
I20240503 01:59:48.689751 8223 raft_consensus.cc:482] T 1d392e08c15a4d4098a350287b3163ef P 65d4632bcdd54ba596acdbbe1b5ef8af [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 01:59:48.689826 8223 raft_consensus.cc:3037] T 1d392e08c15a4d4098a350287b3163ef P 65d4632bcdd54ba596acdbbe1b5ef8af [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:48.690649 8223 raft_consensus.cc:504] T 1d392e08c15a4d4098a350287b3163ef P 65d4632bcdd54ba596acdbbe1b5ef8af [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } }
I20240503 01:59:48.691155 8223 leader_election.cc:288] T 1d392e08c15a4d4098a350287b3163ef P 65d4632bcdd54ba596acdbbe1b5ef8af [CANDIDATE]: Term 2 election: Requested vote from peers b6b1d5677b8d47788b7968f64b3dac3c (127.0.99.132:46067), 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130:34189), de2f076398ee44008622fb5db2a75952 (127.0.99.135:34035)
I20240503 01:59:48.691475 6757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d392e08c15a4d4098a350287b3163ef" candidate_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "b6b1d5677b8d47788b7968f64b3dac3c"
I20240503 01:59:48.691610 6757 raft_consensus.cc:3037] T 1d392e08c15a4d4098a350287b3163ef P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:48.692608 6757 raft_consensus.cc:2454] T 1d392e08c15a4d4098a350287b3163ef P b6b1d5677b8d47788b7968f64b3dac3c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65d4632bcdd54ba596acdbbe1b5ef8af in term 2.
I20240503 01:59:48.692831 6494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d392e08c15a4d4098a350287b3163ef" candidate_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "0c4821d9898d42799c5789d4d2da3f63"
I20240503 01:59:48.692978 6494 raft_consensus.cc:3032] T 1d392e08c15a4d4098a350287b3163ef P 0c4821d9898d42799c5789d4d2da3f63 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 01:59:48.693033 6494 raft_consensus.cc:729] T 1d392e08c15a4d4098a350287b3163ef P 0c4821d9898d42799c5789d4d2da3f63 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0c4821d9898d42799c5789d4d2da3f63, State: Running, Role: LEADER
I20240503 01:59:48.693181 6494 consensus_queue.cc:260] T 1d392e08c15a4d4098a350287b3163ef P 0c4821d9898d42799c5789d4d2da3f63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } }
I20240503 01:59:48.693389 6494 raft_consensus.cc:3037] T 1d392e08c15a4d4098a350287b3163ef P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:48.694180 6494 raft_consensus.cc:2454] T 1d392e08c15a4d4098a350287b3163ef P 0c4821d9898d42799c5789d4d2da3f63 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65d4632bcdd54ba596acdbbe1b5ef8af in term 2.
I20240503 01:59:48.694586 8160 leader_election.cc:302] T 1d392e08c15a4d4098a350287b3163ef P 65d4632bcdd54ba596acdbbe1b5ef8af [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0c4821d9898d42799c5789d4d2da3f63, 65d4632bcdd54ba596acdbbe1b5ef8af, b6b1d5677b8d47788b7968f64b3dac3c; no voters:
I20240503 01:59:48.694815 12474 raft_consensus.cc:2781] T 1d392e08c15a4d4098a350287b3163ef P 65d4632bcdd54ba596acdbbe1b5ef8af [term 2 FOLLOWER]: Leader election won for term 2
I20240503 01:59:48.694988 12474 raft_consensus.cc:686] T 1d392e08c15a4d4098a350287b3163ef P 65d4632bcdd54ba596acdbbe1b5ef8af [term 2 LEADER]: Becoming Leader. State: Replica: 65d4632bcdd54ba596acdbbe1b5ef8af, State: Running, Role: LEADER
I20240503 01:59:48.695127 8355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d392e08c15a4d4098a350287b3163ef" candidate_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "de2f076398ee44008622fb5db2a75952"
I20240503 01:59:48.695150 12474 consensus_queue.cc:237] T 1d392e08c15a4d4098a350287b3163ef P 65d4632bcdd54ba596acdbbe1b5ef8af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } }
I20240503 01:59:48.695243 8355 raft_consensus.cc:3037] T 1d392e08c15a4d4098a350287b3163ef P de2f076398ee44008622fb5db2a75952 [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:48.696008 8355 raft_consensus.cc:2454] T 1d392e08c15a4d4098a350287b3163ef P de2f076398ee44008622fb5db2a75952 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65d4632bcdd54ba596acdbbe1b5ef8af in term 2.
I20240503 01:59:48.697129 6220 catalog_manager.cc:5484] T 1d392e08c15a4d4098a350287b3163ef P 65d4632bcdd54ba596acdbbe1b5ef8af reported cstate change: term changed from 1 to 2, leader changed from 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130) to 65d4632bcdd54ba596acdbbe1b5ef8af (127.0.99.134). New cstate: current_term: 2 leader_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:48.806777 12733 ts_tablet_manager.cc:902] T e5ed3e729bc64ebeac216ce5076b9fd9 P 2d0fd64aaa6548e09e1ebde581c54658: Initiating tablet copy from peer 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130:34189)
I20240503 01:59:48.807360 12733 tablet_copy_client.cc:308] T e5ed3e729bc64ebeac216ce5076b9fd9 P 2d0fd64aaa6548e09e1ebde581c54658: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:34189
I20240503 01:59:48.807827 6516 tablet_copy_service.cc:140] P 0c4821d9898d42799c5789d4d2da3f63: Received BeginTabletCopySession request for tablet e5ed3e729bc64ebeac216ce5076b9fd9 from peer 2d0fd64aaa6548e09e1ebde581c54658 ({username='slave'} at 127.0.99.137:56177)
I20240503 01:59:48.807936 6516 tablet_copy_service.cc:161] P 0c4821d9898d42799c5789d4d2da3f63: Beginning new tablet copy session on tablet e5ed3e729bc64ebeac216ce5076b9fd9 from peer 2d0fd64aaa6548e09e1ebde581c54658 at {username='slave'} at 127.0.99.137:56177: session id = 2d0fd64aaa6548e09e1ebde581c54658-e5ed3e729bc64ebeac216ce5076b9fd9
I20240503 01:59:48.809387 6516 tablet_copy_source_session.cc:205] T e5ed3e729bc64ebeac216ce5076b9fd9 P 0c4821d9898d42799c5789d4d2da3f63: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:48.810439 12733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5ed3e729bc64ebeac216ce5076b9fd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:48.812734 12733 tablet_copy_client.cc:783] T e5ed3e729bc64ebeac216ce5076b9fd9 P 2d0fd64aaa6548e09e1ebde581c54658: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:48.812903 12733 tablet_copy_client.cc:647] T e5ed3e729bc64ebeac216ce5076b9fd9 P 2d0fd64aaa6548e09e1ebde581c54658: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:48.813982 12738 tablet_copy_client.cc:1002] T e5ed3e729bc64ebeac216ce5076b9fd9 P 2d0fd64aaa6548e09e1ebde581c54658: tablet copy: Injecting latency into file download: 1500
I20240503 01:59:48.814299 8485 raft_consensus.cc:1261] T 21d27cca81e8455e9934e1cc604fc2b2 P 7a1404fe1c1b4b388b821c8261b53104 [term 2 FOLLOWER]: Refusing update from remote peer 9264f24d028347d98f7e4e2d1aedeb79: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 01:59:48.814920 12038 consensus_queue.cc:1035] T 21d27cca81e8455e9934e1cc604fc2b2 P 9264f24d028347d98f7e4e2d1aedeb79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 01:59:48.828487 6365 raft_consensus.cc:1261] T 21d27cca81e8455e9934e1cc604fc2b2 P 4127249ba42644388d225d614c935a25 [term 2 FOLLOWER]: Refusing update from remote peer 9264f24d028347d98f7e4e2d1aedeb79: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 01:59:48.829185 12038 consensus_queue.cc:1035] T 21d27cca81e8455e9934e1cc604fc2b2 P 9264f24d028347d98f7e4e2d1aedeb79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 01:59:48.833173 6625 raft_consensus.cc:1261] T 21d27cca81e8455e9934e1cc604fc2b2 P 79163c69f6f04a9abd0208d74e1c61be [term 2 FOLLOWER]: Refusing update from remote peer 9264f24d028347d98f7e4e2d1aedeb79: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 01:59:48.833684 12756 consensus_queue.cc:1035] T 21d27cca81e8455e9934e1cc604fc2b2 P 9264f24d028347d98f7e4e2d1aedeb79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 01:59:48.840960 8080 consensus_queue.cc:237] T 21d27cca81e8455e9934e1cc604fc2b2 P 9264f24d028347d98f7e4e2d1aedeb79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } }
I20240503 01:59:48.841563 12769 ts_tablet_manager.cc:902] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 2d0fd64aaa6548e09e1ebde581c54658: Initiating tablet copy from peer 4127249ba42644388d225d614c935a25 (127.0.99.129:35613)
I20240503 01:59:48.842279 6365 raft_consensus.cc:1261] T 21d27cca81e8455e9934e1cc604fc2b2 P 4127249ba42644388d225d614c935a25 [term 2 FOLLOWER]: Refusing update from remote peer 9264f24d028347d98f7e4e2d1aedeb79: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 01:59:48.842289 8485 raft_consensus.cc:1261] T 21d27cca81e8455e9934e1cc604fc2b2 P 7a1404fe1c1b4b388b821c8261b53104 [term 2 FOLLOWER]: Refusing update from remote peer 9264f24d028347d98f7e4e2d1aedeb79: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 01:59:48.842437 12769 tablet_copy_client.cc:308] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 2d0fd64aaa6548e09e1ebde581c54658: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:35613
I20240503 01:59:48.842792 12679 consensus_queue.cc:1035] T 21d27cca81e8455e9934e1cc604fc2b2 P 9264f24d028347d98f7e4e2d1aedeb79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 01:59:48.843000 12756 consensus_queue.cc:1035] T 21d27cca81e8455e9934e1cc604fc2b2 P 9264f24d028347d98f7e4e2d1aedeb79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 01:59:48.844882 12038 raft_consensus.cc:2932] T 21d27cca81e8455e9934e1cc604fc2b2 P 9264f24d028347d98f7e4e2d1aedeb79 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } }
I20240503 01:59:48.845237 6365 raft_consensus.cc:2932] T 21d27cca81e8455e9934e1cc604fc2b2 P 4127249ba42644388d225d614c935a25 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } }
I20240503 01:59:48.846419 6385 tablet_copy_service.cc:140] P 4127249ba42644388d225d614c935a25: Received BeginTabletCopySession request for tablet 2088c84e4eef4ccfbc70d7b668b8f2cf from peer 2d0fd64aaa6548e09e1ebde581c54658 ({username='slave'} at 127.0.99.137:52533)
I20240503 01:59:48.846508 6385 tablet_copy_service.cc:161] P 4127249ba42644388d225d614c935a25: Beginning new tablet copy session on tablet 2088c84e4eef4ccfbc70d7b668b8f2cf from peer 2d0fd64aaa6548e09e1ebde581c54658 at {username='slave'} at 127.0.99.137:52533: session id = 2d0fd64aaa6548e09e1ebde581c54658-2088c84e4eef4ccfbc70d7b668b8f2cf
I20240503 01:59:48.847550 6207 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 21d27cca81e8455e9934e1cc604fc2b2 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:48.847594 6385 tablet_copy_source_session.cc:205] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 4127249ba42644388d225d614c935a25: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:48.848160 6220 catalog_manager.cc:5484] T 21d27cca81e8455e9934e1cc604fc2b2 P 9264f24d028347d98f7e4e2d1aedeb79 reported cstate change: config changed from index 8 to 10, VOTER 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131) evicted. New cstate: current_term: 2 leader_uuid: "9264f24d028347d98f7e4e2d1aedeb79" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:48.848516 12769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2088c84e4eef4ccfbc70d7b668b8f2cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:48.848524 8485 raft_consensus.cc:2932] T 21d27cca81e8455e9934e1cc604fc2b2 P 7a1404fe1c1b4b388b821c8261b53104 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "7a1404fe1c1b4b388b821c8261b53104" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } }
I20240503 01:59:48.850611 12769 tablet_copy_client.cc:783] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 2d0fd64aaa6548e09e1ebde581c54658: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:48.850775 12769 tablet_copy_client.cc:647] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 2d0fd64aaa6548e09e1ebde581c54658: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:48.851681 6607 tablet_service.cc:1514] Processing DeleteTablet for tablet 21d27cca81e8455e9934e1cc604fc2b2 with delete_type TABLET_DATA_TOMBSTONED (TS 79163c69f6f04a9abd0208d74e1c61be not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:47560
I20240503 01:59:48.852059 12778 tablet_replica.cc:330] T 21d27cca81e8455e9934e1cc604fc2b2 P 79163c69f6f04a9abd0208d74e1c61be: stopping tablet replica
I20240503 01:59:48.852203 12778 raft_consensus.cc:2229] T 21d27cca81e8455e9934e1cc604fc2b2 P 79163c69f6f04a9abd0208d74e1c61be [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:48.852329 12778 raft_consensus.cc:2258] T 21d27cca81e8455e9934e1cc604fc2b2 P 79163c69f6f04a9abd0208d74e1c61be [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:48.852485 12773 tablet_copy_client.cc:1002] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 2d0fd64aaa6548e09e1ebde581c54658: tablet copy: Injecting latency into file download: 1500
I20240503 01:59:48.853929 12778 ts_tablet_manager.cc:1875] T 21d27cca81e8455e9934e1cc604fc2b2 P 79163c69f6f04a9abd0208d74e1c61be: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:48.856398 12778 ts_tablet_manager.cc:1888] T 21d27cca81e8455e9934e1cc604fc2b2 P 79163c69f6f04a9abd0208d74e1c61be: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240503 01:59:48.856468 12778 log.cc:1196] T 21d27cca81e8455e9934e1cc604fc2b2 P 79163c69f6f04a9abd0208d74e1c61be: Deleting WAL directory at /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/21d27cca81e8455e9934e1cc604fc2b2
I20240503 01:59:48.857101 6208 catalog_manager.cc:4830] TS 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131:44005): tablet 21d27cca81e8455e9934e1cc604fc2b2 (table rebalance_test_table_3 [id=2c07d364168a409a98aed9a7c3c1015c]) successfully deleted
I20240503 01:59:49.073194 6757 raft_consensus.cc:1261] T 1d392e08c15a4d4098a350287b3163ef P b6b1d5677b8d47788b7968f64b3dac3c [term 2 FOLLOWER]: Refusing update from remote peer 65d4632bcdd54ba596acdbbe1b5ef8af: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 01:59:49.073681 12474 consensus_queue.cc:1035] T 1d392e08c15a4d4098a350287b3163ef P 65d4632bcdd54ba596acdbbe1b5ef8af [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 01:59:49.147046 6494 raft_consensus.cc:1261] T 1d392e08c15a4d4098a350287b3163ef P 0c4821d9898d42799c5789d4d2da3f63 [term 2 FOLLOWER]: Refusing update from remote peer 65d4632bcdd54ba596acdbbe1b5ef8af: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 01:59:49.147465 12474 consensus_queue.cc:1035] T 1d392e08c15a4d4098a350287b3163ef P 65d4632bcdd54ba596acdbbe1b5ef8af [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 01:59:49.150897 8355 raft_consensus.cc:1261] T 1d392e08c15a4d4098a350287b3163ef P de2f076398ee44008622fb5db2a75952 [term 2 FOLLOWER]: Refusing update from remote peer 65d4632bcdd54ba596acdbbe1b5ef8af: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 01:59:49.152132 12728 consensus_queue.cc:1035] T 1d392e08c15a4d4098a350287b3163ef P 65d4632bcdd54ba596acdbbe1b5ef8af [LEADER]: Connected to new peer: Peer: permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 01:59:49.662035 12525 tablet_copy_client.cc:523] T 0bdcea4e59ec4e2da63b9316f9d718be P 2d0fd64aaa6548e09e1ebde581c54658: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:49.663388 12525 tablet_bootstrap.cc:492] T 0bdcea4e59ec4e2da63b9316f9d718be P 2d0fd64aaa6548e09e1ebde581c54658: Bootstrap starting.
I20240503 01:59:49.683209 12525 tablet_bootstrap.cc:492] T 0bdcea4e59ec4e2da63b9316f9d718be P 2d0fd64aaa6548e09e1ebde581c54658: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:49.683480 12525 tablet_bootstrap.cc:492] T 0bdcea4e59ec4e2da63b9316f9d718be P 2d0fd64aaa6548e09e1ebde581c54658: Bootstrap complete.
I20240503 01:59:49.683653 12525 ts_tablet_manager.cc:1367] T 0bdcea4e59ec4e2da63b9316f9d718be P 2d0fd64aaa6548e09e1ebde581c54658: Time spent bootstrapping tablet: real 0.020s user 0.018s sys 0.000s
I20240503 01:59:49.684075 12525 raft_consensus.cc:348] T 0bdcea4e59ec4e2da63b9316f9d718be P 2d0fd64aaa6548e09e1ebde581c54658 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: true } }
I20240503 01:59:49.684197 12525 raft_consensus.cc:729] T 0bdcea4e59ec4e2da63b9316f9d718be P 2d0fd64aaa6548e09e1ebde581c54658 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2d0fd64aaa6548e09e1ebde581c54658, State: Initialized, Role: LEARNER
I20240503 01:59:49.684309 12525 consensus_queue.cc:260] T 0bdcea4e59ec4e2da63b9316f9d718be P 2d0fd64aaa6548e09e1ebde581c54658 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: true } }
I20240503 01:59:49.684698 12525 ts_tablet_manager.cc:1398] T 0bdcea4e59ec4e2da63b9316f9d718be P 2d0fd64aaa6548e09e1ebde581c54658: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240503 01:59:49.685175 6647 tablet_copy_service.cc:342] P 79163c69f6f04a9abd0208d74e1c61be: Request end of tablet copy session 2d0fd64aaa6548e09e1ebde581c54658-0bdcea4e59ec4e2da63b9316f9d718be received from {username='slave'} at 127.0.99.137:43265
I20240503 01:59:49.685263 6647 tablet_copy_service.cc:433] P 79163c69f6f04a9abd0208d74e1c61be: ending tablet copy session 2d0fd64aaa6548e09e1ebde581c54658-0bdcea4e59ec4e2da63b9316f9d718be on tablet 0bdcea4e59ec4e2da63b9316f9d718be with peer 2d0fd64aaa6548e09e1ebde581c54658
I20240503 01:59:49.713963 8223 consensus_queue.cc:237] T 1d392e08c15a4d4098a350287b3163ef P 65d4632bcdd54ba596acdbbe1b5ef8af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } }
I20240503 01:59:49.715207 8355 raft_consensus.cc:1261] T 1d392e08c15a4d4098a350287b3163ef P de2f076398ee44008622fb5db2a75952 [term 2 FOLLOWER]: Refusing update from remote peer 65d4632bcdd54ba596acdbbe1b5ef8af: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 01:59:49.715238 6757 raft_consensus.cc:1261] T 1d392e08c15a4d4098a350287b3163ef P b6b1d5677b8d47788b7968f64b3dac3c [term 2 FOLLOWER]: Refusing update from remote peer 65d4632bcdd54ba596acdbbe1b5ef8af: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 01:59:49.715720 12918 consensus_queue.cc:1035] T 1d392e08c15a4d4098a350287b3163ef P 65d4632bcdd54ba596acdbbe1b5ef8af [LEADER]: Connected to new peer: Peer: permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 01:59:49.715910 12728 consensus_queue.cc:1035] T 1d392e08c15a4d4098a350287b3163ef P 65d4632bcdd54ba596acdbbe1b5ef8af [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 01:59:49.717475 12918 raft_consensus.cc:2932] T 1d392e08c15a4d4098a350287b3163ef P 65d4632bcdd54ba596acdbbe1b5ef8af [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } }
I20240503 01:59:49.717988 8355 raft_consensus.cc:2932] T 1d392e08c15a4d4098a350287b3163ef P de2f076398ee44008622fb5db2a75952 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } }
I20240503 01:59:49.720274 6214 catalog_manager.cc:5484] T 1d392e08c15a4d4098a350287b3163ef P 65d4632bcdd54ba596acdbbe1b5ef8af reported cstate change: config changed from index 8 to 10, VOTER 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130) evicted. New cstate: current_term: 2 leader_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:49.722965 6476 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d392e08c15a4d4098a350287b3163ef with delete_type TABLET_DATA_TOMBSTONED (TS 0c4821d9898d42799c5789d4d2da3f63 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:39850
I20240503 01:59:49.723299 12924 tablet_replica.cc:330] T 1d392e08c15a4d4098a350287b3163ef P 0c4821d9898d42799c5789d4d2da3f63: stopping tablet replica
I20240503 01:59:49.723502 12924 raft_consensus.cc:2229] T 1d392e08c15a4d4098a350287b3163ef P 0c4821d9898d42799c5789d4d2da3f63 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:49.723665 12924 raft_consensus.cc:2258] T 1d392e08c15a4d4098a350287b3163ef P 0c4821d9898d42799c5789d4d2da3f63 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:49.724050 6207 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1d392e08c15a4d4098a350287b3163ef with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:49.724283 12924 ts_tablet_manager.cc:1875] T 1d392e08c15a4d4098a350287b3163ef P 0c4821d9898d42799c5789d4d2da3f63: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:49.724431 6757 raft_consensus.cc:2932] T 1d392e08c15a4d4098a350287b3163ef P b6b1d5677b8d47788b7968f64b3dac3c [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } }
I20240503 01:59:49.726672 12924 ts_tablet_manager.cc:1888] T 1d392e08c15a4d4098a350287b3163ef P 0c4821d9898d42799c5789d4d2da3f63: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240503 01:59:49.726750 12924 log.cc:1196] T 1d392e08c15a4d4098a350287b3163ef P 0c4821d9898d42799c5789d4d2da3f63: Deleting WAL directory at /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/1d392e08c15a4d4098a350287b3163ef
I20240503 01:59:49.727306 6210 catalog_manager.cc:4830] TS 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130:34189): tablet 1d392e08c15a4d4098a350287b3163ef (table rebalance_test_table_5 [id=ba15a57b1f0e43fd9bae24116dc815d0]) successfully deleted
I20240503 01:59:49.747468 12587 tablet_copy_client.cc:523] T c3337c95b354418faf37684e7440b1c7 P 2d0fd64aaa6548e09e1ebde581c54658: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:49.748627 12587 tablet_bootstrap.cc:492] T c3337c95b354418faf37684e7440b1c7 P 2d0fd64aaa6548e09e1ebde581c54658: Bootstrap starting.
I20240503 01:59:49.757220 9410 raft_consensus.cc:1203] T 0bdcea4e59ec4e2da63b9316f9d718be P 2d0fd64aaa6548e09e1ebde581c54658 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 01:59:49.767470 12587 tablet_bootstrap.cc:492] T c3337c95b354418faf37684e7440b1c7 P 2d0fd64aaa6548e09e1ebde581c54658: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:49.767755 12587 tablet_bootstrap.cc:492] T c3337c95b354418faf37684e7440b1c7 P 2d0fd64aaa6548e09e1ebde581c54658: Bootstrap complete.
I20240503 01:59:49.767900 12587 ts_tablet_manager.cc:1367] T c3337c95b354418faf37684e7440b1c7 P 2d0fd64aaa6548e09e1ebde581c54658: Time spent bootstrapping tablet: real 0.019s user 0.013s sys 0.004s
I20240503 01:59:49.768285 12587 raft_consensus.cc:348] T c3337c95b354418faf37684e7440b1c7 P 2d0fd64aaa6548e09e1ebde581c54658 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: true } }
I20240503 01:59:49.768393 12587 raft_consensus.cc:729] T c3337c95b354418faf37684e7440b1c7 P 2d0fd64aaa6548e09e1ebde581c54658 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2d0fd64aaa6548e09e1ebde581c54658, State: Initialized, Role: LEARNER
I20240503 01:59:49.768464 12587 consensus_queue.cc:260] T c3337c95b354418faf37684e7440b1c7 P 2d0fd64aaa6548e09e1ebde581c54658 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: true } }
I20240503 01:59:49.768723 12587 ts_tablet_manager.cc:1398] T c3337c95b354418faf37684e7440b1c7 P 2d0fd64aaa6548e09e1ebde581c54658: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240503 01:59:49.769310 6647 tablet_copy_service.cc:342] P 79163c69f6f04a9abd0208d74e1c61be: Request end of tablet copy session 2d0fd64aaa6548e09e1ebde581c54658-c3337c95b354418faf37684e7440b1c7 received from {username='slave'} at 127.0.99.137:43265
I20240503 01:59:49.769430 6647 tablet_copy_service.cc:433] P 79163c69f6f04a9abd0208d74e1c61be: ending tablet copy session 2d0fd64aaa6548e09e1ebde581c54658-c3337c95b354418faf37684e7440b1c7 on tablet c3337c95b354418faf37684e7440b1c7 with peer 2d0fd64aaa6548e09e1ebde581c54658
I20240503 01:59:49.907517 9410 raft_consensus.cc:1203] T c3337c95b354418faf37684e7440b1c7 P 2d0fd64aaa6548e09e1ebde581c54658 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 01:59:50.134325 12721 tablet_copy_client.cc:523] T a388d59f6ba94e91958f55b340343fa6 P 2d0fd64aaa6548e09e1ebde581c54658: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:50.135879 12721 tablet_bootstrap.cc:492] T a388d59f6ba94e91958f55b340343fa6 P 2d0fd64aaa6548e09e1ebde581c54658: Bootstrap starting.
I20240503 01:59:50.157703 12721 tablet_bootstrap.cc:492] T a388d59f6ba94e91958f55b340343fa6 P 2d0fd64aaa6548e09e1ebde581c54658: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:50.157943 12721 tablet_bootstrap.cc:492] T a388d59f6ba94e91958f55b340343fa6 P 2d0fd64aaa6548e09e1ebde581c54658: Bootstrap complete.
I20240503 01:59:50.158107 12721 ts_tablet_manager.cc:1367] T a388d59f6ba94e91958f55b340343fa6 P 2d0fd64aaa6548e09e1ebde581c54658: Time spent bootstrapping tablet: real 0.022s user 0.021s sys 0.002s
I20240503 01:59:50.158504 12721 raft_consensus.cc:348] T a388d59f6ba94e91958f55b340343fa6 P 2d0fd64aaa6548e09e1ebde581c54658 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: true } }
I20240503 01:59:50.158607 12721 raft_consensus.cc:729] T a388d59f6ba94e91958f55b340343fa6 P 2d0fd64aaa6548e09e1ebde581c54658 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2d0fd64aaa6548e09e1ebde581c54658, State: Initialized, Role: LEARNER
I20240503 01:59:50.158707 12721 consensus_queue.cc:260] T a388d59f6ba94e91958f55b340343fa6 P 2d0fd64aaa6548e09e1ebde581c54658 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: true } }
I20240503 01:59:50.158937 12721 ts_tablet_manager.cc:1398] T a388d59f6ba94e91958f55b340343fa6 P 2d0fd64aaa6548e09e1ebde581c54658: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:50.159387 6647 tablet_copy_service.cc:342] P 79163c69f6f04a9abd0208d74e1c61be: Request end of tablet copy session 2d0fd64aaa6548e09e1ebde581c54658-a388d59f6ba94e91958f55b340343fa6 received from {username='slave'} at 127.0.99.137:43265
I20240503 01:59:50.159492 6647 tablet_copy_service.cc:433] P 79163c69f6f04a9abd0208d74e1c61be: ending tablet copy session 2d0fd64aaa6548e09e1ebde581c54658-a388d59f6ba94e91958f55b340343fa6 on tablet a388d59f6ba94e91958f55b340343fa6 with peer 2d0fd64aaa6548e09e1ebde581c54658
I20240503 01:59:50.203104 13009 raft_consensus.cc:1050] T 0bdcea4e59ec4e2da63b9316f9d718be P 79163c69f6f04a9abd0208d74e1c61be: attempting to promote NON_VOTER 2d0fd64aaa6548e09e1ebde581c54658 to VOTER
I20240503 01:59:50.203480 13009 consensus_queue.cc:237] T 0bdcea4e59ec4e2da63b9316f9d718be P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } }
I20240503 01:59:50.204775 9410 raft_consensus.cc:1261] T 0bdcea4e59ec4e2da63b9316f9d718be P 2d0fd64aaa6548e09e1ebde581c54658 [term 1 LEARNER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:50.204797 6757 raft_consensus.cc:1261] T 0bdcea4e59ec4e2da63b9316f9d718be P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:50.205168 8355 raft_consensus.cc:1261] T 0bdcea4e59ec4e2da63b9316f9d718be P de2f076398ee44008622fb5db2a75952 [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:50.205235 13011 consensus_queue.cc:1035] T 0bdcea4e59ec4e2da63b9316f9d718be P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:50.205618 9410 raft_consensus.cc:1203] T a388d59f6ba94e91958f55b340343fa6 P 2d0fd64aaa6548e09e1ebde581c54658 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 01:59:50.207090 13009 consensus_queue.cc:1035] T 0bdcea4e59ec4e2da63b9316f9d718be P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:50.207291 12165 consensus_queue.cc:1035] T 0bdcea4e59ec4e2da63b9316f9d718be P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:50.209534 13009 raft_consensus.cc:2932] T 0bdcea4e59ec4e2da63b9316f9d718be P 79163c69f6f04a9abd0208d74e1c61be [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2d0fd64aaa6548e09e1ebde581c54658 (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } } }
I20240503 01:59:50.209985 6757 raft_consensus.cc:2932] T 0bdcea4e59ec4e2da63b9316f9d718be P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2d0fd64aaa6548e09e1ebde581c54658 (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } } }
I20240503 01:59:50.211503 9410 raft_consensus.cc:2932] T 0bdcea4e59ec4e2da63b9316f9d718be P 2d0fd64aaa6548e09e1ebde581c54658 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2d0fd64aaa6548e09e1ebde581c54658 (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } } }
I20240503 01:59:50.212554 6220 catalog_manager.cc:5484] T 0bdcea4e59ec4e2da63b9316f9d718be P 79163c69f6f04a9abd0208d74e1c61be reported cstate change: config changed from index 7 to 8, 2d0fd64aaa6548e09e1ebde581c54658 (127.0.99.137) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "79163c69f6f04a9abd0208d74e1c61be" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:50.214673 8354 raft_consensus.cc:2932] T 0bdcea4e59ec4e2da63b9316f9d718be P de2f076398ee44008622fb5db2a75952 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2d0fd64aaa6548e09e1ebde581c54658 (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } } }
I20240503 01:59:50.217357 6625 consensus_queue.cc:237] T 0bdcea4e59ec4e2da63b9316f9d718be P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } }
I20240503 01:59:50.218288 8354 raft_consensus.cc:1261] T 0bdcea4e59ec4e2da63b9316f9d718be P de2f076398ee44008622fb5db2a75952 [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 01:59:50.218376 9410 raft_consensus.cc:1261] T 0bdcea4e59ec4e2da63b9316f9d718be P 2d0fd64aaa6548e09e1ebde581c54658 [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 01:59:50.218765 13011 consensus_queue.cc:1035] T 0bdcea4e59ec4e2da63b9316f9d718be P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 01:59:50.218933 12165 consensus_queue.cc:1035] T 0bdcea4e59ec4e2da63b9316f9d718be P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 01:59:50.220324 12162 raft_consensus.cc:2932] T 0bdcea4e59ec4e2da63b9316f9d718be P 79163c69f6f04a9abd0208d74e1c61be [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER b6b1d5677b8d47788b7968f64b3dac3c (127.0.99.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } } }
I20240503 01:59:50.220796 9410 raft_consensus.cc:2932] T 0bdcea4e59ec4e2da63b9316f9d718be P 2d0fd64aaa6548e09e1ebde581c54658 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER b6b1d5677b8d47788b7968f64b3dac3c (127.0.99.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } } }
I20240503 01:59:50.221701 6208 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0bdcea4e59ec4e2da63b9316f9d718be with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:50.222422 8354 raft_consensus.cc:2932] T 0bdcea4e59ec4e2da63b9316f9d718be P de2f076398ee44008622fb5db2a75952 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER b6b1d5677b8d47788b7968f64b3dac3c (127.0.99.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } } }
I20240503 01:59:50.222719 6221 catalog_manager.cc:5484] T 0bdcea4e59ec4e2da63b9316f9d718be P 79163c69f6f04a9abd0208d74e1c61be reported cstate change: config changed from index 8 to 9, VOTER b6b1d5677b8d47788b7968f64b3dac3c (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "79163c69f6f04a9abd0208d74e1c61be" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:50.224793 6738 tablet_service.cc:1514] Processing DeleteTablet for tablet 0bdcea4e59ec4e2da63b9316f9d718be with delete_type TABLET_DATA_TOMBSTONED (TS b6b1d5677b8d47788b7968f64b3dac3c not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:51980
I20240503 01:59:50.225096 13015 tablet_replica.cc:330] T 0bdcea4e59ec4e2da63b9316f9d718be P b6b1d5677b8d47788b7968f64b3dac3c: stopping tablet replica
I20240503 01:59:50.225251 13015 raft_consensus.cc:2229] T 0bdcea4e59ec4e2da63b9316f9d718be P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:50.225471 13015 raft_consensus.cc:2258] T 0bdcea4e59ec4e2da63b9316f9d718be P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:50.226049 13015 ts_tablet_manager.cc:1875] T 0bdcea4e59ec4e2da63b9316f9d718be P b6b1d5677b8d47788b7968f64b3dac3c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:50.227977 13015 ts_tablet_manager.cc:1888] T 0bdcea4e59ec4e2da63b9316f9d718be P b6b1d5677b8d47788b7968f64b3dac3c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 01:59:50.228057 13015 log.cc:1196] T 0bdcea4e59ec4e2da63b9316f9d718be P b6b1d5677b8d47788b7968f64b3dac3c: Deleting WAL directory at /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/0bdcea4e59ec4e2da63b9316f9d718be
I20240503 01:59:50.228664 6209 catalog_manager.cc:4830] TS b6b1d5677b8d47788b7968f64b3dac3c (127.0.99.132:46067): tablet 0bdcea4e59ec4e2da63b9316f9d718be (table rebalance_test_table_8 [id=85186d17f9464a408a3790b0978eb502]) successfully deleted
I20240503 01:59:50.314291 12733 tablet_copy_client.cc:523] T e5ed3e729bc64ebeac216ce5076b9fd9 P 2d0fd64aaa6548e09e1ebde581c54658: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:50.316124 12733 tablet_bootstrap.cc:492] T e5ed3e729bc64ebeac216ce5076b9fd9 P 2d0fd64aaa6548e09e1ebde581c54658: Bootstrap starting.
I20240503 01:59:50.324148 13011 raft_consensus.cc:1050] T c3337c95b354418faf37684e7440b1c7 P 79163c69f6f04a9abd0208d74e1c61be: attempting to promote NON_VOTER 2d0fd64aaa6548e09e1ebde581c54658 to VOTER
I20240503 01:59:50.324601 13011 consensus_queue.cc:237] T c3337c95b354418faf37684e7440b1c7 P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } }
I20240503 01:59:50.326476 8354 raft_consensus.cc:1261] T c3337c95b354418faf37684e7440b1c7 P de2f076398ee44008622fb5db2a75952 [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:50.326928 12165 consensus_queue.cc:1035] T c3337c95b354418faf37684e7440b1c7 P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:50.326951 6757 raft_consensus.cc:1261] T c3337c95b354418faf37684e7440b1c7 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:50.327468 12162 consensus_queue.cc:1035] T c3337c95b354418faf37684e7440b1c7 P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:50.327594 9410 raft_consensus.cc:1261] T c3337c95b354418faf37684e7440b1c7 P 2d0fd64aaa6548e09e1ebde581c54658 [term 1 LEARNER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:50.328549 12162 consensus_queue.cc:1035] T c3337c95b354418faf37684e7440b1c7 P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:50.330404 12165 raft_consensus.cc:2932] T c3337c95b354418faf37684e7440b1c7 P 79163c69f6f04a9abd0208d74e1c61be [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2d0fd64aaa6548e09e1ebde581c54658 (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } } }
I20240503 01:59:50.330868 9410 raft_consensus.cc:2932] T c3337c95b354418faf37684e7440b1c7 P 2d0fd64aaa6548e09e1ebde581c54658 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2d0fd64aaa6548e09e1ebde581c54658 (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } } }
I20240503 01:59:50.333127 6220 catalog_manager.cc:5484] T c3337c95b354418faf37684e7440b1c7 P 79163c69f6f04a9abd0208d74e1c61be reported cstate change: config changed from index 7 to 8, 2d0fd64aaa6548e09e1ebde581c54658 (127.0.99.137) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "79163c69f6f04a9abd0208d74e1c61be" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:50.333488 8354 raft_consensus.cc:2932] T c3337c95b354418faf37684e7440b1c7 P de2f076398ee44008622fb5db2a75952 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2d0fd64aaa6548e09e1ebde581c54658 (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } } }
I20240503 01:59:50.337622 6625 consensus_queue.cc:237] T c3337c95b354418faf37684e7440b1c7 P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } }
I20240503 01:59:50.339221 9410 raft_consensus.cc:1261] T c3337c95b354418faf37684e7440b1c7 P 2d0fd64aaa6548e09e1ebde581c54658 [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 01:59:50.339246 8354 raft_consensus.cc:1261] T c3337c95b354418faf37684e7440b1c7 P de2f076398ee44008622fb5db2a75952 [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 01:59:50.339629 13011 consensus_queue.cc:1035] T c3337c95b354418faf37684e7440b1c7 P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 01:59:50.339740 12162 consensus_queue.cc:1035] T c3337c95b354418faf37684e7440b1c7 P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 01:59:50.341707 12162 raft_consensus.cc:2932] T c3337c95b354418faf37684e7440b1c7 P 79163c69f6f04a9abd0208d74e1c61be [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER b6b1d5677b8d47788b7968f64b3dac3c (127.0.99.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } } }
I20240503 01:59:50.342200 9410 raft_consensus.cc:2932] T c3337c95b354418faf37684e7440b1c7 P 2d0fd64aaa6548e09e1ebde581c54658 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER b6b1d5677b8d47788b7968f64b3dac3c (127.0.99.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } } }
I20240503 01:59:50.343335 6208 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c3337c95b354418faf37684e7440b1c7 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:50.343786 8354 raft_consensus.cc:2932] T c3337c95b354418faf37684e7440b1c7 P de2f076398ee44008622fb5db2a75952 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER b6b1d5677b8d47788b7968f64b3dac3c (127.0.99.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } } }
I20240503 01:59:50.345427 6214 catalog_manager.cc:5484] T c3337c95b354418faf37684e7440b1c7 P 2d0fd64aaa6548e09e1ebde581c54658 reported cstate change: config changed from index 8 to 9, VOTER b6b1d5677b8d47788b7968f64b3dac3c (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "79163c69f6f04a9abd0208d74e1c61be" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } } }
I20240503 01:59:50.347847 6738 tablet_service.cc:1514] Processing DeleteTablet for tablet c3337c95b354418faf37684e7440b1c7 with delete_type TABLET_DATA_TOMBSTONED (TS b6b1d5677b8d47788b7968f64b3dac3c not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:51980
I20240503 01:59:50.348078 13015 tablet_replica.cc:330] T c3337c95b354418faf37684e7440b1c7 P b6b1d5677b8d47788b7968f64b3dac3c: stopping tablet replica
I20240503 01:59:50.348217 13015 raft_consensus.cc:2229] T c3337c95b354418faf37684e7440b1c7 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:50.348366 13015 pending_rounds.cc:62] T c3337c95b354418faf37684e7440b1c7 P b6b1d5677b8d47788b7968f64b3dac3c: Trying to abort 1 pending ops.
I20240503 01:59:50.348438 13015 pending_rounds.cc:68] T c3337c95b354418faf37684e7440b1c7 P b6b1d5677b8d47788b7968f64b3dac3c: Aborting op as it isn't in flight: id { term: 1 index: 8 } timestamp: 7023417713968308224 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "c3337c95b354418faf37684e7440b1c7" old_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: true } } } new_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } attrs { replace: true } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } } } }
I20240503 01:59:50.348577 13015 raft_consensus.cc:2894] T c3337c95b354418faf37684e7440b1c7 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Aborting config change with OpId 1.8: Aborted: Op aborted
I20240503 01:59:50.348634 13015 raft_consensus.cc:2866] T c3337c95b354418faf37684e7440b1c7 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240503 01:59:50.348723 13015 raft_consensus.cc:2258] T c3337c95b354418faf37684e7440b1c7 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:50.352816 12769 tablet_copy_client.cc:523] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 2d0fd64aaa6548e09e1ebde581c54658: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:50.354174 12769 tablet_bootstrap.cc:492] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 2d0fd64aaa6548e09e1ebde581c54658: Bootstrap starting.
W20240503 01:59:50.354336 13049 consensus_queue.cc:1159] T c3337c95b354418faf37684e7440b1c7 P b6b1d5677b8d47788b7968f64b3dac3c [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" status { last_received { term: 1 index: 8 } last_committed_idx: 7 last_received_current_leader { term: 1 index: 8 } }
I20240503 01:59:50.354993 13015 ts_tablet_manager.cc:1875] T c3337c95b354418faf37684e7440b1c7 P b6b1d5677b8d47788b7968f64b3dac3c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:50.356904 13015 ts_tablet_manager.cc:1888] T c3337c95b354418faf37684e7440b1c7 P b6b1d5677b8d47788b7968f64b3dac3c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 01:59:50.356971 13015 log.cc:1196] T c3337c95b354418faf37684e7440b1c7 P b6b1d5677b8d47788b7968f64b3dac3c: Deleting WAL directory at /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/c3337c95b354418faf37684e7440b1c7
I20240503 01:59:50.357573 6209 catalog_manager.cc:4830] TS b6b1d5677b8d47788b7968f64b3dac3c (127.0.99.132:46067): tablet c3337c95b354418faf37684e7440b1c7 (table rebalance_test_table_7 [id=dbd6f15c28634e3dac01249aa7cc2e25]) successfully deleted
I20240503 01:59:50.358811 12733 tablet_bootstrap.cc:492] T e5ed3e729bc64ebeac216ce5076b9fd9 P 2d0fd64aaa6548e09e1ebde581c54658: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:50.359081 12733 tablet_bootstrap.cc:492] T e5ed3e729bc64ebeac216ce5076b9fd9 P 2d0fd64aaa6548e09e1ebde581c54658: Bootstrap complete.
I20240503 01:59:50.359263 12733 ts_tablet_manager.cc:1367] T e5ed3e729bc64ebeac216ce5076b9fd9 P 2d0fd64aaa6548e09e1ebde581c54658: Time spent bootstrapping tablet: real 0.043s user 0.024s sys 0.000s
I20240503 01:59:50.359915 12733 raft_consensus.cc:348] T e5ed3e729bc64ebeac216ce5076b9fd9 P 2d0fd64aaa6548e09e1ebde581c54658 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: true } }
I20240503 01:59:50.360018 12733 raft_consensus.cc:729] T e5ed3e729bc64ebeac216ce5076b9fd9 P 2d0fd64aaa6548e09e1ebde581c54658 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2d0fd64aaa6548e09e1ebde581c54658, State: Initialized, Role: LEARNER
I20240503 01:59:50.360114 12733 consensus_queue.cc:260] T e5ed3e729bc64ebeac216ce5076b9fd9 P 2d0fd64aaa6548e09e1ebde581c54658 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: true } }
I20240503 01:59:50.360360 12733 ts_tablet_manager.cc:1398] T e5ed3e729bc64ebeac216ce5076b9fd9 P 2d0fd64aaa6548e09e1ebde581c54658: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240503 01:59:50.365130 6516 tablet_copy_service.cc:342] P 0c4821d9898d42799c5789d4d2da3f63: Request end of tablet copy session 2d0fd64aaa6548e09e1ebde581c54658-e5ed3e729bc64ebeac216ce5076b9fd9 received from {username='slave'} at 127.0.99.137:56177
I20240503 01:59:50.365222 6516 tablet_copy_service.cc:433] P 0c4821d9898d42799c5789d4d2da3f63: ending tablet copy session 2d0fd64aaa6548e09e1ebde581c54658-e5ed3e729bc64ebeac216ce5076b9fd9 on tablet e5ed3e729bc64ebeac216ce5076b9fd9 with peer 2d0fd64aaa6548e09e1ebde581c54658
I20240503 01:59:50.381621 12769 tablet_bootstrap.cc:492] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 2d0fd64aaa6548e09e1ebde581c54658: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:50.381951 12769 tablet_bootstrap.cc:492] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 2d0fd64aaa6548e09e1ebde581c54658: Bootstrap complete.
I20240503 01:59:50.382184 12769 ts_tablet_manager.cc:1367] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 2d0fd64aaa6548e09e1ebde581c54658: Time spent bootstrapping tablet: real 0.028s user 0.022s sys 0.004s
I20240503 01:59:50.382614 12769 raft_consensus.cc:348] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 2d0fd64aaa6548e09e1ebde581c54658 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: true } }
I20240503 01:59:50.382735 12769 raft_consensus.cc:729] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 2d0fd64aaa6548e09e1ebde581c54658 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2d0fd64aaa6548e09e1ebde581c54658, State: Initialized, Role: LEARNER
I20240503 01:59:50.382843 12769 consensus_queue.cc:260] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 2d0fd64aaa6548e09e1ebde581c54658 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: true } }
I20240503 01:59:50.383252 12769 ts_tablet_manager.cc:1398] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 2d0fd64aaa6548e09e1ebde581c54658: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:50.383809 6385 tablet_copy_service.cc:342] P 4127249ba42644388d225d614c935a25: Request end of tablet copy session 2d0fd64aaa6548e09e1ebde581c54658-2088c84e4eef4ccfbc70d7b668b8f2cf received from {username='slave'} at 127.0.99.137:52533
I20240503 01:59:50.383937 6385 tablet_copy_service.cc:433] P 4127249ba42644388d225d614c935a25: ending tablet copy session 2d0fd64aaa6548e09e1ebde581c54658-2088c84e4eef4ccfbc70d7b668b8f2cf on tablet 2088c84e4eef4ccfbc70d7b668b8f2cf with peer 2d0fd64aaa6548e09e1ebde581c54658
I20240503 01:59:50.448042 9410 raft_consensus.cc:1203] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 2d0fd64aaa6548e09e1ebde581c54658 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 01:59:50.709152 9410 raft_consensus.cc:1203] T e5ed3e729bc64ebeac216ce5076b9fd9 P 2d0fd64aaa6548e09e1ebde581c54658 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 01:59:50.805671 12165 raft_consensus.cc:1050] T a388d59f6ba94e91958f55b340343fa6 P 79163c69f6f04a9abd0208d74e1c61be: attempting to promote NON_VOTER 2d0fd64aaa6548e09e1ebde581c54658 to VOTER
I20240503 01:59:50.806126 12165 consensus_queue.cc:237] T a388d59f6ba94e91958f55b340343fa6 P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } }
I20240503 01:59:50.807642 9410 raft_consensus.cc:1261] T a388d59f6ba94e91958f55b340343fa6 P 2d0fd64aaa6548e09e1ebde581c54658 [term 1 LEARNER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:50.807734 6365 raft_consensus.cc:1261] T a388d59f6ba94e91958f55b340343fa6 P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:50.807919 8080 raft_consensus.cc:1261] T a388d59f6ba94e91958f55b340343fa6 P 9264f24d028347d98f7e4e2d1aedeb79 [term 1 FOLLOWER]: Refusing update from remote peer 79163c69f6f04a9abd0208d74e1c61be: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:50.808293 13011 consensus_queue.cc:1035] T a388d59f6ba94e91958f55b340343fa6 P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:50.808447 12162 consensus_queue.cc:1035] T a388d59f6ba94e91958f55b340343fa6 P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:50.809072 13009 consensus_queue.cc:1035] T a388d59f6ba94e91958f55b340343fa6 P 79163c69f6f04a9abd0208d74e1c61be [LEADER]: Connected to new peer: Peer: permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:50.810933 12162 raft_consensus.cc:2932] T a388d59f6ba94e91958f55b340343fa6 P 79163c69f6f04a9abd0208d74e1c61be [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2d0fd64aaa6548e09e1ebde581c54658 (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } } }
I20240503 01:59:50.811415 8080 raft_consensus.cc:2932] T a388d59f6ba94e91958f55b340343fa6 P 9264f24d028347d98f7e4e2d1aedeb79 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2d0fd64aaa6548e09e1ebde581c54658 (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } } }
I20240503 01:59:50.811894 9410 raft_consensus.cc:2932] T a388d59f6ba94e91958f55b340343fa6 P 2d0fd64aaa6548e09e1ebde581c54658 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2d0fd64aaa6548e09e1ebde581c54658 (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } } }
I20240503 01:59:50.813436 6221 catalog_manager.cc:5484] T a388d59f6ba94e91958f55b340343fa6 P 79163c69f6f04a9abd0208d74e1c61be reported cstate change: config changed from index 7 to 8, 2d0fd64aaa6548e09e1ebde581c54658 (127.0.99.137) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "79163c69f6f04a9abd0208d74e1c61be" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:50.813583 6365 raft_consensus.cc:2932] T a388d59f6ba94e91958f55b340343fa6 P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2d0fd64aaa6548e09e1ebde581c54658 (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } } }
I20240503 01:59:50.823019 6625 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a388d59f6ba94e91958f55b340343fa6"
dest_uuid: "79163c69f6f04a9abd0208d74e1c61be"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:56014
I20240503 01:59:50.823123 6625 raft_consensus.cc:595] T a388d59f6ba94e91958f55b340343fa6 P 79163c69f6f04a9abd0208d74e1c61be [term 1 LEADER]: Received request to transfer leadership
I20240503 01:59:50.854214 13124 raft_consensus.cc:1050] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 4127249ba42644388d225d614c935a25: attempting to promote NON_VOTER 2d0fd64aaa6548e09e1ebde581c54658 to VOTER
I20240503 01:59:50.854657 13124 consensus_queue.cc:237] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 4127249ba42644388d225d614c935a25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } }
I20240503 01:59:50.856163 8080 raft_consensus.cc:1261] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 9264f24d028347d98f7e4e2d1aedeb79 [term 1 FOLLOWER]: Refusing update from remote peer 4127249ba42644388d225d614c935a25: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:50.856191 9410 raft_consensus.cc:1261] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 2d0fd64aaa6548e09e1ebde581c54658 [term 1 LEARNER]: Refusing update from remote peer 4127249ba42644388d225d614c935a25: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:50.856304 6625 raft_consensus.cc:1261] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Refusing update from remote peer 4127249ba42644388d225d614c935a25: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:50.856547 12982 consensus_queue.cc:1035] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 4127249ba42644388d225d614c935a25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:50.856729 13123 consensus_queue.cc:1035] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 4127249ba42644388d225d614c935a25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:50.856846 13124 consensus_queue.cc:1035] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 4127249ba42644388d225d614c935a25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:50.859284 13123 raft_consensus.cc:2932] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 4127249ba42644388d225d614c935a25 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2d0fd64aaa6548e09e1ebde581c54658 (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } } }
I20240503 01:59:50.859757 9410 raft_consensus.cc:2932] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 2d0fd64aaa6548e09e1ebde581c54658 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2d0fd64aaa6548e09e1ebde581c54658 (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } } }
I20240503 01:59:50.860014 6625 raft_consensus.cc:2932] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2d0fd64aaa6548e09e1ebde581c54658 (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } } }
I20240503 01:59:50.861351 8080 raft_consensus.cc:2932] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 9264f24d028347d98f7e4e2d1aedeb79 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2d0fd64aaa6548e09e1ebde581c54658 (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } } }
I20240503 01:59:50.862610 6220 catalog_manager.cc:5484] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 2d0fd64aaa6548e09e1ebde581c54658 reported cstate change: config changed from index 7 to 8, 2d0fd64aaa6548e09e1ebde581c54658 (127.0.99.137) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4127249ba42644388d225d614c935a25" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } } }
I20240503 01:59:50.865990 6365 consensus_queue.cc:237] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 4127249ba42644388d225d614c935a25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } }
I20240503 01:59:50.867007 8080 raft_consensus.cc:1261] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 9264f24d028347d98f7e4e2d1aedeb79 [term 1 FOLLOWER]: Refusing update from remote peer 4127249ba42644388d225d614c935a25: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 01:59:50.867178 9410 raft_consensus.cc:1261] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 2d0fd64aaa6548e09e1ebde581c54658 [term 1 FOLLOWER]: Refusing update from remote peer 4127249ba42644388d225d614c935a25: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 01:59:50.867509 13129 consensus_queue.cc:1035] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 4127249ba42644388d225d614c935a25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 01:59:50.867892 13129 consensus_queue.cc:1035] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 4127249ba42644388d225d614c935a25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 01:59:50.869871 12982 raft_consensus.cc:2932] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 4127249ba42644388d225d614c935a25 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } } }
I20240503 01:59:50.870440 8080 raft_consensus.cc:2932] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 9264f24d028347d98f7e4e2d1aedeb79 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } } }
I20240503 01:59:50.871347 6208 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2088c84e4eef4ccfbc70d7b668b8f2cf with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:50.870244 9410 raft_consensus.cc:2932] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 2d0fd64aaa6548e09e1ebde581c54658 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } } }
I20240503 01:59:50.872222 6214 catalog_manager.cc:5484] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 4127249ba42644388d225d614c935a25 reported cstate change: config changed from index 8 to 9, VOTER 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "4127249ba42644388d225d614c935a25" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:50.874631 6607 tablet_service.cc:1514] Processing DeleteTablet for tablet 2088c84e4eef4ccfbc70d7b668b8f2cf with delete_type TABLET_DATA_TOMBSTONED (TS 79163c69f6f04a9abd0208d74e1c61be not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:47560
I20240503 01:59:50.875087 13130 tablet_replica.cc:330] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 79163c69f6f04a9abd0208d74e1c61be: stopping tablet replica
I20240503 01:59:50.875268 13130 raft_consensus.cc:2229] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:50.875586 13130 raft_consensus.cc:2258] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:50.876246 13130 ts_tablet_manager.cc:1875] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 79163c69f6f04a9abd0208d74e1c61be: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:50.877909 13130 ts_tablet_manager.cc:1888] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 79163c69f6f04a9abd0208d74e1c61be: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 01:59:50.877971 13130 log.cc:1196] T 2088c84e4eef4ccfbc70d7b668b8f2cf P 79163c69f6f04a9abd0208d74e1c61be: Deleting WAL directory at /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/2088c84e4eef4ccfbc70d7b668b8f2cf
I20240503 01:59:50.878415 6208 catalog_manager.cc:4830] TS 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131:44005): tablet 2088c84e4eef4ccfbc70d7b668b8f2cf (table rebalance_test_table_5 [id=ba15a57b1f0e43fd9bae24116dc815d0]) successfully deleted
I20240503 01:59:51.045286 6625 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a388d59f6ba94e91958f55b340343fa6"
dest_uuid: "79163c69f6f04a9abd0208d74e1c61be"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:56036
I20240503 01:59:51.045419 6625 raft_consensus.cc:595] T a388d59f6ba94e91958f55b340343fa6 P 79163c69f6f04a9abd0208d74e1c61be [term 1 LEADER]: Received request to transfer leadership
I20240503 01:59:51.060920 6625 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a388d59f6ba94e91958f55b340343fa6"
dest_uuid: "79163c69f6f04a9abd0208d74e1c61be"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:56058
I20240503 01:59:51.061049 6625 raft_consensus.cc:595] T a388d59f6ba94e91958f55b340343fa6 P 79163c69f6f04a9abd0208d74e1c61be [term 1 LEADER]: Received request to transfer leadership
I20240503 01:59:51.188357 12162 raft_consensus.cc:979] T a388d59f6ba94e91958f55b340343fa6 P 79163c69f6f04a9abd0208d74e1c61be: : Instructing follower 2d0fd64aaa6548e09e1ebde581c54658 to start an election
I20240503 01:59:51.188457 12162 raft_consensus.cc:1067] T a388d59f6ba94e91958f55b340343fa6 P 79163c69f6f04a9abd0208d74e1c61be [term 1 LEADER]: Signalling peer 2d0fd64aaa6548e09e1ebde581c54658 to start an election
I20240503 01:59:51.188871 9410 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a388d59f6ba94e91958f55b340343fa6"
dest_uuid: "2d0fd64aaa6548e09e1ebde581c54658"
from {username='slave'} at 127.0.99.131:39699
I20240503 01:59:51.189049 9410 raft_consensus.cc:482] T a388d59f6ba94e91958f55b340343fa6 P 2d0fd64aaa6548e09e1ebde581c54658 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 01:59:51.189155 9410 raft_consensus.cc:3037] T a388d59f6ba94e91958f55b340343fa6 P 2d0fd64aaa6548e09e1ebde581c54658 [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:51.190016 9410 raft_consensus.cc:504] T a388d59f6ba94e91958f55b340343fa6 P 2d0fd64aaa6548e09e1ebde581c54658 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } }
I20240503 01:59:51.191009 9410 leader_election.cc:288] T a388d59f6ba94e91958f55b340343fa6 P 2d0fd64aaa6548e09e1ebde581c54658 [CANDIDATE]: Term 2 election: Requested vote from peers 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131:44005), 4127249ba42644388d225d614c935a25 (127.0.99.129:35613), 9264f24d028347d98f7e4e2d1aedeb79 (127.0.99.133:41713)
I20240503 01:59:51.191403 6365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a388d59f6ba94e91958f55b340343fa6" candidate_uuid: "2d0fd64aaa6548e09e1ebde581c54658" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "4127249ba42644388d225d614c935a25"
I20240503 01:59:51.191406 6625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a388d59f6ba94e91958f55b340343fa6" candidate_uuid: "2d0fd64aaa6548e09e1ebde581c54658" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "79163c69f6f04a9abd0208d74e1c61be"
I20240503 01:59:51.191545 6365 raft_consensus.cc:3037] T a388d59f6ba94e91958f55b340343fa6 P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:51.191560 6625 raft_consensus.cc:3032] T a388d59f6ba94e91958f55b340343fa6 P 79163c69f6f04a9abd0208d74e1c61be [term 1 LEADER]: Stepping down as leader of term 1
I20240503 01:59:51.191610 6625 raft_consensus.cc:729] T a388d59f6ba94e91958f55b340343fa6 P 79163c69f6f04a9abd0208d74e1c61be [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 79163c69f6f04a9abd0208d74e1c61be, State: Running, Role: LEADER
I20240503 01:59:51.191781 6625 consensus_queue.cc:260] T a388d59f6ba94e91958f55b340343fa6 P 79163c69f6f04a9abd0208d74e1c61be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } }
I20240503 01:59:51.192003 6625 raft_consensus.cc:3037] T a388d59f6ba94e91958f55b340343fa6 P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:51.192559 6365 raft_consensus.cc:2454] T a388d59f6ba94e91958f55b340343fa6 P 4127249ba42644388d225d614c935a25 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d0fd64aaa6548e09e1ebde581c54658 in term 2.
I20240503 01:59:51.193020 6625 raft_consensus.cc:2454] T a388d59f6ba94e91958f55b340343fa6 P 79163c69f6f04a9abd0208d74e1c61be [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d0fd64aaa6548e09e1ebde581c54658 in term 2.
I20240503 01:59:51.193451 9339 leader_election.cc:302] T a388d59f6ba94e91958f55b340343fa6 P 2d0fd64aaa6548e09e1ebde581c54658 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 2d0fd64aaa6548e09e1ebde581c54658, 4127249ba42644388d225d614c935a25, 79163c69f6f04a9abd0208d74e1c61be; no voters:
I20240503 01:59:51.193675 12917 raft_consensus.cc:2781] T a388d59f6ba94e91958f55b340343fa6 P 2d0fd64aaa6548e09e1ebde581c54658 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 01:59:51.193877 12917 raft_consensus.cc:686] T a388d59f6ba94e91958f55b340343fa6 P 2d0fd64aaa6548e09e1ebde581c54658 [term 2 LEADER]: Becoming Leader. State: Replica: 2d0fd64aaa6548e09e1ebde581c54658, State: Running, Role: LEADER
I20240503 01:59:51.194164 12917 consensus_queue.cc:237] T a388d59f6ba94e91958f55b340343fa6 P 2d0fd64aaa6548e09e1ebde581c54658 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } }
I20240503 01:59:51.195947 8080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a388d59f6ba94e91958f55b340343fa6" candidate_uuid: "2d0fd64aaa6548e09e1ebde581c54658" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "9264f24d028347d98f7e4e2d1aedeb79"
I20240503 01:59:51.196108 8080 raft_consensus.cc:3037] T a388d59f6ba94e91958f55b340343fa6 P 9264f24d028347d98f7e4e2d1aedeb79 [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:51.196797 6219 catalog_manager.cc:5484] T a388d59f6ba94e91958f55b340343fa6 P 2d0fd64aaa6548e09e1ebde581c54658 reported cstate change: term changed from 1 to 2, leader changed from 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131) to 2d0fd64aaa6548e09e1ebde581c54658 (127.0.99.137). New cstate: current_term: 2 leader_uuid: "2d0fd64aaa6548e09e1ebde581c54658" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:51.196898 8080 raft_consensus.cc:2454] T a388d59f6ba94e91958f55b340343fa6 P 9264f24d028347d98f7e4e2d1aedeb79 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d0fd64aaa6548e09e1ebde581c54658 in term 2.
I20240503 01:59:51.334415 13180 raft_consensus.cc:1050] T e5ed3e729bc64ebeac216ce5076b9fd9 P 0c4821d9898d42799c5789d4d2da3f63: attempting to promote NON_VOTER 2d0fd64aaa6548e09e1ebde581c54658 to VOTER
I20240503 01:59:51.334852 13180 consensus_queue.cc:237] T e5ed3e729bc64ebeac216ce5076b9fd9 P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } }
I20240503 01:59:51.336127 8354 raft_consensus.cc:1261] T e5ed3e729bc64ebeac216ce5076b9fd9 P de2f076398ee44008622fb5db2a75952 [term 1 FOLLOWER]: Refusing update from remote peer 0c4821d9898d42799c5789d4d2da3f63: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:51.336127 9410 raft_consensus.cc:1261] T e5ed3e729bc64ebeac216ce5076b9fd9 P 2d0fd64aaa6548e09e1ebde581c54658 [term 1 LEARNER]: Refusing update from remote peer 0c4821d9898d42799c5789d4d2da3f63: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:51.336534 13181 consensus_queue.cc:1035] T e5ed3e729bc64ebeac216ce5076b9fd9 P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:51.336812 6757 raft_consensus.cc:1261] T e5ed3e729bc64ebeac216ce5076b9fd9 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Refusing update from remote peer 0c4821d9898d42799c5789d4d2da3f63: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:51.337262 13180 consensus_queue.cc:1035] T e5ed3e729bc64ebeac216ce5076b9fd9 P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:51.337527 13181 consensus_queue.cc:1035] T e5ed3e729bc64ebeac216ce5076b9fd9 P 0c4821d9898d42799c5789d4d2da3f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:51.339731 13180 raft_consensus.cc:2932] T e5ed3e729bc64ebeac216ce5076b9fd9 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2d0fd64aaa6548e09e1ebde581c54658 (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } } }
I20240503 01:59:51.340160 9410 raft_consensus.cc:2932] T e5ed3e729bc64ebeac216ce5076b9fd9 P 2d0fd64aaa6548e09e1ebde581c54658 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2d0fd64aaa6548e09e1ebde581c54658 (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } } }
I20240503 01:59:51.340185 6757 raft_consensus.cc:2932] T e5ed3e729bc64ebeac216ce5076b9fd9 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2d0fd64aaa6548e09e1ebde581c54658 (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } } }
I20240503 01:59:51.341478 8354 raft_consensus.cc:2932] T e5ed3e729bc64ebeac216ce5076b9fd9 P de2f076398ee44008622fb5db2a75952 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2d0fd64aaa6548e09e1ebde581c54658 (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } } }
I20240503 01:59:51.342703 6219 catalog_manager.cc:5484] T e5ed3e729bc64ebeac216ce5076b9fd9 P 0c4821d9898d42799c5789d4d2da3f63 reported cstate change: config changed from index 7 to 8, 2d0fd64aaa6548e09e1ebde581c54658 (127.0.99.137) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0c4821d9898d42799c5789d4d2da3f63" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:51.495940 6494 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e5ed3e729bc64ebeac216ce5076b9fd9"
dest_uuid: "0c4821d9898d42799c5789d4d2da3f63"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:56854
I20240503 01:59:51.496078 6494 raft_consensus.cc:595] T e5ed3e729bc64ebeac216ce5076b9fd9 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 LEADER]: Received request to transfer leadership
I20240503 01:59:51.655586 6625 raft_consensus.cc:1261] T a388d59f6ba94e91958f55b340343fa6 P 79163c69f6f04a9abd0208d74e1c61be [term 2 FOLLOWER]: Refusing update from remote peer 2d0fd64aaa6548e09e1ebde581c54658: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 01:59:51.656100 12917 consensus_queue.cc:1035] T a388d59f6ba94e91958f55b340343fa6 P 2d0fd64aaa6548e09e1ebde581c54658 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 01:59:51.716522 8080 raft_consensus.cc:1261] T a388d59f6ba94e91958f55b340343fa6 P 9264f24d028347d98f7e4e2d1aedeb79 [term 2 FOLLOWER]: Refusing update from remote peer 2d0fd64aaa6548e09e1ebde581c54658: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 01:59:51.716965 12917 consensus_queue.cc:1035] T a388d59f6ba94e91958f55b340343fa6 P 2d0fd64aaa6548e09e1ebde581c54658 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 01:59:51.720196 6365 raft_consensus.cc:1261] T a388d59f6ba94e91958f55b340343fa6 P 4127249ba42644388d225d614c935a25 [term 2 FOLLOWER]: Refusing update from remote peer 2d0fd64aaa6548e09e1ebde581c54658: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 01:59:51.720908 13220 consensus_queue.cc:1035] T a388d59f6ba94e91958f55b340343fa6 P 2d0fd64aaa6548e09e1ebde581c54658 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 01:59:51.726722 6494 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e5ed3e729bc64ebeac216ce5076b9fd9"
dest_uuid: "0c4821d9898d42799c5789d4d2da3f63"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:56864
I20240503 01:59:51.726866 6494 raft_consensus.cc:595] T e5ed3e729bc64ebeac216ce5076b9fd9 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 LEADER]: Received request to transfer leadership
I20240503 01:59:51.733048 13183 raft_consensus.cc:979] T e5ed3e729bc64ebeac216ce5076b9fd9 P 0c4821d9898d42799c5789d4d2da3f63: : Instructing follower 2d0fd64aaa6548e09e1ebde581c54658 to start an election
I20240503 01:59:51.733137 13183 raft_consensus.cc:1067] T e5ed3e729bc64ebeac216ce5076b9fd9 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 LEADER]: Signalling peer 2d0fd64aaa6548e09e1ebde581c54658 to start an election
I20240503 01:59:51.733518 9410 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e5ed3e729bc64ebeac216ce5076b9fd9"
dest_uuid: "2d0fd64aaa6548e09e1ebde581c54658"
from {username='slave'} at 127.0.99.130:56853
I20240503 01:59:51.733619 9410 raft_consensus.cc:482] T e5ed3e729bc64ebeac216ce5076b9fd9 P 2d0fd64aaa6548e09e1ebde581c54658 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 01:59:51.733682 9410 raft_consensus.cc:3037] T e5ed3e729bc64ebeac216ce5076b9fd9 P 2d0fd64aaa6548e09e1ebde581c54658 [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:51.734447 9410 raft_consensus.cc:504] T e5ed3e729bc64ebeac216ce5076b9fd9 P 2d0fd64aaa6548e09e1ebde581c54658 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } }
I20240503 01:59:51.734982 9410 leader_election.cc:288] T e5ed3e729bc64ebeac216ce5076b9fd9 P 2d0fd64aaa6548e09e1ebde581c54658 [CANDIDATE]: Term 2 election: Requested vote from peers 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130:34189), b6b1d5677b8d47788b7968f64b3dac3c (127.0.99.132:46067), de2f076398ee44008622fb5db2a75952 (127.0.99.135:34035)
I20240503 01:59:51.735220 6494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ed3e729bc64ebeac216ce5076b9fd9" candidate_uuid: "2d0fd64aaa6548e09e1ebde581c54658" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "0c4821d9898d42799c5789d4d2da3f63"
I20240503 01:59:51.735316 6494 raft_consensus.cc:3032] T e5ed3e729bc64ebeac216ce5076b9fd9 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 01:59:51.735353 6494 raft_consensus.cc:729] T e5ed3e729bc64ebeac216ce5076b9fd9 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0c4821d9898d42799c5789d4d2da3f63, State: Running, Role: LEADER
I20240503 01:59:51.735384 6757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ed3e729bc64ebeac216ce5076b9fd9" candidate_uuid: "2d0fd64aaa6548e09e1ebde581c54658" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "b6b1d5677b8d47788b7968f64b3dac3c"
I20240503 01:59:51.735509 6757 raft_consensus.cc:3037] T e5ed3e729bc64ebeac216ce5076b9fd9 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:51.735497 6494 consensus_queue.cc:260] T e5ed3e729bc64ebeac216ce5076b9fd9 P 0c4821d9898d42799c5789d4d2da3f63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } }
I20240503 01:59:51.735734 6494 raft_consensus.cc:3037] T e5ed3e729bc64ebeac216ce5076b9fd9 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:51.736547 6757 raft_consensus.cc:2454] T e5ed3e729bc64ebeac216ce5076b9fd9 P b6b1d5677b8d47788b7968f64b3dac3c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d0fd64aaa6548e09e1ebde581c54658 in term 2.
I20240503 01:59:51.736706 6494 raft_consensus.cc:2454] T e5ed3e729bc64ebeac216ce5076b9fd9 P 0c4821d9898d42799c5789d4d2da3f63 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d0fd64aaa6548e09e1ebde581c54658 in term 2.
I20240503 01:59:51.737023 9341 leader_election.cc:302] T e5ed3e729bc64ebeac216ce5076b9fd9 P 2d0fd64aaa6548e09e1ebde581c54658 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0c4821d9898d42799c5789d4d2da3f63, 2d0fd64aaa6548e09e1ebde581c54658, b6b1d5677b8d47788b7968f64b3dac3c; no voters:
I20240503 01:59:51.737214 13218 raft_consensus.cc:2781] T e5ed3e729bc64ebeac216ce5076b9fd9 P 2d0fd64aaa6548e09e1ebde581c54658 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 01:59:51.737386 13218 raft_consensus.cc:686] T e5ed3e729bc64ebeac216ce5076b9fd9 P 2d0fd64aaa6548e09e1ebde581c54658 [term 2 LEADER]: Becoming Leader. State: Replica: 2d0fd64aaa6548e09e1ebde581c54658, State: Running, Role: LEADER
I20240503 01:59:51.737527 13218 consensus_queue.cc:237] T e5ed3e729bc64ebeac216ce5076b9fd9 P 2d0fd64aaa6548e09e1ebde581c54658 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } }
I20240503 01:59:51.739293 6214 catalog_manager.cc:5484] T e5ed3e729bc64ebeac216ce5076b9fd9 P 2d0fd64aaa6548e09e1ebde581c54658 reported cstate change: term changed from 1 to 2, leader changed from 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130) to 2d0fd64aaa6548e09e1ebde581c54658 (127.0.99.137). New cstate: current_term: 2 leader_uuid: "2d0fd64aaa6548e09e1ebde581c54658" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:51.740288 8354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ed3e729bc64ebeac216ce5076b9fd9" candidate_uuid: "2d0fd64aaa6548e09e1ebde581c54658" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "de2f076398ee44008622fb5db2a75952"
I20240503 01:59:51.740406 8354 raft_consensus.cc:3037] T e5ed3e729bc64ebeac216ce5076b9fd9 P de2f076398ee44008622fb5db2a75952 [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:51.741365 8354 raft_consensus.cc:2454] T e5ed3e729bc64ebeac216ce5076b9fd9 P de2f076398ee44008622fb5db2a75952 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d0fd64aaa6548e09e1ebde581c54658 in term 2.
I20240503 01:59:52.124802 8354 raft_consensus.cc:1261] T e5ed3e729bc64ebeac216ce5076b9fd9 P de2f076398ee44008622fb5db2a75952 [term 2 FOLLOWER]: Refusing update from remote peer 2d0fd64aaa6548e09e1ebde581c54658: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 01:59:52.125238 13218 consensus_queue.cc:1035] T e5ed3e729bc64ebeac216ce5076b9fd9 P 2d0fd64aaa6548e09e1ebde581c54658 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 01:59:52.134589 6757 raft_consensus.cc:1261] T e5ed3e729bc64ebeac216ce5076b9fd9 P b6b1d5677b8d47788b7968f64b3dac3c [term 2 FOLLOWER]: Refusing update from remote peer 2d0fd64aaa6548e09e1ebde581c54658: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 01:59:52.135061 13218 consensus_queue.cc:1035] T e5ed3e729bc64ebeac216ce5076b9fd9 P 2d0fd64aaa6548e09e1ebde581c54658 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 01:59:52.137955 6494 raft_consensus.cc:1261] T e5ed3e729bc64ebeac216ce5076b9fd9 P 0c4821d9898d42799c5789d4d2da3f63 [term 2 FOLLOWER]: Refusing update from remote peer 2d0fd64aaa6548e09e1ebde581c54658: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 01:59:52.138356 13220 consensus_queue.cc:1035] T e5ed3e729bc64ebeac216ce5076b9fd9 P 2d0fd64aaa6548e09e1ebde581c54658 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 01:59:52.149966 9410 consensus_queue.cc:237] T e5ed3e729bc64ebeac216ce5076b9fd9 P 2d0fd64aaa6548e09e1ebde581c54658 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } }
I20240503 01:59:52.150935 9409 consensus_queue.cc:237] T a388d59f6ba94e91958f55b340343fa6 P 2d0fd64aaa6548e09e1ebde581c54658 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } }
I20240503 01:59:52.153115 6757 raft_consensus.cc:1261] T e5ed3e729bc64ebeac216ce5076b9fd9 P b6b1d5677b8d47788b7968f64b3dac3c [term 2 FOLLOWER]: Refusing update from remote peer 2d0fd64aaa6548e09e1ebde581c54658: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 01:59:52.153231 8354 raft_consensus.cc:1261] T e5ed3e729bc64ebeac216ce5076b9fd9 P de2f076398ee44008622fb5db2a75952 [term 2 FOLLOWER]: Refusing update from remote peer 2d0fd64aaa6548e09e1ebde581c54658: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 01:59:52.153506 13220 consensus_queue.cc:1035] T e5ed3e729bc64ebeac216ce5076b9fd9 P 2d0fd64aaa6548e09e1ebde581c54658 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 01:59:52.154006 12917 consensus_queue.cc:1035] T e5ed3e729bc64ebeac216ce5076b9fd9 P 2d0fd64aaa6548e09e1ebde581c54658 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 01:59:52.154171 8093 raft_consensus.cc:1261] T a388d59f6ba94e91958f55b340343fa6 P 9264f24d028347d98f7e4e2d1aedeb79 [term 2 FOLLOWER]: Refusing update from remote peer 2d0fd64aaa6548e09e1ebde581c54658: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 01:59:52.154503 6365 raft_consensus.cc:1261] T a388d59f6ba94e91958f55b340343fa6 P 4127249ba42644388d225d614c935a25 [term 2 FOLLOWER]: Refusing update from remote peer 2d0fd64aaa6548e09e1ebde581c54658: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 01:59:52.154525 12917 consensus_queue.cc:1035] T a388d59f6ba94e91958f55b340343fa6 P 2d0fd64aaa6548e09e1ebde581c54658 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 01:59:52.154937 12917 consensus_queue.cc:1035] T a388d59f6ba94e91958f55b340343fa6 P 2d0fd64aaa6548e09e1ebde581c54658 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 01:59:52.155906 13218 raft_consensus.cc:2932] T e5ed3e729bc64ebeac216ce5076b9fd9 P 2d0fd64aaa6548e09e1ebde581c54658 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } } }
I20240503 01:59:52.156337 8354 raft_consensus.cc:2932] T e5ed3e729bc64ebeac216ce5076b9fd9 P de2f076398ee44008622fb5db2a75952 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } } }
I20240503 01:59:52.159565 6221 catalog_manager.cc:5484] T e5ed3e729bc64ebeac216ce5076b9fd9 P de2f076398ee44008622fb5db2a75952 reported cstate change: config changed from index 8 to 10, VOTER 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130) evicted. New cstate: current_term: 2 leader_uuid: "2d0fd64aaa6548e09e1ebde581c54658" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } } }
I20240503 01:59:52.160502 6757 raft_consensus.cc:2932] T e5ed3e729bc64ebeac216ce5076b9fd9 P b6b1d5677b8d47788b7968f64b3dac3c [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } } }
I20240503 01:59:52.161146 12917 raft_consensus.cc:2932] T a388d59f6ba94e91958f55b340343fa6 P 2d0fd64aaa6548e09e1ebde581c54658 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } } }
I20240503 01:59:52.161557 6365 raft_consensus.cc:2932] T a388d59f6ba94e91958f55b340343fa6 P 4127249ba42644388d225d614c935a25 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } } }
I20240503 01:59:52.163218 6476 tablet_service.cc:1514] Processing DeleteTablet for tablet e5ed3e729bc64ebeac216ce5076b9fd9 with delete_type TABLET_DATA_TOMBSTONED (TS 0c4821d9898d42799c5789d4d2da3f63 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:39850
I20240503 01:59:52.164247 13248 tablet_replica.cc:330] T e5ed3e729bc64ebeac216ce5076b9fd9 P 0c4821d9898d42799c5789d4d2da3f63: stopping tablet replica
I20240503 01:59:52.164402 13248 raft_consensus.cc:2229] T e5ed3e729bc64ebeac216ce5076b9fd9 P 0c4821d9898d42799c5789d4d2da3f63 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:52.164506 13248 raft_consensus.cc:2258] T e5ed3e729bc64ebeac216ce5076b9fd9 P 0c4821d9898d42799c5789d4d2da3f63 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:52.164371 8093 raft_consensus.cc:2932] T a388d59f6ba94e91958f55b340343fa6 P 9264f24d028347d98f7e4e2d1aedeb79 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } } }
I20240503 01:59:52.165273 13248 ts_tablet_manager.cc:1875] T e5ed3e729bc64ebeac216ce5076b9fd9 P 0c4821d9898d42799c5789d4d2da3f63: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:52.165177 6214 catalog_manager.cc:5484] T a388d59f6ba94e91958f55b340343fa6 P 4127249ba42644388d225d614c935a25 reported cstate change: config changed from index 8 to 10, VOTER 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131) evicted. New cstate: current_term: 2 leader_uuid: "2d0fd64aaa6548e09e1ebde581c54658" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } } peers { permanent_uuid: "9264f24d028347d98f7e4e2d1aedeb79" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "2d0fd64aaa6548e09e1ebde581c54658" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 40613 } attrs { promote: false } } }
I20240503 01:59:52.168301 6607 tablet_service.cc:1514] Processing DeleteTablet for tablet a388d59f6ba94e91958f55b340343fa6 with delete_type TABLET_DATA_TOMBSTONED (TS 79163c69f6f04a9abd0208d74e1c61be not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:47560
I20240503 01:59:52.168629 13256 tablet_replica.cc:330] T a388d59f6ba94e91958f55b340343fa6 P 79163c69f6f04a9abd0208d74e1c61be: stopping tablet replica
I20240503 01:59:52.168843 13256 raft_consensus.cc:2229] T a388d59f6ba94e91958f55b340343fa6 P 79163c69f6f04a9abd0208d74e1c61be [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:52.169092 13256 raft_consensus.cc:2258] T a388d59f6ba94e91958f55b340343fa6 P 79163c69f6f04a9abd0208d74e1c61be [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:52.169149 13248 ts_tablet_manager.cc:1888] T e5ed3e729bc64ebeac216ce5076b9fd9 P 0c4821d9898d42799c5789d4d2da3f63: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240503 01:59:52.169255 13248 log.cc:1196] T e5ed3e729bc64ebeac216ce5076b9fd9 P 0c4821d9898d42799c5789d4d2da3f63: Deleting WAL directory at /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/e5ed3e729bc64ebeac216ce5076b9fd9
I20240503 01:59:52.169939 6210 catalog_manager.cc:4830] TS 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130:34189): tablet e5ed3e729bc64ebeac216ce5076b9fd9 (table rebalance_test_table_0 [id=2e2540d8e10e43848a2e486f767d1576]) successfully deleted
I20240503 01:59:52.170428 13256 ts_tablet_manager.cc:1875] T a388d59f6ba94e91958f55b340343fa6 P 79163c69f6f04a9abd0208d74e1c61be: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:52.172232 13256 ts_tablet_manager.cc:1888] T a388d59f6ba94e91958f55b340343fa6 P 79163c69f6f04a9abd0208d74e1c61be: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240503 01:59:52.172295 13256 log.cc:1196] T a388d59f6ba94e91958f55b340343fa6 P 79163c69f6f04a9abd0208d74e1c61be: Deleting WAL directory at /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/a388d59f6ba94e91958f55b340343fa6
I20240503 01:59:52.172816 6208 catalog_manager.cc:4830] TS 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131:44005): tablet a388d59f6ba94e91958f55b340343fa6 (table rebalance_test_table_1 [id=91bdb83bc45b4b61b094a0180eacf8e3]) successfully deleted
I20240503 01:59:52.177959 6209 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a388d59f6ba94e91958f55b340343fa6 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:52.178121 6209 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e5ed3e729bc64ebeac216ce5076b9fd9 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:52.247596 6476 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 01:59:52.248143 9390 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:52.251610 6607 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 01:59:52.256685 8073 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:52.261546 8466 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:52.263940 8204 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:52.285522 6344 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 01:59:52.287422 8335 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:52.295979 6738 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 01:59:52.328950 6365 consensus_queue.cc:237] T 470b553976064bdd8c321799ac718274 P 4127249ba42644388d225d614c935a25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: true } }
I20240503 01:59:52.330518 8223 raft_consensus.cc:1261] T 470b553976064bdd8c321799ac718274 P 65d4632bcdd54ba596acdbbe1b5ef8af [term 1 FOLLOWER]: Refusing update from remote peer 4127249ba42644388d225d614c935a25: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:52.330663 6494 raft_consensus.cc:1261] T 470b553976064bdd8c321799ac718274 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Refusing update from remote peer 4127249ba42644388d225d614c935a25: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:52.330971 13319 consensus_queue.cc:1035] T 470b553976064bdd8c321799ac718274 P 4127249ba42644388d225d614c935a25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:52.331200 13318 consensus_queue.cc:1035] T 470b553976064bdd8c321799ac718274 P 4127249ba42644388d225d614c935a25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:52.333351 13224 raft_consensus.cc:2932] T 470b553976064bdd8c321799ac718274 P 4127249ba42644388d225d614c935a25 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER de2f076398ee44008622fb5db2a75952 (127.0.99.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: true } } }
I20240503 01:59:52.333824 6494 raft_consensus.cc:2932] T 470b553976064bdd8c321799ac718274 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER de2f076398ee44008622fb5db2a75952 (127.0.99.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: true } } }
I20240503 01:59:52.336156 8223 raft_consensus.cc:2932] T 470b553976064bdd8c321799ac718274 P 65d4632bcdd54ba596acdbbe1b5ef8af [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER de2f076398ee44008622fb5db2a75952 (127.0.99.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: true } } }
W20240503 01:59:52.336479 6301 consensus_peers.cc:485] T 470b553976064bdd8c321799ac718274 P 4127249ba42644388d225d614c935a25 -> Peer de2f076398ee44008622fb5db2a75952 (127.0.99.135:34035): Couldn't send request to peer de2f076398ee44008622fb5db2a75952. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 470b553976064bdd8c321799ac718274. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:52.336896 6217 catalog_manager.cc:5484] T 470b553976064bdd8c321799ac718274 P 0c4821d9898d42799c5789d4d2da3f63 reported cstate change: config changed from index 6 to 7, NON_VOTER de2f076398ee44008622fb5db2a75952 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "4127249ba42644388d225d614c935a25" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: true } } }
I20240503 01:59:52.345602 6365 consensus_queue.cc:237] T 0a369bc048e74fc8ba666d3318d66ae8 P 4127249ba42644388d225d614c935a25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: true } }
I20240503 01:59:52.347038 6625 raft_consensus.cc:1261] T 0a369bc048e74fc8ba666d3318d66ae8 P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Refusing update from remote peer 4127249ba42644388d225d614c935a25: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:52.347348 6757 raft_consensus.cc:1261] T 0a369bc048e74fc8ba666d3318d66ae8 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Refusing update from remote peer 4127249ba42644388d225d614c935a25: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 01:59:52.347357 6301 consensus_peers.cc:485] T 0a369bc048e74fc8ba666d3318d66ae8 P 4127249ba42644388d225d614c935a25 -> Peer de2f076398ee44008622fb5db2a75952 (127.0.99.135:34035): Couldn't send request to peer de2f076398ee44008622fb5db2a75952. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0a369bc048e74fc8ba666d3318d66ae8. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:52.347397 13319 consensus_queue.cc:1035] T 0a369bc048e74fc8ba666d3318d66ae8 P 4127249ba42644388d225d614c935a25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:52.347756 13323 consensus_queue.cc:1035] T 0a369bc048e74fc8ba666d3318d66ae8 P 4127249ba42644388d225d614c935a25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:52.349077 13319 raft_consensus.cc:2932] T 0a369bc048e74fc8ba666d3318d66ae8 P 4127249ba42644388d225d614c935a25 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER de2f076398ee44008622fb5db2a75952 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: true } } }
I20240503 01:59:52.349526 6625 raft_consensus.cc:2932] T 0a369bc048e74fc8ba666d3318d66ae8 P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER de2f076398ee44008622fb5db2a75952 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: true } } }
I20240503 01:59:52.351567 6757 raft_consensus.cc:2932] T 0a369bc048e74fc8ba666d3318d66ae8 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER de2f076398ee44008622fb5db2a75952 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: true } } }
I20240503 01:59:52.352890 6214 catalog_manager.cc:5484] T 0a369bc048e74fc8ba666d3318d66ae8 P 79163c69f6f04a9abd0208d74e1c61be reported cstate change: config changed from index -1 to 4, NON_VOTER de2f076398ee44008622fb5db2a75952 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "4127249ba42644388d225d614c935a25" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: true } } }
I20240503 01:59:52.796010 13363 ts_tablet_manager.cc:902] T 470b553976064bdd8c321799ac718274 P de2f076398ee44008622fb5db2a75952: Initiating tablet copy from peer 4127249ba42644388d225d614c935a25 (127.0.99.129:35613)
I20240503 01:59:52.796691 13363 tablet_copy_client.cc:308] T 470b553976064bdd8c321799ac718274 P de2f076398ee44008622fb5db2a75952: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:35613
I20240503 01:59:52.807688 6385 tablet_copy_service.cc:140] P 4127249ba42644388d225d614c935a25: Received BeginTabletCopySession request for tablet 470b553976064bdd8c321799ac718274 from peer de2f076398ee44008622fb5db2a75952 ({username='slave'} at 127.0.99.135:51835)
I20240503 01:59:52.807796 6385 tablet_copy_service.cc:161] P 4127249ba42644388d225d614c935a25: Beginning new tablet copy session on tablet 470b553976064bdd8c321799ac718274 from peer de2f076398ee44008622fb5db2a75952 at {username='slave'} at 127.0.99.135:51835: session id = de2f076398ee44008622fb5db2a75952-470b553976064bdd8c321799ac718274
I20240503 01:59:52.808521 6385 tablet_copy_source_session.cc:205] T 470b553976064bdd8c321799ac718274 P 4127249ba42644388d225d614c935a25: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:52.809422 13363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 470b553976064bdd8c321799ac718274. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:52.811297 13363 tablet_copy_client.cc:783] T 470b553976064bdd8c321799ac718274 P de2f076398ee44008622fb5db2a75952: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:52.811445 13363 tablet_copy_client.cc:647] T 470b553976064bdd8c321799ac718274 P de2f076398ee44008622fb5db2a75952: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:52.812491 13364 tablet_copy_client.cc:1002] T 470b553976064bdd8c321799ac718274 P de2f076398ee44008622fb5db2a75952: tablet copy: Injecting latency into file download: 1500
I20240503 01:59:52.953238 13369 ts_tablet_manager.cc:902] T 0a369bc048e74fc8ba666d3318d66ae8 P de2f076398ee44008622fb5db2a75952: Initiating tablet copy from peer 4127249ba42644388d225d614c935a25 (127.0.99.129:35613)
I20240503 01:59:52.953842 13369 tablet_copy_client.cc:308] T 0a369bc048e74fc8ba666d3318d66ae8 P de2f076398ee44008622fb5db2a75952: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:35613
I20240503 01:59:52.954473 6385 tablet_copy_service.cc:140] P 4127249ba42644388d225d614c935a25: Received BeginTabletCopySession request for tablet 0a369bc048e74fc8ba666d3318d66ae8 from peer de2f076398ee44008622fb5db2a75952 ({username='slave'} at 127.0.99.135:51835)
I20240503 01:59:52.954583 6385 tablet_copy_service.cc:161] P 4127249ba42644388d225d614c935a25: Beginning new tablet copy session on tablet 0a369bc048e74fc8ba666d3318d66ae8 from peer de2f076398ee44008622fb5db2a75952 at {username='slave'} at 127.0.99.135:51835: session id = de2f076398ee44008622fb5db2a75952-0a369bc048e74fc8ba666d3318d66ae8
I20240503 01:59:52.955565 6385 tablet_copy_source_session.cc:205] T 0a369bc048e74fc8ba666d3318d66ae8 P 4127249ba42644388d225d614c935a25: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:52.956688 13369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a369bc048e74fc8ba666d3318d66ae8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:52.958943 13369 tablet_copy_client.cc:783] T 0a369bc048e74fc8ba666d3318d66ae8 P de2f076398ee44008622fb5db2a75952: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:52.959089 13369 tablet_copy_client.cc:647] T 0a369bc048e74fc8ba666d3318d66ae8 P de2f076398ee44008622fb5db2a75952: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:52.960067 13373 tablet_copy_client.cc:1002] T 0a369bc048e74fc8ba666d3318d66ae8 P de2f076398ee44008622fb5db2a75952: tablet copy: Injecting latency into file download: 1500
I20240503 01:59:54.312871 13363 tablet_copy_client.cc:523] T 470b553976064bdd8c321799ac718274 P de2f076398ee44008622fb5db2a75952: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:54.314311 13363 tablet_bootstrap.cc:492] T 470b553976064bdd8c321799ac718274 P de2f076398ee44008622fb5db2a75952: Bootstrap starting.
I20240503 01:59:54.332477 13363 tablet_bootstrap.cc:492] T 470b553976064bdd8c321799ac718274 P de2f076398ee44008622fb5db2a75952: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:54.332772 13363 tablet_bootstrap.cc:492] T 470b553976064bdd8c321799ac718274 P de2f076398ee44008622fb5db2a75952: Bootstrap complete.
I20240503 01:59:54.332945 13363 ts_tablet_manager.cc:1367] T 470b553976064bdd8c321799ac718274 P de2f076398ee44008622fb5db2a75952: Time spent bootstrapping tablet: real 0.019s user 0.016s sys 0.004s
I20240503 01:59:54.333367 13363 raft_consensus.cc:348] T 470b553976064bdd8c321799ac718274 P de2f076398ee44008622fb5db2a75952 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: true } }
I20240503 01:59:54.333492 13363 raft_consensus.cc:729] T 470b553976064bdd8c321799ac718274 P de2f076398ee44008622fb5db2a75952 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: de2f076398ee44008622fb5db2a75952, State: Initialized, Role: LEARNER
I20240503 01:59:54.333645 13363 consensus_queue.cc:260] T 470b553976064bdd8c321799ac718274 P de2f076398ee44008622fb5db2a75952 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: true } }
I20240503 01:59:54.333985 13363 ts_tablet_manager.cc:1398] T 470b553976064bdd8c321799ac718274 P de2f076398ee44008622fb5db2a75952: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:54.334662 6385 tablet_copy_service.cc:342] P 4127249ba42644388d225d614c935a25: Request end of tablet copy session de2f076398ee44008622fb5db2a75952-470b553976064bdd8c321799ac718274 received from {username='slave'} at 127.0.99.135:51835
I20240503 01:59:54.334756 6385 tablet_copy_service.cc:433] P 4127249ba42644388d225d614c935a25: ending tablet copy session de2f076398ee44008622fb5db2a75952-470b553976064bdd8c321799ac718274 on tablet 470b553976064bdd8c321799ac718274 with peer de2f076398ee44008622fb5db2a75952
I20240503 01:59:54.340729 8354 raft_consensus.cc:1203] T 470b553976064bdd8c321799ac718274 P de2f076398ee44008622fb5db2a75952 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 01:59:54.460381 13369 tablet_copy_client.cc:523] T 0a369bc048e74fc8ba666d3318d66ae8 P de2f076398ee44008622fb5db2a75952: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:54.461722 13369 tablet_bootstrap.cc:492] T 0a369bc048e74fc8ba666d3318d66ae8 P de2f076398ee44008622fb5db2a75952: Bootstrap starting.
I20240503 01:59:54.478044 13369 tablet_bootstrap.cc:492] T 0a369bc048e74fc8ba666d3318d66ae8 P de2f076398ee44008622fb5db2a75952: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:54.478287 13369 tablet_bootstrap.cc:492] T 0a369bc048e74fc8ba666d3318d66ae8 P de2f076398ee44008622fb5db2a75952: Bootstrap complete.
I20240503 01:59:54.478447 13369 ts_tablet_manager.cc:1367] T 0a369bc048e74fc8ba666d3318d66ae8 P de2f076398ee44008622fb5db2a75952: Time spent bootstrapping tablet: real 0.017s user 0.016s sys 0.000s
I20240503 01:59:54.478865 13369 raft_consensus.cc:348] T 0a369bc048e74fc8ba666d3318d66ae8 P de2f076398ee44008622fb5db2a75952 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: true } }
I20240503 01:59:54.478978 13369 raft_consensus.cc:729] T 0a369bc048e74fc8ba666d3318d66ae8 P de2f076398ee44008622fb5db2a75952 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: de2f076398ee44008622fb5db2a75952, State: Initialized, Role: LEARNER
I20240503 01:59:54.479082 13369 consensus_queue.cc:260] T 0a369bc048e74fc8ba666d3318d66ae8 P de2f076398ee44008622fb5db2a75952 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: true } }
I20240503 01:59:54.479437 13369 ts_tablet_manager.cc:1398] T 0a369bc048e74fc8ba666d3318d66ae8 P de2f076398ee44008622fb5db2a75952: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240503 01:59:54.479959 6385 tablet_copy_service.cc:342] P 4127249ba42644388d225d614c935a25: Request end of tablet copy session de2f076398ee44008622fb5db2a75952-0a369bc048e74fc8ba666d3318d66ae8 received from {username='slave'} at 127.0.99.135:51835
I20240503 01:59:54.480051 6385 tablet_copy_service.cc:433] P 4127249ba42644388d225d614c935a25: ending tablet copy session de2f076398ee44008622fb5db2a75952-0a369bc048e74fc8ba666d3318d66ae8 on tablet 0a369bc048e74fc8ba666d3318d66ae8 with peer de2f076398ee44008622fb5db2a75952
I20240503 01:59:54.734766 13471 raft_consensus.cc:1050] T 470b553976064bdd8c321799ac718274 P 4127249ba42644388d225d614c935a25: attempting to promote NON_VOTER de2f076398ee44008622fb5db2a75952 to VOTER
I20240503 01:59:54.735260 13471 consensus_queue.cc:237] T 470b553976064bdd8c321799ac718274 P 4127249ba42644388d225d614c935a25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } }
I20240503 01:59:54.736889 6494 raft_consensus.cc:1261] T 470b553976064bdd8c321799ac718274 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Refusing update from remote peer 4127249ba42644388d225d614c935a25: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:54.736892 8223 raft_consensus.cc:1261] T 470b553976064bdd8c321799ac718274 P 65d4632bcdd54ba596acdbbe1b5ef8af [term 1 FOLLOWER]: Refusing update from remote peer 4127249ba42644388d225d614c935a25: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:54.737005 8354 raft_consensus.cc:1261] T 470b553976064bdd8c321799ac718274 P de2f076398ee44008622fb5db2a75952 [term 1 LEARNER]: Refusing update from remote peer 4127249ba42644388d225d614c935a25: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:54.737313 13474 consensus_queue.cc:1035] T 470b553976064bdd8c321799ac718274 P 4127249ba42644388d225d614c935a25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:54.737499 13471 consensus_queue.cc:1035] T 470b553976064bdd8c321799ac718274 P 4127249ba42644388d225d614c935a25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:54.737679 13472 consensus_queue.cc:1035] T 470b553976064bdd8c321799ac718274 P 4127249ba42644388d225d614c935a25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:54.740196 13474 raft_consensus.cc:2932] T 470b553976064bdd8c321799ac718274 P 4127249ba42644388d225d614c935a25 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, de2f076398ee44008622fb5db2a75952 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } }
I20240503 01:59:54.740833 8223 raft_consensus.cc:2932] T 470b553976064bdd8c321799ac718274 P 65d4632bcdd54ba596acdbbe1b5ef8af [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, de2f076398ee44008622fb5db2a75952 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } }
I20240503 01:59:54.741027 6494 raft_consensus.cc:2932] T 470b553976064bdd8c321799ac718274 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, de2f076398ee44008622fb5db2a75952 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } }
I20240503 01:59:54.742960 6214 catalog_manager.cc:5484] T 470b553976064bdd8c321799ac718274 P 4127249ba42644388d225d614c935a25 reported cstate change: config changed from index 7 to 8, de2f076398ee44008622fb5db2a75952 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4127249ba42644388d225d614c935a25" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:54.743218 8354 raft_consensus.cc:2932] T 470b553976064bdd8c321799ac718274 P de2f076398ee44008622fb5db2a75952 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, de2f076398ee44008622fb5db2a75952 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } }
I20240503 01:59:54.920449 6365 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "470b553976064bdd8c321799ac718274"
dest_uuid: "4127249ba42644388d225d614c935a25"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:50428
I20240503 01:59:54.920576 6365 raft_consensus.cc:595] T 470b553976064bdd8c321799ac718274 P 4127249ba42644388d225d614c935a25 [term 1 LEADER]: Received request to transfer leadership
I20240503 01:59:54.975803 8354 raft_consensus.cc:1203] T 0a369bc048e74fc8ba666d3318d66ae8 P de2f076398ee44008622fb5db2a75952 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 01:59:55.125625 13472 raft_consensus.cc:979] T 470b553976064bdd8c321799ac718274 P 4127249ba42644388d225d614c935a25: : Instructing follower de2f076398ee44008622fb5db2a75952 to start an election
I20240503 01:59:55.125747 13472 raft_consensus.cc:1067] T 470b553976064bdd8c321799ac718274 P 4127249ba42644388d225d614c935a25 [term 1 LEADER]: Signalling peer de2f076398ee44008622fb5db2a75952 to start an election
I20240503 01:59:55.126250 8354 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "470b553976064bdd8c321799ac718274"
dest_uuid: "de2f076398ee44008622fb5db2a75952"
from {username='slave'} at 127.0.99.129:53761
I20240503 01:59:55.126453 8354 raft_consensus.cc:482] T 470b553976064bdd8c321799ac718274 P de2f076398ee44008622fb5db2a75952 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 01:59:55.126528 8354 raft_consensus.cc:3037] T 470b553976064bdd8c321799ac718274 P de2f076398ee44008622fb5db2a75952 [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:55.127532 8354 raft_consensus.cc:504] T 470b553976064bdd8c321799ac718274 P de2f076398ee44008622fb5db2a75952 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } }
I20240503 01:59:55.128870 8354 leader_election.cc:288] T 470b553976064bdd8c321799ac718274 P de2f076398ee44008622fb5db2a75952 [CANDIDATE]: Term 2 election: Requested vote from peers 4127249ba42644388d225d614c935a25 (127.0.99.129:35613), 0c4821d9898d42799c5789d4d2da3f63 (127.0.99.130:34189), 65d4632bcdd54ba596acdbbe1b5ef8af (127.0.99.134:35795)
I20240503 01:59:55.129436 6494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "470b553976064bdd8c321799ac718274" candidate_uuid: "de2f076398ee44008622fb5db2a75952" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "0c4821d9898d42799c5789d4d2da3f63"
I20240503 01:59:55.129573 6494 raft_consensus.cc:3037] T 470b553976064bdd8c321799ac718274 P 0c4821d9898d42799c5789d4d2da3f63 [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:55.129676 6365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "470b553976064bdd8c321799ac718274" candidate_uuid: "de2f076398ee44008622fb5db2a75952" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "4127249ba42644388d225d614c935a25"
I20240503 01:59:55.129779 6365 raft_consensus.cc:3032] T 470b553976064bdd8c321799ac718274 P 4127249ba42644388d225d614c935a25 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 01:59:55.129817 6365 raft_consensus.cc:729] T 470b553976064bdd8c321799ac718274 P 4127249ba42644388d225d614c935a25 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4127249ba42644388d225d614c935a25, State: Running, Role: LEADER
I20240503 01:59:55.129951 6365 consensus_queue.cc:260] T 470b553976064bdd8c321799ac718274 P 4127249ba42644388d225d614c935a25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } }
I20240503 01:59:55.130154 6365 raft_consensus.cc:3037] T 470b553976064bdd8c321799ac718274 P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:55.130609 6494 raft_consensus.cc:2454] T 470b553976064bdd8c321799ac718274 P 0c4821d9898d42799c5789d4d2da3f63 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de2f076398ee44008622fb5db2a75952 in term 2.
I20240503 01:59:55.131176 6365 raft_consensus.cc:2454] T 470b553976064bdd8c321799ac718274 P 4127249ba42644388d225d614c935a25 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de2f076398ee44008622fb5db2a75952 in term 2.
I20240503 01:59:55.131492 8289 leader_election.cc:302] T 470b553976064bdd8c321799ac718274 P de2f076398ee44008622fb5db2a75952 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0c4821d9898d42799c5789d4d2da3f63, 4127249ba42644388d225d614c935a25, de2f076398ee44008622fb5db2a75952; no voters:
I20240503 01:59:55.131747 13449 raft_consensus.cc:2781] T 470b553976064bdd8c321799ac718274 P de2f076398ee44008622fb5db2a75952 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 01:59:55.131923 13449 raft_consensus.cc:686] T 470b553976064bdd8c321799ac718274 P de2f076398ee44008622fb5db2a75952 [term 2 LEADER]: Becoming Leader. State: Replica: de2f076398ee44008622fb5db2a75952, State: Running, Role: LEADER
I20240503 01:59:55.132220 13449 consensus_queue.cc:237] T 470b553976064bdd8c321799ac718274 P de2f076398ee44008622fb5db2a75952 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } }
I20240503 01:59:55.134907 6218 catalog_manager.cc:5484] T 470b553976064bdd8c321799ac718274 P de2f076398ee44008622fb5db2a75952 reported cstate change: term changed from 1 to 2, leader changed from 4127249ba42644388d225d614c935a25 (127.0.99.129) to de2f076398ee44008622fb5db2a75952 (127.0.99.135). New cstate: current_term: 2 leader_uuid: "de2f076398ee44008622fb5db2a75952" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:55.135641 8223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "470b553976064bdd8c321799ac718274" candidate_uuid: "de2f076398ee44008622fb5db2a75952" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af"
I20240503 01:59:55.135845 8223 raft_consensus.cc:3037] T 470b553976064bdd8c321799ac718274 P 65d4632bcdd54ba596acdbbe1b5ef8af [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:55.136909 8223 raft_consensus.cc:2454] T 470b553976064bdd8c321799ac718274 P 65d4632bcdd54ba596acdbbe1b5ef8af [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de2f076398ee44008622fb5db2a75952 in term 2.
I20240503 01:59:55.530175 8223 raft_consensus.cc:1261] T 470b553976064bdd8c321799ac718274 P 65d4632bcdd54ba596acdbbe1b5ef8af [term 2 FOLLOWER]: Refusing update from remote peer de2f076398ee44008622fb5db2a75952: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 01:59:55.530628 13449 consensus_queue.cc:1035] T 470b553976064bdd8c321799ac718274 P de2f076398ee44008622fb5db2a75952 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 01:59:55.579382 13534 raft_consensus.cc:1050] T 0a369bc048e74fc8ba666d3318d66ae8 P 4127249ba42644388d225d614c935a25: attempting to promote NON_VOTER de2f076398ee44008622fb5db2a75952 to VOTER
I20240503 01:59:55.579834 13534 consensus_queue.cc:237] T 0a369bc048e74fc8ba666d3318d66ae8 P 4127249ba42644388d225d614c935a25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } }
I20240503 01:59:55.581054 8354 raft_consensus.cc:1261] T 0a369bc048e74fc8ba666d3318d66ae8 P de2f076398ee44008622fb5db2a75952 [term 1 LEARNER]: Refusing update from remote peer 4127249ba42644388d225d614c935a25: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:55.581243 6625 raft_consensus.cc:1261] T 0a369bc048e74fc8ba666d3318d66ae8 P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Refusing update from remote peer 4127249ba42644388d225d614c935a25: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:55.581426 6757 raft_consensus.cc:1261] T 0a369bc048e74fc8ba666d3318d66ae8 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Refusing update from remote peer 4127249ba42644388d225d614c935a25: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:55.581642 13536 consensus_queue.cc:1035] T 0a369bc048e74fc8ba666d3318d66ae8 P 4127249ba42644388d225d614c935a25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:55.581840 13471 consensus_queue.cc:1035] T 0a369bc048e74fc8ba666d3318d66ae8 P 4127249ba42644388d225d614c935a25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:55.581991 13534 consensus_queue.cc:1035] T 0a369bc048e74fc8ba666d3318d66ae8 P 4127249ba42644388d225d614c935a25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:55.582957 6365 raft_consensus.cc:1261] T 470b553976064bdd8c321799ac718274 P 4127249ba42644388d225d614c935a25 [term 2 FOLLOWER]: Refusing update from remote peer de2f076398ee44008622fb5db2a75952: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 01:59:55.583416 13449 consensus_queue.cc:1035] T 470b553976064bdd8c321799ac718274 P de2f076398ee44008622fb5db2a75952 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 01:59:55.584415 13534 raft_consensus.cc:2932] T 0a369bc048e74fc8ba666d3318d66ae8 P 4127249ba42644388d225d614c935a25 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, de2f076398ee44008622fb5db2a75952 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } }
I20240503 01:59:55.585122 8354 raft_consensus.cc:2932] T 0a369bc048e74fc8ba666d3318d66ae8 P de2f076398ee44008622fb5db2a75952 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, de2f076398ee44008622fb5db2a75952 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } }
I20240503 01:59:55.585953 6757 raft_consensus.cc:2932] T 0a369bc048e74fc8ba666d3318d66ae8 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, de2f076398ee44008622fb5db2a75952 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } }
I20240503 01:59:55.587541 6218 catalog_manager.cc:5484] T 0a369bc048e74fc8ba666d3318d66ae8 P de2f076398ee44008622fb5db2a75952 reported cstate change: config changed from index 4 to 5, de2f076398ee44008622fb5db2a75952 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4127249ba42644388d225d614c935a25" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } }
I20240503 01:59:55.589021 6625 raft_consensus.cc:2932] T 0a369bc048e74fc8ba666d3318d66ae8 P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, de2f076398ee44008622fb5db2a75952 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } }
I20240503 01:59:55.590678 6494 raft_consensus.cc:1261] T 470b553976064bdd8c321799ac718274 P 0c4821d9898d42799c5789d4d2da3f63 [term 2 FOLLOWER]: Refusing update from remote peer de2f076398ee44008622fb5db2a75952: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 01:59:55.592283 13449 consensus_queue.cc:1035] T 470b553976064bdd8c321799ac718274 P de2f076398ee44008622fb5db2a75952 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 01:59:55.770249 6365 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0a369bc048e74fc8ba666d3318d66ae8"
dest_uuid: "4127249ba42644388d225d614c935a25"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:50494
I20240503 01:59:55.770392 6365 raft_consensus.cc:595] T 0a369bc048e74fc8ba666d3318d66ae8 P 4127249ba42644388d225d614c935a25 [term 1 LEADER]: Received request to transfer leadership
I20240503 01:59:55.987255 6365 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0a369bc048e74fc8ba666d3318d66ae8"
dest_uuid: "4127249ba42644388d225d614c935a25"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:50520
I20240503 01:59:55.987406 6365 raft_consensus.cc:595] T 0a369bc048e74fc8ba666d3318d66ae8 P 4127249ba42644388d225d614c935a25 [term 1 LEADER]: Received request to transfer leadership
I20240503 01:59:55.999696 13472 raft_consensus.cc:979] T 0a369bc048e74fc8ba666d3318d66ae8 P 4127249ba42644388d225d614c935a25: : Instructing follower de2f076398ee44008622fb5db2a75952 to start an election
I20240503 01:59:55.999817 13472 raft_consensus.cc:1067] T 0a369bc048e74fc8ba666d3318d66ae8 P 4127249ba42644388d225d614c935a25 [term 1 LEADER]: Signalling peer de2f076398ee44008622fb5db2a75952 to start an election
I20240503 01:59:56.000265 8354 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0a369bc048e74fc8ba666d3318d66ae8"
dest_uuid: "de2f076398ee44008622fb5db2a75952"
from {username='slave'} at 127.0.99.129:53761
I20240503 01:59:56.000406 8354 raft_consensus.cc:482] T 0a369bc048e74fc8ba666d3318d66ae8 P de2f076398ee44008622fb5db2a75952 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 01:59:56.000474 8354 raft_consensus.cc:3037] T 0a369bc048e74fc8ba666d3318d66ae8 P de2f076398ee44008622fb5db2a75952 [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:56.001469 8354 raft_consensus.cc:504] T 0a369bc048e74fc8ba666d3318d66ae8 P de2f076398ee44008622fb5db2a75952 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } }
I20240503 01:59:56.001932 8354 leader_election.cc:288] T 0a369bc048e74fc8ba666d3318d66ae8 P de2f076398ee44008622fb5db2a75952 [CANDIDATE]: Term 2 election: Requested vote from peers 4127249ba42644388d225d614c935a25 (127.0.99.129:35613), 79163c69f6f04a9abd0208d74e1c61be (127.0.99.131:44005), b6b1d5677b8d47788b7968f64b3dac3c (127.0.99.132:46067)
I20240503 01:59:56.002182 6365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a369bc048e74fc8ba666d3318d66ae8" candidate_uuid: "de2f076398ee44008622fb5db2a75952" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "4127249ba42644388d225d614c935a25"
I20240503 01:59:56.002312 6365 raft_consensus.cc:3032] T 0a369bc048e74fc8ba666d3318d66ae8 P 4127249ba42644388d225d614c935a25 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 01:59:56.002321 6625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a369bc048e74fc8ba666d3318d66ae8" candidate_uuid: "de2f076398ee44008622fb5db2a75952" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "79163c69f6f04a9abd0208d74e1c61be"
I20240503 01:59:56.002382 6365 raft_consensus.cc:729] T 0a369bc048e74fc8ba666d3318d66ae8 P 4127249ba42644388d225d614c935a25 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4127249ba42644388d225d614c935a25, State: Running, Role: LEADER
I20240503 01:59:56.002450 6625 raft_consensus.cc:3037] T 0a369bc048e74fc8ba666d3318d66ae8 P 79163c69f6f04a9abd0208d74e1c61be [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:56.002542 6365 consensus_queue.cc:260] T 0a369bc048e74fc8ba666d3318d66ae8 P 4127249ba42644388d225d614c935a25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } }
I20240503 01:59:56.002599 6757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a369bc048e74fc8ba666d3318d66ae8" candidate_uuid: "de2f076398ee44008622fb5db2a75952" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "b6b1d5677b8d47788b7968f64b3dac3c"
I20240503 01:59:56.002712 6365 raft_consensus.cc:3037] T 0a369bc048e74fc8ba666d3318d66ae8 P 4127249ba42644388d225d614c935a25 [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:56.002710 6757 raft_consensus.cc:3037] T 0a369bc048e74fc8ba666d3318d66ae8 P b6b1d5677b8d47788b7968f64b3dac3c [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:56.003419 6365 raft_consensus.cc:2454] T 0a369bc048e74fc8ba666d3318d66ae8 P 4127249ba42644388d225d614c935a25 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de2f076398ee44008622fb5db2a75952 in term 2.
I20240503 01:59:56.003458 6625 raft_consensus.cc:2454] T 0a369bc048e74fc8ba666d3318d66ae8 P 79163c69f6f04a9abd0208d74e1c61be [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de2f076398ee44008622fb5db2a75952 in term 2.
I20240503 01:59:56.003719 6757 raft_consensus.cc:2454] T 0a369bc048e74fc8ba666d3318d66ae8 P b6b1d5677b8d47788b7968f64b3dac3c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de2f076398ee44008622fb5db2a75952 in term 2.
I20240503 01:59:56.004137 8290 leader_election.cc:302] T 0a369bc048e74fc8ba666d3318d66ae8 P de2f076398ee44008622fb5db2a75952 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 4127249ba42644388d225d614c935a25, b6b1d5677b8d47788b7968f64b3dac3c, de2f076398ee44008622fb5db2a75952; no voters:
I20240503 01:59:56.004333 13541 raft_consensus.cc:2781] T 0a369bc048e74fc8ba666d3318d66ae8 P de2f076398ee44008622fb5db2a75952 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 01:59:56.004405 13541 raft_consensus.cc:686] T 0a369bc048e74fc8ba666d3318d66ae8 P de2f076398ee44008622fb5db2a75952 [term 2 LEADER]: Becoming Leader. State: Replica: de2f076398ee44008622fb5db2a75952, State: Running, Role: LEADER
I20240503 01:59:56.004540 13541 consensus_queue.cc:237] T 0a369bc048e74fc8ba666d3318d66ae8 P de2f076398ee44008622fb5db2a75952 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } }
I20240503 01:59:56.006439 6217 catalog_manager.cc:5484] T 0a369bc048e74fc8ba666d3318d66ae8 P de2f076398ee44008622fb5db2a75952 reported cstate change: term changed from 1 to 2, leader changed from 4127249ba42644388d225d614c935a25 (127.0.99.129) to de2f076398ee44008622fb5db2a75952 (127.0.99.135). New cstate: current_term: 2 leader_uuid: "de2f076398ee44008622fb5db2a75952" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:56.061640 8354 consensus_queue.cc:237] T 470b553976064bdd8c321799ac718274 P de2f076398ee44008622fb5db2a75952 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } }
I20240503 01:59:56.062793 6494 raft_consensus.cc:1261] T 470b553976064bdd8c321799ac718274 P 0c4821d9898d42799c5789d4d2da3f63 [term 2 FOLLOWER]: Refusing update from remote peer de2f076398ee44008622fb5db2a75952: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 01:59:56.062956 8224 raft_consensus.cc:1261] T 470b553976064bdd8c321799ac718274 P 65d4632bcdd54ba596acdbbe1b5ef8af [term 2 FOLLOWER]: Refusing update from remote peer de2f076398ee44008622fb5db2a75952: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 01:59:56.063169 13541 consensus_queue.cc:1035] T 470b553976064bdd8c321799ac718274 P de2f076398ee44008622fb5db2a75952 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 01:59:56.063495 13511 consensus_queue.cc:1035] T 470b553976064bdd8c321799ac718274 P de2f076398ee44008622fb5db2a75952 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 01:59:56.065030 13541 raft_consensus.cc:2932] T 470b553976064bdd8c321799ac718274 P de2f076398ee44008622fb5db2a75952 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 4127249ba42644388d225d614c935a25 (127.0.99.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } }
I20240503 01:59:56.065505 6494 raft_consensus.cc:2932] T 470b553976064bdd8c321799ac718274 P 0c4821d9898d42799c5789d4d2da3f63 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 4127249ba42644388d225d614c935a25 (127.0.99.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } }
I20240503 01:59:56.065754 8224 raft_consensus.cc:2932] T 470b553976064bdd8c321799ac718274 P 65d4632bcdd54ba596acdbbe1b5ef8af [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 4127249ba42644388d225d614c935a25 (127.0.99.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } }
I20240503 01:59:56.068138 6217 catalog_manager.cc:5484] T 470b553976064bdd8c321799ac718274 P 0c4821d9898d42799c5789d4d2da3f63 reported cstate change: config changed from index 8 to 10, VOTER 4127249ba42644388d225d614c935a25 (127.0.99.129) evicted. New cstate: current_term: 2 leader_uuid: "de2f076398ee44008622fb5db2a75952" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0c4821d9898d42799c5789d4d2da3f63" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34189 } } peers { permanent_uuid: "65d4632bcdd54ba596acdbbe1b5ef8af" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35795 } attrs { promote: false } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } }
I20240503 01:59:56.070638 6344 tablet_service.cc:1514] Processing DeleteTablet for tablet 470b553976064bdd8c321799ac718274 with delete_type TABLET_DATA_TOMBSTONED (TS 4127249ba42644388d225d614c935a25 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:57398
I20240503 01:59:56.071066 13576 tablet_replica.cc:330] T 470b553976064bdd8c321799ac718274 P 4127249ba42644388d225d614c935a25: stopping tablet replica
I20240503 01:59:56.071223 13576 raft_consensus.cc:2229] T 470b553976064bdd8c321799ac718274 P 4127249ba42644388d225d614c935a25 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:56.071379 13576 raft_consensus.cc:2258] T 470b553976064bdd8c321799ac718274 P 4127249ba42644388d225d614c935a25 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:56.072232 13576 ts_tablet_manager.cc:1875] T 470b553976064bdd8c321799ac718274 P 4127249ba42644388d225d614c935a25: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:56.074326 13576 ts_tablet_manager.cc:1888] T 470b553976064bdd8c321799ac718274 P 4127249ba42644388d225d614c935a25: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240503 01:59:56.074398 13576 log.cc:1196] T 470b553976064bdd8c321799ac718274 P 4127249ba42644388d225d614c935a25: Deleting WAL directory at /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/470b553976064bdd8c321799ac718274
I20240503 01:59:56.074945 6208 catalog_manager.cc:4830] TS 4127249ba42644388d225d614c935a25 (127.0.99.129:35613): tablet 470b553976064bdd8c321799ac718274 (table rebalance_test_table_2 [id=4a248d80763944458620bf40775e2098]) successfully deleted
I20240503 01:59:56.089973 6210 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 470b553976064bdd8c321799ac718274 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:56.383949 6757 raft_consensus.cc:1261] T 0a369bc048e74fc8ba666d3318d66ae8 P b6b1d5677b8d47788b7968f64b3dac3c [term 2 FOLLOWER]: Refusing update from remote peer de2f076398ee44008622fb5db2a75952: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 01:59:56.384423 13541 consensus_queue.cc:1035] T 0a369bc048e74fc8ba666d3318d66ae8 P de2f076398ee44008622fb5db2a75952 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:56.570124 6365 raft_consensus.cc:1261] T 0a369bc048e74fc8ba666d3318d66ae8 P 4127249ba42644388d225d614c935a25 [term 2 FOLLOWER]: Refusing update from remote peer de2f076398ee44008622fb5db2a75952: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 01:59:56.570583 13541 consensus_queue.cc:1035] T 0a369bc048e74fc8ba666d3318d66ae8 P de2f076398ee44008622fb5db2a75952 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4127249ba42644388d225d614c935a25" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35613 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:56.573144 6625 raft_consensus.cc:1261] T 0a369bc048e74fc8ba666d3318d66ae8 P 79163c69f6f04a9abd0208d74e1c61be [term 2 FOLLOWER]: Refusing update from remote peer de2f076398ee44008622fb5db2a75952: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 01:59:56.573760 13541 consensus_queue.cc:1035] T 0a369bc048e74fc8ba666d3318d66ae8 P de2f076398ee44008622fb5db2a75952 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:57.074391 8354 consensus_queue.cc:237] T 0a369bc048e74fc8ba666d3318d66ae8 P de2f076398ee44008622fb5db2a75952 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } }
I20240503 01:59:57.075520 6625 raft_consensus.cc:1261] T 0a369bc048e74fc8ba666d3318d66ae8 P 79163c69f6f04a9abd0208d74e1c61be [term 2 FOLLOWER]: Refusing update from remote peer de2f076398ee44008622fb5db2a75952: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 01:59:57.075552 6757 raft_consensus.cc:1261] T 0a369bc048e74fc8ba666d3318d66ae8 P b6b1d5677b8d47788b7968f64b3dac3c [term 2 FOLLOWER]: Refusing update from remote peer de2f076398ee44008622fb5db2a75952: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 01:59:57.075911 13541 consensus_queue.cc:1035] T 0a369bc048e74fc8ba666d3318d66ae8 P de2f076398ee44008622fb5db2a75952 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:57.076069 13511 consensus_queue.cc:1035] T 0a369bc048e74fc8ba666d3318d66ae8 P de2f076398ee44008622fb5db2a75952 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:57.077685 13449 raft_consensus.cc:2932] T 0a369bc048e74fc8ba666d3318d66ae8 P de2f076398ee44008622fb5db2a75952 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 4127249ba42644388d225d614c935a25 (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } }
I20240503 01:59:57.078088 6625 raft_consensus.cc:2932] T 0a369bc048e74fc8ba666d3318d66ae8 P 79163c69f6f04a9abd0208d74e1c61be [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 4127249ba42644388d225d614c935a25 (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } }
I20240503 01:59:57.078116 6757 raft_consensus.cc:2932] T 0a369bc048e74fc8ba666d3318d66ae8 P b6b1d5677b8d47788b7968f64b3dac3c [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 4127249ba42644388d225d614c935a25 (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } } }
I20240503 01:59:57.080040 6210 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0a369bc048e74fc8ba666d3318d66ae8 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:57.080210 6221 catalog_manager.cc:5484] T 0a369bc048e74fc8ba666d3318d66ae8 P de2f076398ee44008622fb5db2a75952 reported cstate change: config changed from index 5 to 7, VOTER 4127249ba42644388d225d614c935a25 (127.0.99.129) evicted. New cstate: current_term: 2 leader_uuid: "de2f076398ee44008622fb5db2a75952" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79163c69f6f04a9abd0208d74e1c61be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44005 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6b1d5677b8d47788b7968f64b3dac3c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "de2f076398ee44008622fb5db2a75952" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34035 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:57.083043 6344 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a369bc048e74fc8ba666d3318d66ae8 with delete_type TABLET_DATA_TOMBSTONED (TS 4127249ba42644388d225d614c935a25 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:57398
I20240503 01:59:57.083459 13621 tablet_replica.cc:330] T 0a369bc048e74fc8ba666d3318d66ae8 P 4127249ba42644388d225d614c935a25: stopping tablet replica
I20240503 01:59:57.083618 13621 raft_consensus.cc:2229] T 0a369bc048e74fc8ba666d3318d66ae8 P 4127249ba42644388d225d614c935a25 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:57.083787 13621 raft_consensus.cc:2258] T 0a369bc048e74fc8ba666d3318d66ae8 P 4127249ba42644388d225d614c935a25 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:57.084510 13621 ts_tablet_manager.cc:1875] T 0a369bc048e74fc8ba666d3318d66ae8 P 4127249ba42644388d225d614c935a25: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:57.086511 13621 ts_tablet_manager.cc:1888] T 0a369bc048e74fc8ba666d3318d66ae8 P 4127249ba42644388d225d614c935a25: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 01:59:57.086575 13621 log.cc:1196] T 0a369bc048e74fc8ba666d3318d66ae8 P 4127249ba42644388d225d614c935a25: Deleting WAL directory at /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/0a369bc048e74fc8ba666d3318d66ae8
I20240503 01:59:57.087057 6208 catalog_manager.cc:4830] TS 4127249ba42644388d225d614c935a25 (127.0.99.129:35613): tablet 0a369bc048e74fc8ba666d3318d66ae8 (table rebalance_test_table_6 [id=80cb1486f22e4bee930434fdfc3463d2]) successfully deleted
I20240503 01:59:57.315557 9390 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:57.315976 6476 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 01:59:57.316331 6344 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 01:59:57.327467 8073 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:57.335609 8466 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:57.344033 8204 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:57.345300 6607 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 01:59:57.346951 8335 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:57.375797 6738 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 01:59:57.470121 6344 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 01:59:57.471983 9390 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:57.473667 6476 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 01:59:57.476833 8204 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:57.487076 8466 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:57.487527 6738 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 01:59:57.493034 8073 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:57.503917 8335 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:57.518643 6607 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 01:59:57.765136 6344 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 01:59:57.767275 9390 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:57.770752 8204 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:57.771031 6476 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 01:59:57.785830 8335 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:57.787865 6738 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 01:59:57.793292 8073 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:57.800786 6607 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 01:59:57.827745 8466 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:57.928882 9390 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:57.930584 6476 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 01:59:57.931998 8204 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:57.939484 6344 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 01:59:57.941967 8335 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:57.947844 6738 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 01:59:57.957340 8466 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:57.960736 8073 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:57.984139 6607 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 01:59:58.104516 8204 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:58.108673 6476 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 01:59:58.109488 9390 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:58.114151 8335 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:58.123200 6738 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 01:59:58.125196 8073 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:58.126047 6607 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 01:59:58.148967 8466 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:58.158692 6344 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 01:59:58.265656 6344 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 01:59:58.266685 9390 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:58.274895 6476 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 01:59:58.276099 8204 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:58.277685 8466 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:58.287843 6738 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 01:59:58.288959 8335 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:58.294951 8073 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:58.307006 6607 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
6a09095f39a24e218f23f2963783bca2 | 127.0.99.190:38597 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.99.148:34293 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
0c4821d9898d42799c5789d4d2da3f63 | 127.0.99.130:34189 | HEALTHY | <none> | 6 | 0
2d0fd64aaa6548e09e1ebde581c54658 | 127.0.99.137:40613 | HEALTHY | <none> | 2 | 0
4127249ba42644388d225d614c935a25 | 127.0.99.129:35613 | HEALTHY | <none> | 6 | 0
65d4632bcdd54ba596acdbbe1b5ef8af | 127.0.99.134:35795 | HEALTHY | <none> | 2 | 0
79163c69f6f04a9abd0208d74e1c61be | 127.0.99.131:44005 | HEALTHY | <none> | 7 | 0
7a1404fe1c1b4b388b821c8261b53104 | 127.0.99.136:46759 | HEALTHY | <none> | 0 | 0
9264f24d028347d98f7e4e2d1aedeb79 | 127.0.99.133:41713 | HEALTHY | <none> | 1 | 0
b6b1d5677b8d47788b7968f64b3dac3c | 127.0.99.132:46067 | HEALTHY | <none> | 4 | 0
de2f076398ee44008622fb5db2a75952 | 127.0.99.135:34035 | HEALTHY | <none> | 2 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 9
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
---------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 9 server(s) checked
local_ip_for_outbound_sockets | 127.0.99.129 | experimental | 127.0.99.129:35613
local_ip_for_outbound_sockets | 127.0.99.130 | experimental | 127.0.99.130:34189
local_ip_for_outbound_sockets | 127.0.99.131 | experimental | 127.0.99.131:44005
local_ip_for_outbound_sockets | 127.0.99.132 | experimental | 127.0.99.132:46067
local_ip_for_outbound_sockets | 127.0.99.133 | experimental | 127.0.99.133:41713
local_ip_for_outbound_sockets | 127.0.99.134 | experimental | 127.0.99.134:35795
local_ip_for_outbound_sockets | 127.0.99.135 | experimental | 127.0.99.135:34035
local_ip_for_outbound_sockets | 127.0.99.136 | experimental | 127.0.99.136:46759
local_ip_for_outbound_sockets | 127.0.99.137 | experimental | 127.0.99.137:40613
never_fsync | true | unsafe,advanced | all 9 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 9 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 9 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 9 server(s) checked
server_dump_info_format | pb | hidden | all 9 server(s) checked
server_dump_info_path | /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.99.129:35613
server_dump_info_path | /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.0.99.130:34189
server_dump_info_path | /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.99.131:44005
server_dump_info_path | /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.0.99.132:46067
server_dump_info_path | /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.0.99.133:41713
server_dump_info_path | /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.0.99.134:35795
server_dump_info_path | /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.0.99.135:34035
server_dump_info_path | /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb | hidden | 127.0.99.136:46759
server_dump_info_path | /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-8/data/info.pb | hidden | 127.0.99.137:40613
tablet_copy_download_file_inject_latency_ms | 1500 | hidden | all 9 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.99.148:34293 | all 9 server(s) checked
time_source | builtin | all 9 server(s) checked
Version Summary
Version | Servers
-----------------+--------------------------
1.18.0-SNAPSHOT | all 10 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
rebalance_test_table_0 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_5 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_6 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_7 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_8 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_9 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 10
First Quartile | 10
Median | 10
Third Quartile | 10
Maximum | 10
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 9
Tables | 10
Tablets | 30
Replicas | 90
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240503 01:59:58.339624 398 log_verifier.cc:126] Checking tablet 0a369bc048e74fc8ba666d3318d66ae8
I20240503 01:59:58.366030 398 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 0a369bc048e74fc8ba666d3318d66ae8
I20240503 01:59:58.366094 398 log_verifier.cc:126] Checking tablet 0bdcea4e59ec4e2da63b9316f9d718be
I20240503 01:59:58.393713 398 log_verifier.cc:177] Verified matching terms for 9 ops in tablet 0bdcea4e59ec4e2da63b9316f9d718be
I20240503 01:59:58.393810 398 log_verifier.cc:126] Checking tablet 120d5dd4f26c42399702141094ce7982
I20240503 01:59:58.422422 398 log_verifier.cc:177] Verified matching terms for 9 ops in tablet 120d5dd4f26c42399702141094ce7982
I20240503 01:59:58.422499 398 log_verifier.cc:126] Checking tablet 1d392e08c15a4d4098a350287b3163ef
I20240503 01:59:58.450525 398 log_verifier.cc:177] Verified matching terms for 10 ops in tablet 1d392e08c15a4d4098a350287b3163ef
I20240503 01:59:58.450608 398 log_verifier.cc:126] Checking tablet 2088c84e4eef4ccfbc70d7b668b8f2cf
I20240503 01:59:58.481671 398 log_verifier.cc:177] Verified matching terms for 9 ops in tablet 2088c84e4eef4ccfbc70d7b668b8f2cf
I20240503 01:59:58.481747 398 log_verifier.cc:126] Checking tablet 21d27cca81e8455e9934e1cc604fc2b2
I20240503 01:59:58.508116 398 log_verifier.cc:177] Verified matching terms for 10 ops in tablet 21d27cca81e8455e9934e1cc604fc2b2
I20240503 01:59:58.508188 398 log_verifier.cc:126] Checking tablet 23b3b3d386544e5d9bfae8041cdf01a0
I20240503 01:59:58.534718 398 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 23b3b3d386544e5d9bfae8041cdf01a0
I20240503 01:59:58.534798 398 log_verifier.cc:126] Checking tablet 2ac8004211e64c3481e75377092fa93b
I20240503 01:59:58.563426 398 log_verifier.cc:177] Verified matching terms for 9 ops in tablet 2ac8004211e64c3481e75377092fa93b
I20240503 01:59:58.563494 398 log_verifier.cc:126] Checking tablet 2c8c328220e64e168c74fea7619f0c3b
I20240503 01:59:58.590708 398 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 2c8c328220e64e168c74fea7619f0c3b
I20240503 01:59:58.590776 398 log_verifier.cc:126] Checking tablet 470b553976064bdd8c321799ac718274
I20240503 01:59:58.618275 398 log_verifier.cc:177] Verified matching terms for 10 ops in tablet 470b553976064bdd8c321799ac718274
I20240503 01:59:58.618341 398 log_verifier.cc:126] Checking tablet 4e7ceb1453804246a41bd1dda0e39d02
I20240503 01:59:58.646278 398 log_verifier.cc:177] Verified matching terms for 10 ops in tablet 4e7ceb1453804246a41bd1dda0e39d02
I20240503 01:59:58.646366 398 log_verifier.cc:126] Checking tablet 5b599bd1fea6409fbe485c7a35c0891c
I20240503 01:59:58.674217 398 log_verifier.cc:177] Verified matching terms for 9 ops in tablet 5b599bd1fea6409fbe485c7a35c0891c
I20240503 01:59:58.674295 398 log_verifier.cc:126] Checking tablet 67150a2f5cb147b8b57a8c8040b567e1
I20240503 01:59:58.701004 398 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 67150a2f5cb147b8b57a8c8040b567e1
I20240503 01:59:58.701067 398 log_verifier.cc:126] Checking tablet 679fc86ea2ad4bd6a9160fddf4c586d7
I20240503 01:59:58.740420 398 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 679fc86ea2ad4bd6a9160fddf4c586d7
I20240503 01:59:58.740496 398 log_verifier.cc:126] Checking tablet 82cfe99cd6fe4a82a7e35298ec634b8b
I20240503 01:59:58.768771 398 log_verifier.cc:177] Verified matching terms for 9 ops in tablet 82cfe99cd6fe4a82a7e35298ec634b8b
I20240503 01:59:58.768837 398 log_verifier.cc:126] Checking tablet 885b787d0cbb4ddabd47cb7c42a78c74
I20240503 01:59:58.795173 398 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 885b787d0cbb4ddabd47cb7c42a78c74
I20240503 01:59:58.795238 398 log_verifier.cc:126] Checking tablet 897e88ac93484a11829189e027de8bae
I20240503 01:59:58.825371 398 log_verifier.cc:177] Verified matching terms for 9 ops in tablet 897e88ac93484a11829189e027de8bae
I20240503 01:59:58.825439 398 log_verifier.cc:126] Checking tablet a388d59f6ba94e91958f55b340343fa6
I20240503 01:59:58.853237 398 log_verifier.cc:177] Verified matching terms for 10 ops in tablet a388d59f6ba94e91958f55b340343fa6
I20240503 01:59:58.853312 398 log_verifier.cc:126] Checking tablet a4983426b2d843e7a9ed335c79178678
I20240503 01:59:58.879484 398 log_verifier.cc:177] Verified matching terms for 6 ops in tablet a4983426b2d843e7a9ed335c79178678
I20240503 01:59:58.879555 398 log_verifier.cc:126] Checking tablet a5926dae08184952bc693342c2c61ebf
I20240503 01:59:58.909291 398 log_verifier.cc:177] Verified matching terms for 9 ops in tablet a5926dae08184952bc693342c2c61ebf
I20240503 01:59:58.909384 398 log_verifier.cc:126] Checking tablet a655d914858447b59fe5763e43c0807d
I20240503 01:59:58.936698 398 log_verifier.cc:177] Verified matching terms for 6 ops in tablet a655d914858447b59fe5763e43c0807d
I20240503 01:59:58.936769 398 log_verifier.cc:126] Checking tablet a9a74e04ce3745b2b479e96a3f3dc86e
I20240503 01:59:58.963239 398 log_verifier.cc:177] Verified matching terms for 9 ops in tablet a9a74e04ce3745b2b479e96a3f3dc86e
I20240503 01:59:58.963311 398 log_verifier.cc:126] Checking tablet c3337c95b354418faf37684e7440b1c7
I20240503 01:59:58.989980 398 log_verifier.cc:177] Verified matching terms for 9 ops in tablet c3337c95b354418faf37684e7440b1c7
I20240503 01:59:58.990065 398 log_verifier.cc:126] Checking tablet ccb34a4b52244d90850ab04e21299f55
I20240503 01:59:59.019534 398 log_verifier.cc:177] Verified matching terms for 9 ops in tablet ccb34a4b52244d90850ab04e21299f55
I20240503 01:59:59.019603 398 log_verifier.cc:126] Checking tablet d6ee8175874c4063986033b0a80ccb82
I20240503 01:59:59.047284 398 log_verifier.cc:177] Verified matching terms for 9 ops in tablet d6ee8175874c4063986033b0a80ccb82
I20240503 01:59:59.047360 398 log_verifier.cc:126] Checking tablet dc7d272c4d324e10959c62362e0d81f2
I20240503 01:59:59.074142 398 log_verifier.cc:177] Verified matching terms for 6 ops in tablet dc7d272c4d324e10959c62362e0d81f2
I20240503 01:59:59.074210 398 log_verifier.cc:126] Checking tablet e5a00d4ce3a64ca5a7c5bd65c556f429
I20240503 01:59:59.104755 398 log_verifier.cc:177] Verified matching terms for 6 ops in tablet e5a00d4ce3a64ca5a7c5bd65c556f429
I20240503 01:59:59.104831 398 log_verifier.cc:126] Checking tablet e5ed3e729bc64ebeac216ce5076b9fd9
I20240503 01:59:59.147998 398 log_verifier.cc:177] Verified matching terms for 10 ops in tablet e5ed3e729bc64ebeac216ce5076b9fd9
I20240503 01:59:59.148064 398 log_verifier.cc:126] Checking tablet f11a8034585a4d57be0e9b1e5e8bcd85
I20240503 01:59:59.197360 398 log_verifier.cc:177] Verified matching terms for 9 ops in tablet f11a8034585a4d57be0e9b1e5e8bcd85
I20240503 01:59:59.197428 398 log_verifier.cc:126] Checking tablet ff8f06c5ad8442ee942bf900212fe858
I20240503 01:59:59.244957 398 log_verifier.cc:177] Verified matching terms for 9 ops in tablet ff8f06c5ad8442ee942bf900212fe858
I20240503 01:59:59.245421 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task42rE7p/build/debug/bin/kudu with pid 6283
I20240503 01:59:59.252774 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task42rE7p/build/debug/bin/kudu with pid 6414
I20240503 01:59:59.260206 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task42rE7p/build/debug/bin/kudu with pid 6545
I20240503 01:59:59.267094 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task42rE7p/build/debug/bin/kudu with pid 6676
I20240503 01:59:59.273696 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task42rE7p/build/debug/bin/kudu with pid 8011
I20240503 01:59:59.279673 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task42rE7p/build/debug/bin/kudu with pid 8142
I20240503 01:59:59.285789 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task42rE7p/build/debug/bin/kudu with pid 8273
I20240503 01:59:59.291998 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task42rE7p/build/debug/bin/kudu with pid 8403
I20240503 01:59:59.297966 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task42rE7p/build/debug/bin/kudu with pid 9207
I20240503 01:59:59.305476 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task42rE7p/build/debug/bin/kudu with pid 6192
2024-05-03T01:59:59Z chronyd exiting
[ OK ] TserverAddedDuringRebalancingTest.TserverStarts/1 (34043 ms)
[----------] 1 test from TserverAddedDuringRebalancingTest (34043 ms total)
[----------] 1 test from RebalancerAndSingleReplicaTablets
[ RUN ] RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove/5
I20240503 01:59:59.416925 398 test_util.cc:268] Using random seed: 330998360
I20240503 01:59:59.417682 398 ts_itest-base.cc:112] Starting cluster with:
I20240503 01:59:59.417722 398 ts_itest-base.cc:113] --------------
I20240503 01:59:59.417737 398 ts_itest-base.cc:114] 4 tablet servers
I20240503 01:59:59.417752 398 ts_itest-base.cc:115] 1 replicas per TS
I20240503 01:59:59.417768 398 ts_itest-base.cc:116] --------------
2024-05-03T01:59:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T01:59:59Z Disabled control of system clock
I20240503 01:59:59.426185 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task42rE7p/build/debug/bin/kudu
/tmp/dist-test-task42rE7p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:44351
--webserver_interface=127.0.99.190
--webserver_port=0
--builtin_ntp_servers=127.0.99.148:40197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.99.190:44351
--raft_prepare_replacement_before_eviction=true
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false with env {}
W20240503 01:59:59.538245 13896 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:59.538466 13896 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:59:59.538537 13896 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:59.541594 13896 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:59:59.541656 13896 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 01:59:59.541680 13896 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:59:59.541702 13896 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 01:59:59.541739 13896 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 01:59:59.545612 13896 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:40197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.0.99.190:44351
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:44351
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.99.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.13896
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:59:59.546643 13896 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:59:59.547874 13896 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:59.556385 13901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:59.557286 13902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:59.558178 13904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:59.559687 13896 server_base.cc:1034] running on GCE node
I20240503 01:59:59.560165 13896 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:59:59.561077 13896 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:59:59.562261 13896 hybrid_clock.cc:648] HybridClock initialized: now 1714701599562231 us; error 44 us; skew 500 ppm
I20240503 01:59:59.564224 13896 webserver.cc:460] Webserver started at http://127.0.99.190:35603/ using document root <none> and password file <none>
I20240503 01:59:59.564839 13896 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:59.564906 13896 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:59.565081 13896 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:59:59.566697 13896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "66a949c20cbb4391876ef040890466e1"
format_stamp: "Formatted at 2024-05-03 01:59:59 on dist-test-slave-dtz3"
I20240503 01:59:59.567160 13896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "66a949c20cbb4391876ef040890466e1"
format_stamp: "Formatted at 2024-05-03 01:59:59 on dist-test-slave-dtz3"
I20240503 01:59:59.570468 13896 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 01:59:59.572590 13910 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:59.573700 13896 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:59.573827 13896 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/wal
uuid: "66a949c20cbb4391876ef040890466e1"
format_stamp: "Formatted at 2024-05-03 01:59:59 on dist-test-slave-dtz3"
I20240503 01:59:59.573920 13896 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:59.587114 13896 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:59.587692 13896 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:59:59.587842 13896 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:59.594424 13896 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.190:44351
I20240503 01:59:59.594436 13962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.190:44351 every 8 connection(s)
I20240503 01:59:59.595389 13896 server_base.cc:1166] Dumped server information to /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 01:59:59.597826 13963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:59.601068 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task42rE7p/build/debug/bin/kudu as pid 13896
I20240503 01:59:59.601212 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 01:59:59.602988 13963 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66a949c20cbb4391876ef040890466e1: Bootstrap starting.
I20240503 01:59:59.604967 13963 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 66a949c20cbb4391876ef040890466e1: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:59.605635 13963 log.cc:824] T 00000000000000000000000000000000 P 66a949c20cbb4391876ef040890466e1: Log is configured to *not* fsync() on all Append() calls
I20240503 01:59:59.607407 13963 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66a949c20cbb4391876ef040890466e1: No bootstrap required, opened a new log
I20240503 01:59:59.610069 13963 raft_consensus.cc:348] T 00000000000000000000000000000000 P 66a949c20cbb4391876ef040890466e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66a949c20cbb4391876ef040890466e1" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 44351 } }
I20240503 01:59:59.610245 13963 raft_consensus.cc:374] T 00000000000000000000000000000000 P 66a949c20cbb4391876ef040890466e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:59.610303 13963 raft_consensus.cc:729] T 00000000000000000000000000000000 P 66a949c20cbb4391876ef040890466e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66a949c20cbb4391876ef040890466e1, State: Initialized, Role: FOLLOWER
I20240503 01:59:59.610656 13963 consensus_queue.cc:260] T 00000000000000000000000000000000 P 66a949c20cbb4391876ef040890466e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66a949c20cbb4391876ef040890466e1" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 44351 } }
I20240503 01:59:59.610774 13963 raft_consensus.cc:388] T 00000000000000000000000000000000 P 66a949c20cbb4391876ef040890466e1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 01:59:59.610852 13963 raft_consensus.cc:482] T 00000000000000000000000000000000 P 66a949c20cbb4391876ef040890466e1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 01:59:59.610983 13963 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 66a949c20cbb4391876ef040890466e1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:59.611685 13963 raft_consensus.cc:504] T 00000000000000000000000000000000 P 66a949c20cbb4391876ef040890466e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66a949c20cbb4391876ef040890466e1" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 44351 } }
I20240503 01:59:59.611982 13963 leader_election.cc:302] T 00000000000000000000000000000000 P 66a949c20cbb4391876ef040890466e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 66a949c20cbb4391876ef040890466e1; no voters:
I20240503 01:59:59.612224 13963 leader_election.cc:288] T 00000000000000000000000000000000 P 66a949c20cbb4391876ef040890466e1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 01:59:59.612735 13968 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 66a949c20cbb4391876ef040890466e1 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:59.612965 13968 raft_consensus.cc:686] T 00000000000000000000000000000000 P 66a949c20cbb4391876ef040890466e1 [term 1 LEADER]: Becoming Leader. State: Replica: 66a949c20cbb4391876ef040890466e1, State: Running, Role: LEADER
I20240503 01:59:59.613237 13968 consensus_queue.cc:237] T 00000000000000000000000000000000 P 66a949c20cbb4391876ef040890466e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66a949c20cbb4391876ef040890466e1" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 44351 } }
I20240503 01:59:59.613274 13963 sys_catalog.cc:564] T 00000000000000000000000000000000 P 66a949c20cbb4391876ef040890466e1 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 01:59:59.614679 13969 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66a949c20cbb4391876ef040890466e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "66a949c20cbb4391876ef040890466e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66a949c20cbb4391876ef040890466e1" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 44351 } } }
I20240503 01:59:59.614800 13969 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66a949c20cbb4391876ef040890466e1 [sys.catalog]: This master's current role is: LEADER
I20240503 01:59:59.615065 13970 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66a949c20cbb4391876ef040890466e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 66a949c20cbb4391876ef040890466e1. Latest consensus state: current_term: 1 leader_uuid: "66a949c20cbb4391876ef040890466e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66a949c20cbb4391876ef040890466e1" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 44351 } } }
I20240503 01:59:59.615161 13970 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66a949c20cbb4391876ef040890466e1 [sys.catalog]: This master's current role is: LEADER
I20240503 01:59:59.615131 13978 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 01:59:59.618206 13978 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 01:59:59.622795 13978 catalog_manager.cc:1347] Generated new cluster ID: 80d0aa96df8941d0b68fb21c899b2494
I20240503 01:59:59.622859 13978 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 01:59:59.631184 13978 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 01:59:59.632014 13978 catalog_manager.cc:1504] Loading token signing keys...
I20240503 01:59:59.637017 13978 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 66a949c20cbb4391876ef040890466e1: Generated new TSK 0
I20240503 01:59:59.637591 13978 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 01:59:59.642025 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task42rE7p/build/debug/bin/kudu
/tmp/dist-test-task42rE7p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.129:0
--local_ip_for_outbound_sockets=127.0.99.129
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:44351
--builtin_ntp_servers=127.0.99.148:40197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 01:59:59.744124 13987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:59.744333 13987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:59:59.744402 13987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:59.747291 13987 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:59:59.747359 13987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:59:59.747458 13987 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.129
I20240503 01:59:59.751104 13987 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:40197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:44351
--never_fsync=true
--heap_profile_path=/tmp/kudu.13987
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.129
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:59:59.751892 13987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:59:59.752998 13987 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:59.761502 13992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:59.762792 13993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:59.763711 13995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:59.764698 13987 server_base.cc:1034] running on GCE node
I20240503 01:59:59.765038 13987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:59:59.765559 13987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:59:59.766723 13987 hybrid_clock.cc:648] HybridClock initialized: now 1714701599766713 us; error 24 us; skew 500 ppm
I20240503 01:59:59.768668 13987 webserver.cc:460] Webserver started at http://127.0.99.129:33205/ using document root <none> and password file <none>
I20240503 01:59:59.769181 13987 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:59.769232 13987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:59.769392 13987 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:59:59.770968 13987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "4cd4b11bee2b4878bed3e10d2d76f368"
format_stamp: "Formatted at 2024-05-03 01:59:59 on dist-test-slave-dtz3"
I20240503 01:59:59.771435 13987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "4cd4b11bee2b4878bed3e10d2d76f368"
format_stamp: "Formatted at 2024-05-03 01:59:59 on dist-test-slave-dtz3"
I20240503 01:59:59.774565 13987 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 01:59:59.776563 14001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:59.777763 13987 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:59.777889 13987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "4cd4b11bee2b4878bed3e10d2d76f368"
format_stamp: "Formatted at 2024-05-03 01:59:59 on dist-test-slave-dtz3"
I20240503 01:59:59.777974 13987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:59.790630 13987 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:59.791173 13987 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:59:59.791321 13987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:59.791813 13987 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:59.792618 13987 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:59:59.792721 13987 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:59.792779 13987 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:59:59.792807 13987 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:59.801645 13987 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.129:37199
I20240503 01:59:59.801720 14114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.129:37199 every 8 connection(s)
I20240503 01:59:59.802524 13987 server_base.cc:1166] Dumped server information to /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 01:59:59.807732 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task42rE7p/build/debug/bin/kudu as pid 13987
I20240503 01:59:59.807857 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 01:59:59.810104 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task42rE7p/build/debug/bin/kudu
/tmp/dist-test-task42rE7p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.130:0
--local_ip_for_outbound_sockets=127.0.99.130
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:44351
--builtin_ntp_servers=127.0.99.148:40197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 01:59:59.811374 14115 heartbeater.cc:347] Connected to a master server at 127.0.99.190:44351
I20240503 01:59:59.811616 14115 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:59.812119 14115 heartbeater.cc:511] Master 127.0.99.190:44351 requested a full tablet report, sending...
I20240503 01:59:59.813653 13927 ts_manager.cc:194] Registered new tserver with Master: 4cd4b11bee2b4878bed3e10d2d76f368 (127.0.99.129:37199)
I20240503 01:59:59.815431 13927 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.129:47321
W20240503 01:59:59.917358 14118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:59.917536 14118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:59:59.917580 14118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:59.920336 14118 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:59:59.920403 14118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:59:59.920464 14118 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.130
I20240503 01:59:59.923835 14118 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:40197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:44351
--never_fsync=true
--heap_profile_path=/tmp/kudu.14118
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.130
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:59:59.924521 14118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:59:59.925583 14118 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:59.934353 14126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:59.935240 14123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:59.936481 14124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:59.937455 14118 server_base.cc:1034] running on GCE node
I20240503 01:59:59.937847 14118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:59:59.938341 14118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:59:59.939497 14118 hybrid_clock.cc:648] HybridClock initialized: now 1714701599939480 us; error 30 us; skew 500 ppm
I20240503 01:59:59.941355 14118 webserver.cc:460] Webserver started at http://127.0.99.130:39849/ using document root <none> and password file <none>
I20240503 01:59:59.941874 14118 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:59.941936 14118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:59.942106 14118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:59:59.943619 14118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "bf353eb308de425ea6e214b9c6defab3"
format_stamp: "Formatted at 2024-05-03 01:59:59 on dist-test-slave-dtz3"
I20240503 01:59:59.944058 14118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "bf353eb308de425ea6e214b9c6defab3"
format_stamp: "Formatted at 2024-05-03 01:59:59 on dist-test-slave-dtz3"
I20240503 01:59:59.947211 14118 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 01:59:59.949206 14132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:59.950152 14118 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:59.950277 14118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "bf353eb308de425ea6e214b9c6defab3"
format_stamp: "Formatted at 2024-05-03 01:59:59 on dist-test-slave-dtz3"
I20240503 01:59:59.950359 14118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:59.974851 14118 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:59.975438 14118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:59:59.975615 14118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:59.976187 14118 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:59.977121 14118 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:59:59.977172 14118 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:59.977232 14118 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:59:59.977267 14118 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:59.986255 14118 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.130:36547
I20240503 01:59:59.986308 14245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.130:36547 every 8 connection(s)
I20240503 01:59:59.987190 14118 server_base.cc:1166] Dumped server information to /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 01:59:59.995591 14246 heartbeater.cc:347] Connected to a master server at 127.0.99.190:44351
I20240503 01:59:59.995798 14246 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:59.996276 14246 heartbeater.cc:511] Master 127.0.99.190:44351 requested a full tablet report, sending...
I20240503 01:59:59.996421 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task42rE7p/build/debug/bin/kudu as pid 14118
I20240503 01:59:59.996556 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 01:59:59.997313 13927 ts_manager.cc:194] Registered new tserver with Master: bf353eb308de425ea6e214b9c6defab3 (127.0.99.130:36547)
I20240503 01:59:59.997869 13927 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.130:50193
I20240503 01:59:59.998693 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task42rE7p/build/debug/bin/kudu
/tmp/dist-test-task42rE7p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.131:0
--local_ip_for_outbound_sockets=127.0.99.131
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:44351
--builtin_ntp_servers=127.0.99.148:40197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 02:00:00.101811 14249 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:00:00.102021 14249 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:00:00.102103 14249 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:00:00.105051 14249 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:00:00.105136 14249 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:00:00.105228 14249 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.131
I20240503 02:00:00.108664 14249 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:40197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:44351
--never_fsync=true
--heap_profile_path=/tmp/kudu.14249
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.131
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 02:00:00.109494 14249 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:00:00.110612 14249 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:00:00.119958 14254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:00.121060 14257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:00.121994 14255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:00:00.123014 14249 server_base.cc:1034] running on GCE node
I20240503 02:00:00.123391 14249 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:00:00.123932 14249 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:00:00.125104 14249 hybrid_clock.cc:648] HybridClock initialized: now 1714701600125081 us; error 45 us; skew 500 ppm
I20240503 02:00:00.127022 14249 webserver.cc:460] Webserver started at http://127.0.99.131:36555/ using document root <none> and password file <none>
I20240503 02:00:00.127521 14249 fs_manager.cc:359] Metadata directory not provided
I20240503 02:00:00.127591 14249 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:00:00.127755 14249 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:00:00.129321 14249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "8796d8bfc685436c8889ffa947644a58"
format_stamp: "Formatted at 2024-05-03 02:00:00 on dist-test-slave-dtz3"
I20240503 02:00:00.129781 14249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "8796d8bfc685436c8889ffa947644a58"
format_stamp: "Formatted at 2024-05-03 02:00:00 on dist-test-slave-dtz3"
I20240503 02:00:00.132995 14249 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240503 02:00:00.135042 14263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:00.135975 14249 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:00.136073 14249 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "8796d8bfc685436c8889ffa947644a58"
format_stamp: "Formatted at 2024-05-03 02:00:00 on dist-test-slave-dtz3"
I20240503 02:00:00.136158 14249 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:00:00.154045 14249 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:00:00.154599 14249 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:00:00.154743 14249 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:00:00.155265 14249 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:00:00.156100 14249 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:00:00.156162 14249 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:00.156239 14249 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:00:00.156265 14249 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:00.165094 14249 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.131:45811
I20240503 02:00:00.165122 14376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.131:45811 every 8 connection(s)
I20240503 02:00:00.165948 14249 server_base.cc:1166] Dumped server information to /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 02:00:00.174067 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task42rE7p/build/debug/bin/kudu as pid 14249
I20240503 02:00:00.174139 14377 heartbeater.cc:347] Connected to a master server at 127.0.99.190:44351
I20240503 02:00:00.174178 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 02:00:00.174335 14377 heartbeater.cc:464] Registering TS with master...
I20240503 02:00:00.174800 14377 heartbeater.cc:511] Master 127.0.99.190:44351 requested a full tablet report, sending...
I20240503 02:00:00.175663 13927 ts_manager.cc:194] Registered new tserver with Master: 8796d8bfc685436c8889ffa947644a58 (127.0.99.131:45811)
I20240503 02:00:00.176046 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task42rE7p/build/debug/bin/kudu
/tmp/dist-test-task42rE7p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.132:0
--local_ip_for_outbound_sockets=127.0.99.132
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:44351
--builtin_ntp_servers=127.0.99.148:40197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:00:00.176201 13927 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.131:55427
W20240503 02:00:00.279738 14380 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:00:00.279946 14380 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:00:00.279989 14380 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:00:00.282895 14380 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:00:00.282959 14380 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:00:00.283032 14380 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.132
I20240503 02:00:00.286509 14380 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:40197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:44351
--never_fsync=true
--heap_profile_path=/tmp/kudu.14380
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.132
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 02:00:00.287330 14380 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:00:00.288396 14380 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:00:00.297598 14388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:00.298791 14386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:00.299755 14385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:00:00.300900 14380 server_base.cc:1034] running on GCE node
I20240503 02:00:00.301303 14380 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:00:00.301856 14380 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:00:00.303022 14380 hybrid_clock.cc:648] HybridClock initialized: now 1714701600303000 us; error 37 us; skew 500 ppm
I20240503 02:00:00.304977 14380 webserver.cc:460] Webserver started at http://127.0.99.132:42465/ using document root <none> and password file <none>
I20240503 02:00:00.305531 14380 fs_manager.cc:359] Metadata directory not provided
I20240503 02:00:00.305595 14380 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:00:00.305783 14380 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:00:00.307386 14380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "eccecde0730943b8a29023ca7f42d4ff"
format_stamp: "Formatted at 2024-05-03 02:00:00 on dist-test-slave-dtz3"
I20240503 02:00:00.307859 14380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "eccecde0730943b8a29023ca7f42d4ff"
format_stamp: "Formatted at 2024-05-03 02:00:00 on dist-test-slave-dtz3"
I20240503 02:00:00.311193 14380 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20240503 02:00:00.313452 14394 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:00.314536 14380 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:00.314646 14380 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "eccecde0730943b8a29023ca7f42d4ff"
format_stamp: "Formatted at 2024-05-03 02:00:00 on dist-test-slave-dtz3"
I20240503 02:00:00.314740 14380 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:00:00.335788 14380 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:00:00.336510 14380 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:00:00.336735 14380 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:00:00.337281 14380 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:00:00.338151 14380 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:00:00.338199 14380 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:00.338260 14380 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:00:00.338292 14380 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:00.347257 14380 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.132:42443
I20240503 02:00:00.347285 14507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.132:42443 every 8 connection(s)
I20240503 02:00:00.348192 14380 server_base.cc:1166] Dumped server information to /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 02:00:00.351711 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task42rE7p/build/debug/bin/kudu as pid 14380
I20240503 02:00:00.351840 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 02:00:00.357414 14508 heartbeater.cc:347] Connected to a master server at 127.0.99.190:44351
I20240503 02:00:00.357623 14508 heartbeater.cc:464] Registering TS with master...
I20240503 02:00:00.358119 14508 heartbeater.cc:511] Master 127.0.99.190:44351 requested a full tablet report, sending...
I20240503 02:00:00.359114 13927 ts_manager.cc:194] Registered new tserver with Master: eccecde0730943b8a29023ca7f42d4ff (127.0.99.132:42443)
I20240503 02:00:00.359700 13927 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.132:60643
I20240503 02:00:00.366024 398 external_mini_cluster.cc:930] 4 TS(s) registered with all masters
I20240503 02:00:00.375432 13927 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48154:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240503 02:00:00.391273 14180 tablet_service.cc:1467] Processing CreateTablet for tablet 67434687758243feab59f655a925bce9 (DEFAULT_TABLE table=TestTable [id=e8887fe4dc524c1a9586a5f473169918]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:00.392103 14180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67434687758243feab59f655a925bce9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:00.397001 14521 tablet_bootstrap.cc:492] T 67434687758243feab59f655a925bce9 P bf353eb308de425ea6e214b9c6defab3: Bootstrap starting.
I20240503 02:00:00.398748 14521 tablet_bootstrap.cc:654] T 67434687758243feab59f655a925bce9 P bf353eb308de425ea6e214b9c6defab3: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:00.399407 14521 log.cc:824] T 67434687758243feab59f655a925bce9 P bf353eb308de425ea6e214b9c6defab3: Log is configured to *not* fsync() on all Append() calls
I20240503 02:00:00.400835 14521 tablet_bootstrap.cc:492] T 67434687758243feab59f655a925bce9 P bf353eb308de425ea6e214b9c6defab3: No bootstrap required, opened a new log
I20240503 02:00:00.401001 14521 ts_tablet_manager.cc:1367] T 67434687758243feab59f655a925bce9 P bf353eb308de425ea6e214b9c6defab3: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.001s
I20240503 02:00:00.403818 14521 raft_consensus.cc:348] T 67434687758243feab59f655a925bce9 P bf353eb308de425ea6e214b9c6defab3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf353eb308de425ea6e214b9c6defab3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36547 } }
I20240503 02:00:00.403992 14521 raft_consensus.cc:374] T 67434687758243feab59f655a925bce9 P bf353eb308de425ea6e214b9c6defab3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:00.404052 14521 raft_consensus.cc:729] T 67434687758243feab59f655a925bce9 P bf353eb308de425ea6e214b9c6defab3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf353eb308de425ea6e214b9c6defab3, State: Initialized, Role: FOLLOWER
I20240503 02:00:00.404420 14521 consensus_queue.cc:260] T 67434687758243feab59f655a925bce9 P bf353eb308de425ea6e214b9c6defab3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf353eb308de425ea6e214b9c6defab3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36547 } }
I20240503 02:00:00.404539 14521 raft_consensus.cc:388] T 67434687758243feab59f655a925bce9 P bf353eb308de425ea6e214b9c6defab3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:00:00.404600 14521 raft_consensus.cc:482] T 67434687758243feab59f655a925bce9 P bf353eb308de425ea6e214b9c6defab3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:00:00.404721 14521 raft_consensus.cc:3037] T 67434687758243feab59f655a925bce9 P bf353eb308de425ea6e214b9c6defab3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:00.405472 14521 raft_consensus.cc:504] T 67434687758243feab59f655a925bce9 P bf353eb308de425ea6e214b9c6defab3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf353eb308de425ea6e214b9c6defab3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36547 } }
I20240503 02:00:00.405757 14521 leader_election.cc:302] T 67434687758243feab59f655a925bce9 P bf353eb308de425ea6e214b9c6defab3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bf353eb308de425ea6e214b9c6defab3; no voters:
I20240503 02:00:00.406021 14521 leader_election.cc:288] T 67434687758243feab59f655a925bce9 P bf353eb308de425ea6e214b9c6defab3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:00:00.406131 14523 raft_consensus.cc:2781] T 67434687758243feab59f655a925bce9 P bf353eb308de425ea6e214b9c6defab3 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:00.406360 14523 raft_consensus.cc:686] T 67434687758243feab59f655a925bce9 P bf353eb308de425ea6e214b9c6defab3 [term 1 LEADER]: Becoming Leader. State: Replica: bf353eb308de425ea6e214b9c6defab3, State: Running, Role: LEADER
I20240503 02:00:00.406682 14246 heartbeater.cc:503] Master 127.0.99.190:44351 was elected leader, sending a full tablet report...
I20240503 02:00:00.406635 14523 consensus_queue.cc:237] T 67434687758243feab59f655a925bce9 P bf353eb308de425ea6e214b9c6defab3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf353eb308de425ea6e214b9c6defab3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36547 } }
I20240503 02:00:00.406806 14521 ts_tablet_manager.cc:1398] T 67434687758243feab59f655a925bce9 P bf353eb308de425ea6e214b9c6defab3: Time spent starting tablet: real 0.006s user 0.000s sys 0.006s
I20240503 02:00:00.409884 13927 catalog_manager.cc:5484] T 67434687758243feab59f655a925bce9 P bf353eb308de425ea6e214b9c6defab3 reported cstate change: term changed from 0 to 1, leader changed from <none> to bf353eb308de425ea6e214b9c6defab3 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "bf353eb308de425ea6e214b9c6defab3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf353eb308de425ea6e214b9c6defab3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36547 } health_report { overall_health: HEALTHY } } }
I20240503 02:00:00.418169 398 external_mini_cluster.cc:930] 4 TS(s) registered with all masters
I20240503 02:00:00.419519 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 4cd4b11bee2b4878bed3e10d2d76f368 to finish bootstrapping
I20240503 02:00:00.425112 398 ts_itest-base.cc:233] Waiting for 1 tablets on tserver bf353eb308de425ea6e214b9c6defab3 to finish bootstrapping
I20240503 02:00:00.429780 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 8796d8bfc685436c8889ffa947644a58 to finish bootstrapping
I20240503 02:00:00.435138 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver eccecde0730943b8a29023ca7f42d4ff to finish bootstrapping
I20240503 02:00:00.440492 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task42rE7p/build/debug/bin/kudu with pid 14249
I20240503 02:00:00.446563 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task42rE7p/build/debug/bin/kudu with pid 14380
I20240503 02:00:00.817852 14115 heartbeater.cc:503] Master 127.0.99.190:44351 was elected leader, sending a full tablet report...
I20240503 02:00:04.205612 13927 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48154:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:00:04.220254 14048 tablet_service.cc:1467] Processing CreateTablet for tablet 9a3118fc5d8740b68dbcfed1056069b2 (DEFAULT_TABLE table=rebalance_test_table_0 [id=b2cfd5eb27e64e97bf78d62f6aa62114]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:04.220242 14049 tablet_service.cc:1467] Processing CreateTablet for tablet 3a18f1f16b5a42578a713f0c133c8ef7 (DEFAULT_TABLE table=rebalance_test_table_0 [id=b2cfd5eb27e64e97bf78d62f6aa62114]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:04.221510 14049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a18f1f16b5a42578a713f0c133c8ef7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:04.221920 14048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a3118fc5d8740b68dbcfed1056069b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:04.220212 14047 tablet_service.cc:1467] Processing CreateTablet for tablet b215d743523b43e295662f1e351c43c5 (DEFAULT_TABLE table=rebalance_test_table_0 [id=b2cfd5eb27e64e97bf78d62f6aa62114]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:04.226670 14047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b215d743523b43e295662f1e351c43c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:04.228018 14538 tablet_bootstrap.cc:492] T 9a3118fc5d8740b68dbcfed1056069b2 P 4cd4b11bee2b4878bed3e10d2d76f368: Bootstrap starting.
I20240503 02:00:04.230147 14538 tablet_bootstrap.cc:654] T 9a3118fc5d8740b68dbcfed1056069b2 P 4cd4b11bee2b4878bed3e10d2d76f368: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:04.230878 14538 log.cc:824] T 9a3118fc5d8740b68dbcfed1056069b2 P 4cd4b11bee2b4878bed3e10d2d76f368: Log is configured to *not* fsync() on all Append() calls
I20240503 02:00:04.232692 14538 tablet_bootstrap.cc:492] T 9a3118fc5d8740b68dbcfed1056069b2 P 4cd4b11bee2b4878bed3e10d2d76f368: No bootstrap required, opened a new log
I20240503 02:00:04.232863 14538 ts_tablet_manager.cc:1367] T 9a3118fc5d8740b68dbcfed1056069b2 P 4cd4b11bee2b4878bed3e10d2d76f368: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20240503 02:00:04.235574 14538 raft_consensus.cc:348] T 9a3118fc5d8740b68dbcfed1056069b2 P 4cd4b11bee2b4878bed3e10d2d76f368 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cd4b11bee2b4878bed3e10d2d76f368" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37199 } }
I20240503 02:00:04.235761 14538 raft_consensus.cc:374] T 9a3118fc5d8740b68dbcfed1056069b2 P 4cd4b11bee2b4878bed3e10d2d76f368 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:04.235813 14538 raft_consensus.cc:729] T 9a3118fc5d8740b68dbcfed1056069b2 P 4cd4b11bee2b4878bed3e10d2d76f368 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cd4b11bee2b4878bed3e10d2d76f368, State: Initialized, Role: FOLLOWER
I20240503 02:00:04.236294 14538 consensus_queue.cc:260] T 9a3118fc5d8740b68dbcfed1056069b2 P 4cd4b11bee2b4878bed3e10d2d76f368 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cd4b11bee2b4878bed3e10d2d76f368" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37199 } }
I20240503 02:00:04.236416 14538 raft_consensus.cc:388] T 9a3118fc5d8740b68dbcfed1056069b2 P 4cd4b11bee2b4878bed3e10d2d76f368 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:00:04.236461 14538 raft_consensus.cc:482] T 9a3118fc5d8740b68dbcfed1056069b2 P 4cd4b11bee2b4878bed3e10d2d76f368 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:00:04.236577 14538 raft_consensus.cc:3037] T 9a3118fc5d8740b68dbcfed1056069b2 P 4cd4b11bee2b4878bed3e10d2d76f368 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:04.237494 14538 raft_consensus.cc:504] T 9a3118fc5d8740b68dbcfed1056069b2 P 4cd4b11bee2b4878bed3e10d2d76f368 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cd4b11bee2b4878bed3e10d2d76f368" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37199 } }
I20240503 02:00:04.237879 14538 leader_election.cc:302] T 9a3118fc5d8740b68dbcfed1056069b2 P 4cd4b11bee2b4878bed3e10d2d76f368 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4cd4b11bee2b4878bed3e10d2d76f368; no voters:
I20240503 02:00:04.238190 14538 leader_election.cc:288] T 9a3118fc5d8740b68dbcfed1056069b2 P 4cd4b11bee2b4878bed3e10d2d76f368 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:00:04.238382 14540 raft_consensus.cc:2781] T 9a3118fc5d8740b68dbcfed1056069b2 P 4cd4b11bee2b4878bed3e10d2d76f368 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:04.238709 14540 raft_consensus.cc:686] T 9a3118fc5d8740b68dbcfed1056069b2 P 4cd4b11bee2b4878bed3e10d2d76f368 [term 1 LEADER]: Becoming Leader. State: Replica: 4cd4b11bee2b4878bed3e10d2d76f368, State: Running, Role: LEADER
I20240503 02:00:04.239049 14538 ts_tablet_manager.cc:1398] T 9a3118fc5d8740b68dbcfed1056069b2 P 4cd4b11bee2b4878bed3e10d2d76f368: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240503 02:00:04.239090 14540 consensus_queue.cc:237] T 9a3118fc5d8740b68dbcfed1056069b2 P 4cd4b11bee2b4878bed3e10d2d76f368 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cd4b11bee2b4878bed3e10d2d76f368" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37199 } }
I20240503 02:00:04.239279 14538 tablet_bootstrap.cc:492] T 3a18f1f16b5a42578a713f0c133c8ef7 P 4cd4b11bee2b4878bed3e10d2d76f368: Bootstrap starting.
I20240503 02:00:04.240106 14538 tablet_bootstrap.cc:654] T 3a18f1f16b5a42578a713f0c133c8ef7 P 4cd4b11bee2b4878bed3e10d2d76f368: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:04.241370 14538 tablet_bootstrap.cc:492] T 3a18f1f16b5a42578a713f0c133c8ef7 P 4cd4b11bee2b4878bed3e10d2d76f368: No bootstrap required, opened a new log
I20240503 02:00:04.241513 14538 ts_tablet_manager.cc:1367] T 3a18f1f16b5a42578a713f0c133c8ef7 P 4cd4b11bee2b4878bed3e10d2d76f368: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 02:00:04.242237 14538 raft_consensus.cc:348] T 3a18f1f16b5a42578a713f0c133c8ef7 P 4cd4b11bee2b4878bed3e10d2d76f368 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cd4b11bee2b4878bed3e10d2d76f368" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37199 } }
I20240503 02:00:04.242314 14538 raft_consensus.cc:374] T 3a18f1f16b5a42578a713f0c133c8ef7 P 4cd4b11bee2b4878bed3e10d2d76f368 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:04.242354 14538 raft_consensus.cc:729] T 3a18f1f16b5a42578a713f0c133c8ef7 P 4cd4b11bee2b4878bed3e10d2d76f368 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cd4b11bee2b4878bed3e10d2d76f368, State: Initialized, Role: FOLLOWER
I20240503 02:00:04.242300 13927 catalog_manager.cc:5484] T 9a3118fc5d8740b68dbcfed1056069b2 P 4cd4b11bee2b4878bed3e10d2d76f368 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4cd4b11bee2b4878bed3e10d2d76f368 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "4cd4b11bee2b4878bed3e10d2d76f368" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cd4b11bee2b4878bed3e10d2d76f368" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37199 } health_report { overall_health: HEALTHY } } }
I20240503 02:00:04.242480 14538 consensus_queue.cc:260] T 3a18f1f16b5a42578a713f0c133c8ef7 P 4cd4b11bee2b4878bed3e10d2d76f368 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cd4b11bee2b4878bed3e10d2d76f368" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37199 } }
I20240503 02:00:04.242558 14538 raft_consensus.cc:388] T 3a18f1f16b5a42578a713f0c133c8ef7 P 4cd4b11bee2b4878bed3e10d2d76f368 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:00:04.242602 14538 raft_consensus.cc:482] T 3a18f1f16b5a42578a713f0c133c8ef7 P 4cd4b11bee2b4878bed3e10d2d76f368 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:00:04.242650 14538 raft_consensus.cc:3037] T 3a18f1f16b5a42578a713f0c133c8ef7 P 4cd4b11bee2b4878bed3e10d2d76f368 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:04.243350 14538 raft_consensus.cc:504] T 3a18f1f16b5a42578a713f0c133c8ef7 P 4cd4b11bee2b4878bed3e10d2d76f368 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cd4b11bee2b4878bed3e10d2d76f368" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37199 } }
I20240503 02:00:04.243484 14538 leader_election.cc:302] T 3a18f1f16b5a42578a713f0c133c8ef7 P 4cd4b11bee2b4878bed3e10d2d76f368 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4cd4b11bee2b4878bed3e10d2d76f368; no voters:
I20240503 02:00:04.243583 14538 leader_election.cc:288] T 3a18f1f16b5a42578a713f0c133c8ef7 P 4cd4b11bee2b4878bed3e10d2d76f368 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:00:04.243638 14540 raft_consensus.cc:2781] T 3a18f1f16b5a42578a713f0c133c8ef7 P 4cd4b11bee2b4878bed3e10d2d76f368 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:04.243763 14540 raft_consensus.cc:686] T 3a18f1f16b5a42578a713f0c133c8ef7 P 4cd4b11bee2b4878bed3e10d2d76f368 [term 1 LEADER]: Becoming Leader. State: Replica: 4cd4b11bee2b4878bed3e10d2d76f368, State: Running, Role: LEADER
I20240503 02:00:04.243918 14540 consensus_queue.cc:237] T 3a18f1f16b5a42578a713f0c133c8ef7 P 4cd4b11bee2b4878bed3e10d2d76f368 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cd4b11bee2b4878bed3e10d2d76f368" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37199 } }
I20240503 02:00:04.243947 14538 ts_tablet_manager.cc:1398] T 3a18f1f16b5a42578a713f0c133c8ef7 P 4cd4b11bee2b4878bed3e10d2d76f368: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 02:00:04.244148 14538 tablet_bootstrap.cc:492] T b215d743523b43e295662f1e351c43c5 P 4cd4b11bee2b4878bed3e10d2d76f368: Bootstrap starting.
I20240503 02:00:04.245105 14538 tablet_bootstrap.cc:654] T b215d743523b43e295662f1e351c43c5 P 4cd4b11bee2b4878bed3e10d2d76f368: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:04.246253 14538 tablet_bootstrap.cc:492] T b215d743523b43e295662f1e351c43c5 P 4cd4b11bee2b4878bed3e10d2d76f368: No bootstrap required, opened a new log
I20240503 02:00:04.246316 14538 ts_tablet_manager.cc:1367] T b215d743523b43e295662f1e351c43c5 P 4cd4b11bee2b4878bed3e10d2d76f368: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:04.246527 13927 catalog_manager.cc:5484] T 3a18f1f16b5a42578a713f0c133c8ef7 P 4cd4b11bee2b4878bed3e10d2d76f368 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4cd4b11bee2b4878bed3e10d2d76f368 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "4cd4b11bee2b4878bed3e10d2d76f368" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cd4b11bee2b4878bed3e10d2d76f368" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37199 } health_report { overall_health: HEALTHY } } }
I20240503 02:00:04.246628 14538 raft_consensus.cc:348] T b215d743523b43e295662f1e351c43c5 P 4cd4b11bee2b4878bed3e10d2d76f368 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cd4b11bee2b4878bed3e10d2d76f368" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37199 } }
I20240503 02:00:04.246707 14538 raft_consensus.cc:374] T b215d743523b43e295662f1e351c43c5 P 4cd4b11bee2b4878bed3e10d2d76f368 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:04.246744 14538 raft_consensus.cc:729] T b215d743523b43e295662f1e351c43c5 P 4cd4b11bee2b4878bed3e10d2d76f368 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cd4b11bee2b4878bed3e10d2d76f368, State: Initialized, Role: FOLLOWER
I20240503 02:00:04.246852 14538 consensus_queue.cc:260] T b215d743523b43e295662f1e351c43c5 P 4cd4b11bee2b4878bed3e10d2d76f368 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cd4b11bee2b4878bed3e10d2d76f368" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37199 } }
I20240503 02:00:04.246908 14538 raft_consensus.cc:388] T b215d743523b43e295662f1e351c43c5 P 4cd4b11bee2b4878bed3e10d2d76f368 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:00:04.246930 14538 raft_consensus.cc:482] T b215d743523b43e295662f1e351c43c5 P 4cd4b11bee2b4878bed3e10d2d76f368 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:00:04.246977 14538 raft_consensus.cc:3037] T b215d743523b43e295662f1e351c43c5 P 4cd4b11bee2b4878bed3e10d2d76f368 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:04.248035 14538 raft_consensus.cc:504] T b215d743523b43e295662f1e351c43c5 P 4cd4b11bee2b4878bed3e10d2d76f368 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cd4b11bee2b4878bed3e10d2d76f368" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37199 } }
I20240503 02:00:04.248154 14538 leader_election.cc:302] T b215d743523b43e295662f1e351c43c5 P 4cd4b11bee2b4878bed3e10d2d76f368 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4cd4b11bee2b4878bed3e10d2d76f368; no voters:
I20240503 02:00:04.248275 14538 leader_election.cc:288] T b215d743523b43e295662f1e351c43c5 P 4cd4b11bee2b4878bed3e10d2d76f368 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:00:04.248329 14540 raft_consensus.cc:2781] T b215d743523b43e295662f1e351c43c5 P 4cd4b11bee2b4878bed3e10d2d76f368 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:04.248443 14540 raft_consensus.cc:686] T b215d743523b43e295662f1e351c43c5 P 4cd4b11bee2b4878bed3e10d2d76f368 [term 1 LEADER]: Becoming Leader. State: Replica: 4cd4b11bee2b4878bed3e10d2d76f368, State: Running, Role: LEADER
I20240503 02:00:04.248564 14540 consensus_queue.cc:237] T b215d743523b43e295662f1e351c43c5 P 4cd4b11bee2b4878bed3e10d2d76f368 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cd4b11bee2b4878bed3e10d2d76f368" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37199 } }
I20240503 02:00:04.249061 14538 ts_tablet_manager.cc:1398] T b215d743523b43e295662f1e351c43c5 P 4cd4b11bee2b4878bed3e10d2d76f368: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:04.249823 13927 catalog_manager.cc:5484] T b215d743523b43e295662f1e351c43c5 P 4cd4b11bee2b4878bed3e10d2d76f368 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4cd4b11bee2b4878bed3e10d2d76f368 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "4cd4b11bee2b4878bed3e10d2d76f368" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cd4b11bee2b4878bed3e10d2d76f368" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37199 } health_report { overall_health: HEALTHY } } }
I20240503 02:00:04.427492 13926 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48154:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:00:04.434279 14178 tablet_service.cc:1467] Processing CreateTablet for tablet a09a07623e334e3186aa1706b8b24a5d (DEFAULT_TABLE table=rebalance_test_table_1 [id=4f2c23da333840fc9ee80731edc5d68f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:04.434252 14180 tablet_service.cc:1467] Processing CreateTablet for tablet 939505a3617342259e1789eb0dd9bb27 (DEFAULT_TABLE table=rebalance_test_table_1 [id=4f2c23da333840fc9ee80731edc5d68f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:04.434257 14179 tablet_service.cc:1467] Processing CreateTablet for tablet 6b2be0c3787d47788616c682f3492a52 (DEFAULT_TABLE table=rebalance_test_table_1 [id=4f2c23da333840fc9ee80731edc5d68f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:04.434794 14178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a09a07623e334e3186aa1706b8b24a5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:04.435036 14180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 939505a3617342259e1789eb0dd9bb27. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:04.435312 14179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b2be0c3787d47788616c682f3492a52. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:04.437748 14568 tablet_bootstrap.cc:492] T a09a07623e334e3186aa1706b8b24a5d P bf353eb308de425ea6e214b9c6defab3: Bootstrap starting.
I20240503 02:00:04.438624 14568 tablet_bootstrap.cc:654] T a09a07623e334e3186aa1706b8b24a5d P bf353eb308de425ea6e214b9c6defab3: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:04.439782 14568 tablet_bootstrap.cc:492] T a09a07623e334e3186aa1706b8b24a5d P bf353eb308de425ea6e214b9c6defab3: No bootstrap required, opened a new log
I20240503 02:00:04.439873 14568 ts_tablet_manager.cc:1367] T a09a07623e334e3186aa1706b8b24a5d P bf353eb308de425ea6e214b9c6defab3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:04.440483 14568 raft_consensus.cc:348] T a09a07623e334e3186aa1706b8b24a5d P bf353eb308de425ea6e214b9c6defab3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf353eb308de425ea6e214b9c6defab3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36547 } }
I20240503 02:00:04.440591 14568 raft_consensus.cc:374] T a09a07623e334e3186aa1706b8b24a5d P bf353eb308de425ea6e214b9c6defab3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:04.440627 14568 raft_consensus.cc:729] T a09a07623e334e3186aa1706b8b24a5d P bf353eb308de425ea6e214b9c6defab3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf353eb308de425ea6e214b9c6defab3, State: Initialized, Role: FOLLOWER
I20240503 02:00:04.440826 14568 consensus_queue.cc:260] T a09a07623e334e3186aa1706b8b24a5d P bf353eb308de425ea6e214b9c6defab3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf353eb308de425ea6e214b9c6defab3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36547 } }
I20240503 02:00:04.440943 14568 raft_consensus.cc:388] T a09a07623e334e3186aa1706b8b24a5d P bf353eb308de425ea6e214b9c6defab3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:00:04.441005 14568 raft_consensus.cc:482] T a09a07623e334e3186aa1706b8b24a5d P bf353eb308de425ea6e214b9c6defab3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:00:04.441058 14568 raft_consensus.cc:3037] T a09a07623e334e3186aa1706b8b24a5d P bf353eb308de425ea6e214b9c6defab3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:04.441859 14568 raft_consensus.cc:504] T a09a07623e334e3186aa1706b8b24a5d P bf353eb308de425ea6e214b9c6defab3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf353eb308de425ea6e214b9c6defab3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36547 } }
I20240503 02:00:04.441984 14568 leader_election.cc:302] T a09a07623e334e3186aa1706b8b24a5d P bf353eb308de425ea6e214b9c6defab3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bf353eb308de425ea6e214b9c6defab3; no voters:
I20240503 02:00:04.442140 14568 leader_election.cc:288] T a09a07623e334e3186aa1706b8b24a5d P bf353eb308de425ea6e214b9c6defab3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:00:04.442325 14570 raft_consensus.cc:2781] T a09a07623e334e3186aa1706b8b24a5d P bf353eb308de425ea6e214b9c6defab3 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:04.442441 14568 ts_tablet_manager.cc:1398] T a09a07623e334e3186aa1706b8b24a5d P bf353eb308de425ea6e214b9c6defab3: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 02:00:04.442519 14570 raft_consensus.cc:686] T a09a07623e334e3186aa1706b8b24a5d P bf353eb308de425ea6e214b9c6defab3 [term 1 LEADER]: Becoming Leader. State: Replica: bf353eb308de425ea6e214b9c6defab3, State: Running, Role: LEADER
I20240503 02:00:04.442625 14568 tablet_bootstrap.cc:492] T 6b2be0c3787d47788616c682f3492a52 P bf353eb308de425ea6e214b9c6defab3: Bootstrap starting.
I20240503 02:00:04.442747 14570 consensus_queue.cc:237] T a09a07623e334e3186aa1706b8b24a5d P bf353eb308de425ea6e214b9c6defab3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf353eb308de425ea6e214b9c6defab3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36547 } }
I20240503 02:00:04.443651 14568 tablet_bootstrap.cc:654] T 6b2be0c3787d47788616c682f3492a52 P bf353eb308de425ea6e214b9c6defab3: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:04.444537 13927 catalog_manager.cc:5484] T a09a07623e334e3186aa1706b8b24a5d P bf353eb308de425ea6e214b9c6defab3 reported cstate change: term changed from 0 to 1, leader changed from <none> to bf353eb308de425ea6e214b9c6defab3 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "bf353eb308de425ea6e214b9c6defab3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf353eb308de425ea6e214b9c6defab3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36547 } health_report { overall_health: HEALTHY } } }
I20240503 02:00:04.445165 14568 tablet_bootstrap.cc:492] T 6b2be0c3787d47788616c682f3492a52 P bf353eb308de425ea6e214b9c6defab3: No bootstrap required, opened a new log
I20240503 02:00:04.445241 14568 ts_tablet_manager.cc:1367] T 6b2be0c3787d47788616c682f3492a52 P bf353eb308de425ea6e214b9c6defab3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:04.445860 14568 raft_consensus.cc:348] T 6b2be0c3787d47788616c682f3492a52 P bf353eb308de425ea6e214b9c6defab3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf353eb308de425ea6e214b9c6defab3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36547 } }
I20240503 02:00:04.445981 14568 raft_consensus.cc:374] T 6b2be0c3787d47788616c682f3492a52 P bf353eb308de425ea6e214b9c6defab3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:04.446018 14568 raft_consensus.cc:729] T 6b2be0c3787d47788616c682f3492a52 P bf353eb308de425ea6e214b9c6defab3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf353eb308de425ea6e214b9c6defab3, State: Initialized, Role: FOLLOWER
I20240503 02:00:04.446187 14568 consensus_queue.cc:260] T 6b2be0c3787d47788616c682f3492a52 P bf353eb308de425ea6e214b9c6defab3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf353eb308de425ea6e214b9c6defab3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36547 } }
I20240503 02:00:04.446286 14568 raft_consensus.cc:388] T 6b2be0c3787d47788616c682f3492a52 P bf353eb308de425ea6e214b9c6defab3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:00:04.446372 14568 raft_consensus.cc:482] T 6b2be0c3787d47788616c682f3492a52 P bf353eb308de425ea6e214b9c6defab3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:00:04.446437 14568 raft_consensus.cc:3037] T 6b2be0c3787d47788616c682f3492a52 P bf353eb308de425ea6e214b9c6defab3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:04.447517 14568 raft_consensus.cc:504] T 6b2be0c3787d47788616c682f3492a52 P bf353eb308de425ea6e214b9c6defab3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf353eb308de425ea6e214b9c6defab3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36547 } }
I20240503 02:00:04.447660 14568 leader_election.cc:302] T 6b2be0c3787d47788616c682f3492a52 P bf353eb308de425ea6e214b9c6defab3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bf353eb308de425ea6e214b9c6defab3; no voters:
I20240503 02:00:04.447764 14568 leader_election.cc:288] T 6b2be0c3787d47788616c682f3492a52 P bf353eb308de425ea6e214b9c6defab3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:00:04.448077 14574 raft_consensus.cc:2781] T 6b2be0c3787d47788616c682f3492a52 P bf353eb308de425ea6e214b9c6defab3 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:04.448228 14574 raft_consensus.cc:686] T 6b2be0c3787d47788616c682f3492a52 P bf353eb308de425ea6e214b9c6defab3 [term 1 LEADER]: Becoming Leader. State: Replica: bf353eb308de425ea6e214b9c6defab3, State: Running, Role: LEADER
I20240503 02:00:04.448364 14574 consensus_queue.cc:237] T 6b2be0c3787d47788616c682f3492a52 P bf353eb308de425ea6e214b9c6defab3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf353eb308de425ea6e214b9c6defab3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36547 } }
I20240503 02:00:04.448096 14568 ts_tablet_manager.cc:1398] T 6b2be0c3787d47788616c682f3492a52 P bf353eb308de425ea6e214b9c6defab3: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20240503 02:00:04.448990 14568 tablet_bootstrap.cc:492] T 939505a3617342259e1789eb0dd9bb27 P bf353eb308de425ea6e214b9c6defab3: Bootstrap starting.
I20240503 02:00:04.450218 14568 tablet_bootstrap.cc:654] T 939505a3617342259e1789eb0dd9bb27 P bf353eb308de425ea6e214b9c6defab3: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:04.451923 14568 tablet_bootstrap.cc:492] T 939505a3617342259e1789eb0dd9bb27 P bf353eb308de425ea6e214b9c6defab3: No bootstrap required, opened a new log
I20240503 02:00:04.452091 14568 ts_tablet_manager.cc:1367] T 939505a3617342259e1789eb0dd9bb27 P bf353eb308de425ea6e214b9c6defab3: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.003s
I20240503 02:00:04.452605 14568 raft_consensus.cc:348] T 939505a3617342259e1789eb0dd9bb27 P bf353eb308de425ea6e214b9c6defab3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf353eb308de425ea6e214b9c6defab3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36547 } }
I20240503 02:00:04.452764 14568 raft_consensus.cc:374] T 939505a3617342259e1789eb0dd9bb27 P bf353eb308de425ea6e214b9c6defab3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:04.452837 14568 raft_consensus.cc:729] T 939505a3617342259e1789eb0dd9bb27 P bf353eb308de425ea6e214b9c6defab3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf353eb308de425ea6e214b9c6defab3, State: Initialized, Role: FOLLOWER
I20240503 02:00:04.452998 14568 consensus_queue.cc:260] T 939505a3617342259e1789eb0dd9bb27 P bf353eb308de425ea6e214b9c6defab3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf353eb308de425ea6e214b9c6defab3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36547 } }
I20240503 02:00:04.453092 14568 raft_consensus.cc:388] T 939505a3617342259e1789eb0dd9bb27 P bf353eb308de425ea6e214b9c6defab3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:00:04.453202 14568 raft_consensus.cc:482] T 939505a3617342259e1789eb0dd9bb27 P bf353eb308de425ea6e214b9c6defab3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:00:04.453310 14568 raft_consensus.cc:3037] T 939505a3617342259e1789eb0dd9bb27 P bf353eb308de425ea6e214b9c6defab3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:04.454269 14568 raft_consensus.cc:504] T 939505a3617342259e1789eb0dd9bb27 P bf353eb308de425ea6e214b9c6defab3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf353eb308de425ea6e214b9c6defab3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36547 } }
I20240503 02:00:04.454406 14568 leader_election.cc:302] T 939505a3617342259e1789eb0dd9bb27 P bf353eb308de425ea6e214b9c6defab3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bf353eb308de425ea6e214b9c6defab3; no voters:
I20240503 02:00:04.454510 14568 leader_election.cc:288] T 939505a3617342259e1789eb0dd9bb27 P bf353eb308de425ea6e214b9c6defab3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:00:04.454663 14570 raft_consensus.cc:2781] T 939505a3617342259e1789eb0dd9bb27 P bf353eb308de425ea6e214b9c6defab3 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:04.454725 14568 ts_tablet_manager.cc:1398] T 939505a3617342259e1789eb0dd9bb27 P bf353eb308de425ea6e214b9c6defab3: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:04.454866 14570 raft_consensus.cc:686] T 939505a3617342259e1789eb0dd9bb27 P bf353eb308de425ea6e214b9c6defab3 [term 1 LEADER]: Becoming Leader. State: Replica: bf353eb308de425ea6e214b9c6defab3, State: Running, Role: LEADER
I20240503 02:00:04.455040 14570 consensus_queue.cc:237] T 939505a3617342259e1789eb0dd9bb27 P bf353eb308de425ea6e214b9c6defab3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf353eb308de425ea6e214b9c6defab3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36547 } }
I20240503 02:00:04.455180 13927 catalog_manager.cc:5484] T 6b2be0c3787d47788616c682f3492a52 P bf353eb308de425ea6e214b9c6defab3 reported cstate change: term changed from 0 to 1, leader changed from <none> to bf353eb308de425ea6e214b9c6defab3 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "bf353eb308de425ea6e214b9c6defab3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf353eb308de425ea6e214b9c6defab3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36547 } health_report { overall_health: HEALTHY } } }
I20240503 02:00:04.458568 13926 catalog_manager.cc:5484] T 939505a3617342259e1789eb0dd9bb27 P bf353eb308de425ea6e214b9c6defab3 reported cstate change: term changed from 0 to 1, leader changed from <none> to bf353eb308de425ea6e214b9c6defab3 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "bf353eb308de425ea6e214b9c6defab3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf353eb308de425ea6e214b9c6defab3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36547 } health_report { overall_health: HEALTHY } } }
I20240503 02:00:04.661056 13926 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48154:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:00:04.667618 14048 tablet_service.cc:1467] Processing CreateTablet for tablet 56be7b02c43a431b8748cf9ee169fa33 (DEFAULT_TABLE table=rebalance_test_table_2 [id=a6374d70a5db4b98b3c18735981c8c76]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:04.667618 14049 tablet_service.cc:1467] Processing CreateTablet for tablet 47cc87614328411d9a2efba497aa7c81 (DEFAULT_TABLE table=rebalance_test_table_2 [id=a6374d70a5db4b98b3c18735981c8c76]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:04.667680 14047 tablet_service.cc:1467] Processing CreateTablet for tablet 61d98eabfd344a9b8f8895b865848e20 (DEFAULT_TABLE table=rebalance_test_table_2 [id=a6374d70a5db4b98b3c18735981c8c76]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:04.668150 14048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56be7b02c43a431b8748cf9ee169fa33. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:04.668615 14049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47cc87614328411d9a2efba497aa7c81. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:04.668812 14047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61d98eabfd344a9b8f8895b865848e20. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:04.670938 14538 tablet_bootstrap.cc:492] T 56be7b02c43a431b8748cf9ee169fa33 P 4cd4b11bee2b4878bed3e10d2d76f368: Bootstrap starting.
I20240503 02:00:04.671866 14538 tablet_bootstrap.cc:654] T 56be7b02c43a431b8748cf9ee169fa33 P 4cd4b11bee2b4878bed3e10d2d76f368: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:04.673092 14538 tablet_bootstrap.cc:492] T 56be7b02c43a431b8748cf9ee169fa33 P 4cd4b11bee2b4878bed3e10d2d76f368: No bootstrap required, opened a new log
I20240503 02:00:04.673177 14538 ts_tablet_manager.cc:1367] T 56be7b02c43a431b8748cf9ee169fa33 P 4cd4b11bee2b4878bed3e10d2d76f368: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:04.673698 14538 raft_consensus.cc:348] T 56be7b02c43a431b8748cf9ee169fa33 P 4cd4b11bee2b4878bed3e10d2d76f368 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cd4b11bee2b4878bed3e10d2d76f368" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37199 } }
I20240503 02:00:04.673817 14538 raft_consensus.cc:374] T 56be7b02c43a431b8748cf9ee169fa33 P 4cd4b11bee2b4878bed3e10d2d76f368 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:04.673859 14538 raft_consensus.cc:729] T 56be7b02c43a431b8748cf9ee169fa33 P 4cd4b11bee2b4878bed3e10d2d76f368 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cd4b11bee2b4878bed3e10d2d76f368, State: Initialized, Role: FOLLOWER
I20240503 02:00:04.674001 14538 consensus_queue.cc:260] T 56be7b02c43a431b8748cf9ee169fa33 P 4cd4b11bee2b4878bed3e10d2d76f368 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cd4b11bee2b4878bed3e10d2d76f368" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37199 } }
I20240503 02:00:04.674085 14538 raft_consensus.cc:388] T 56be7b02c43a431b8748cf9ee169fa33 P 4cd4b11bee2b4878bed3e10d2d76f368 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:00:04.674130 14538 raft_consensus.cc:482] T 56be7b02c43a431b8748cf9ee169fa33 P 4cd4b11bee2b4878bed3e10d2d76f368 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:00:04.674183 14538 raft_consensus.cc:3037] T 56be7b02c43a431b8748cf9ee169fa33 P 4cd4b11bee2b4878bed3e10d2d76f368 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:04.674909 14538 raft_consensus.cc:504] T 56be7b02c43a431b8748cf9ee169fa33 P 4cd4b11bee2b4878bed3e10d2d76f368 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cd4b11bee2b4878bed3e10d2d76f368" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37199 } }
I20240503 02:00:04.675036 14538 leader_election.cc:302] T 56be7b02c43a431b8748cf9ee169fa33 P 4cd4b11bee2b4878bed3e10d2d76f368 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4cd4b11bee2b4878bed3e10d2d76f368; no voters:
I20240503 02:00:04.675132 14538 leader_election.cc:288] T 56be7b02c43a431b8748cf9ee169fa33 P 4cd4b11bee2b4878bed3e10d2d76f368 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:00:04.675244 14540 raft_consensus.cc:2781] T 56be7b02c43a431b8748cf9ee169fa33 P 4cd4b11bee2b4878bed3e10d2d76f368 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:04.675410 14540 raft_consensus.cc:686] T 56be7b02c43a431b8748cf9ee169fa33 P 4cd4b11bee2b4878bed3e10d2d76f368 [term 1 LEADER]: Becoming Leader. State: Replica: 4cd4b11bee2b4878bed3e10d2d76f368, State: Running, Role: LEADER
I20240503 02:00:04.675393 14538 ts_tablet_manager.cc:1398] T 56be7b02c43a431b8748cf9ee169fa33 P 4cd4b11bee2b4878bed3e10d2d76f368: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:00:04.675616 14538 tablet_bootstrap.cc:492] T 61d98eabfd344a9b8f8895b865848e20 P 4cd4b11bee2b4878bed3e10d2d76f368: Bootstrap starting.
I20240503 02:00:04.675603 14540 consensus_queue.cc:237] T 56be7b02c43a431b8748cf9ee169fa33 P 4cd4b11bee2b4878bed3e10d2d76f368 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cd4b11bee2b4878bed3e10d2d76f368" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37199 } }
I20240503 02:00:04.676553 14538 tablet_bootstrap.cc:654] T 61d98eabfd344a9b8f8895b865848e20 P 4cd4b11bee2b4878bed3e10d2d76f368: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:04.677507 13926 catalog_manager.cc:5484] T 56be7b02c43a431b8748cf9ee169fa33 P 4cd4b11bee2b4878bed3e10d2d76f368 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4cd4b11bee2b4878bed3e10d2d76f368 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "4cd4b11bee2b4878bed3e10d2d76f368" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cd4b11bee2b4878bed3e10d2d76f368" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37199 } health_report { overall_health: HEALTHY } } }
I20240503 02:00:04.677946 14538 tablet_bootstrap.cc:492] T 61d98eabfd344a9b8f8895b865848e20 P 4cd4b11bee2b4878bed3e10d2d76f368: No bootstrap required, opened a new log
I20240503 02:00:04.678086 14538 ts_tablet_manager.cc:1367] T 61d98eabfd344a9b8f8895b865848e20 P 4cd4b11bee2b4878bed3e10d2d76f368: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:04.678623 14538 raft_consensus.cc:348] T 61d98eabfd344a9b8f8895b865848e20 P 4cd4b11bee2b4878bed3e10d2d76f368 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cd4b11bee2b4878bed3e10d2d76f368" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37199 } }
I20240503 02:00:04.678747 14538 raft_consensus.cc:374] T 61d98eabfd344a9b8f8895b865848e20 P 4cd4b11bee2b4878bed3e10d2d76f368 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:04.678791 14538 raft_consensus.cc:729] T 61d98eabfd344a9b8f8895b865848e20 P 4cd4b11bee2b4878bed3e10d2d76f368 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cd4b11bee2b4878bed3e10d2d76f368, State: Initialized, Role: FOLLOWER
I20240503 02:00:04.678939 14538 consensus_queue.cc:260] T 61d98eabfd344a9b8f8895b865848e20 P 4cd4b11bee2b4878bed3e10d2d76f368 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cd4b11bee2b4878bed3e10d2d76f368" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37199 } }
I20240503 02:00:04.679018 14538 raft_consensus.cc:388] T 61d98eabfd344a9b8f8895b865848e20 P 4cd4b11bee2b4878bed3e10d2d76f368 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:00:04.679052 14538 raft_consensus.cc:482] T 61d98eabfd344a9b8f8895b865848e20 P 4cd4b11bee2b4878bed3e10d2d76f368 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:00:04.679095 14538 raft_consensus.cc:3037] T 61d98eabfd344a9b8f8895b865848e20 P 4cd4b11bee2b4878bed3e10d2d76f368 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:04.680001 14538 raft_consensus.cc:504] T 61d98eabfd344a9b8f8895b865848e20 P 4cd4b11bee2b4878bed3e10d2d76f368 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cd4b11bee2b4878bed3e10d2d76f368" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37199 } }
I20240503 02:00:04.680126 14538 leader_election.cc:302] T 61d98eabfd344a9b8f8895b865848e20 P 4cd4b11bee2b4878bed3e10d2d76f368 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4cd4b11bee2b4878bed3e10d2d76f368; no voters:
I20240503 02:00:04.680227 14538 leader_election.cc:288] T 61d98eabfd344a9b8f8895b865848e20 P 4cd4b11bee2b4878bed3e10d2d76f368 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:00:04.680315 14540 raft_consensus.cc:2781] T 61d98eabfd344a9b8f8895b865848e20 P 4cd4b11bee2b4878bed3e10d2d76f368 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:04.680467 14540 raft_consensus.cc:686] T 61d98eabfd344a9b8f8895b865848e20 P 4cd4b11bee2b4878bed3e10d2d76f368 [term 1 LEADER]: Becoming Leader. State: Replica: 4cd4b11bee2b4878bed3e10d2d76f368, State: Running, Role: LEADER
I20240503 02:00:04.680511 14538 ts_tablet_manager.cc:1398] T 61d98eabfd344a9b8f8895b865848e20 P 4cd4b11bee2b4878bed3e10d2d76f368: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 02:00:04.680580 14540 consensus_queue.cc:237] T 61d98eabfd344a9b8f8895b865848e20 P 4cd4b11bee2b4878bed3e10d2d76f368 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cd4b11bee2b4878bed3e10d2d76f368" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37199 } }
I20240503 02:00:04.680709 14538 tablet_bootstrap.cc:492] T 47cc87614328411d9a2efba497aa7c81 P 4cd4b11bee2b4878bed3e10d2d76f368: Bootstrap starting.
I20240503 02:00:04.682101 14538 tablet_bootstrap.cc:654] T 47cc87614328411d9a2efba497aa7c81 P 4cd4b11bee2b4878bed3e10d2d76f368: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:04.682072 13927 catalog_manager.cc:5484] T 61d98eabfd344a9b8f8895b865848e20 P 4cd4b11bee2b4878bed3e10d2d76f368 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4cd4b11bee2b4878bed3e10d2d76f368 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "4cd4b11bee2b4878bed3e10d2d76f368" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cd4b11bee2b4878bed3e10d2d76f368" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37199 } health_report { overall_health: HEALTHY } } }
I20240503 02:00:04.683378 14538 tablet_bootstrap.cc:492] T 47cc87614328411d9a2efba497aa7c81 P 4cd4b11bee2b4878bed3e10d2d76f368: No bootstrap required, opened a new log
I20240503 02:00:04.683468 14538 ts_tablet_manager.cc:1367] T 47cc87614328411d9a2efba497aa7c81 P 4cd4b11bee2b4878bed3e10d2d76f368: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:04.683890 14538 raft_consensus.cc:348] T 47cc87614328411d9a2efba497aa7c81 P 4cd4b11bee2b4878bed3e10d2d76f368 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cd4b11bee2b4878bed3e10d2d76f368" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37199 } }
I20240503 02:00:04.683977 14538 raft_consensus.cc:374] T 47cc87614328411d9a2efba497aa7c81 P 4cd4b11bee2b4878bed3e10d2d76f368 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:04.684012 14538 raft_consensus.cc:729] T 47cc87614328411d9a2efba497aa7c81 P 4cd4b11bee2b4878bed3e10d2d76f368 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cd4b11bee2b4878bed3e10d2d76f368, State: Initialized, Role: FOLLOWER
I20240503 02:00:04.684162 14538 consensus_queue.cc:260] T 47cc87614328411d9a2efba497aa7c81 P 4cd4b11bee2b4878bed3e10d2d76f368 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cd4b11bee2b4878bed3e10d2d76f368" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37199 } }
I20240503 02:00:04.684234 14538 raft_consensus.cc:388] T 47cc87614328411d9a2efba497aa7c81 P 4cd4b11bee2b4878bed3e10d2d76f368 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:00:04.684280 14538 raft_consensus.cc:482] T 47cc87614328411d9a2efba497aa7c81 P 4cd4b11bee2b4878bed3e10d2d76f368 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:00:04.684330 14538 raft_consensus.cc:3037] T 47cc87614328411d9a2efba497aa7c81 P 4cd4b11bee2b4878bed3e10d2d76f368 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:04.685070 14538 raft_consensus.cc:504] T 47cc87614328411d9a2efba497aa7c81 P 4cd4b11bee2b4878bed3e10d2d76f368 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cd4b11bee2b4878bed3e10d2d76f368" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37199 } }
I20240503 02:00:04.685196 14538 leader_election.cc:302] T 47cc87614328411d9a2efba497aa7c81 P 4cd4b11bee2b4878bed3e10d2d76f368 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4cd4b11bee2b4878bed3e10d2d76f368; no voters:
I20240503 02:00:04.685293 14538 leader_election.cc:288] T 47cc87614328411d9a2efba497aa7c81 P 4cd4b11bee2b4878bed3e10d2d76f368 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:00:04.685355 14540 raft_consensus.cc:2781] T 47cc87614328411d9a2efba497aa7c81 P 4cd4b11bee2b4878bed3e10d2d76f368 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:04.685477 14540 raft_consensus.cc:686] T 47cc87614328411d9a2efba497aa7c81 P 4cd4b11bee2b4878bed3e10d2d76f368 [term 1 LEADER]: Becoming Leader. State: Replica: 4cd4b11bee2b4878bed3e10d2d76f368, State: Running, Role: LEADER
I20240503 02:00:04.685549 14538 ts_tablet_manager.cc:1398] T 47cc87614328411d9a2efba497aa7c81 P 4cd4b11bee2b4878bed3e10d2d76f368: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:04.685585 14540 consensus_queue.cc:237] T 47cc87614328411d9a2efba497aa7c81 P 4cd4b11bee2b4878bed3e10d2d76f368 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cd4b11bee2b4878bed3e10d2d76f368" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37199 } }
I20240503 02:00:04.686995 13927 catalog_manager.cc:5484] T 47cc87614328411d9a2efba497aa7c81 P 4cd4b11bee2b4878bed3e10d2d76f368 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4cd4b11bee2b4878bed3e10d2d76f368 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "4cd4b11bee2b4878bed3e10d2d76f368" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cd4b11bee2b4878bed3e10d2d76f368" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37199 } health_report { overall_health: HEALTHY } } }
I20240503 02:00:04.879925 13927 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48154:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:00:04.886488 14180 tablet_service.cc:1467] Processing CreateTablet for tablet 13842711e9fd4227ba148b04eecdd679 (DEFAULT_TABLE table=rebalance_test_table_3 [id=6bde0a0a52c041ab96301658f5350a71]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:04.886813 14180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13842711e9fd4227ba148b04eecdd679. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:04.886674 14179 tablet_service.cc:1467] Processing CreateTablet for tablet 56e174b9915546cfbe179435fc8b7685 (DEFAULT_TABLE table=rebalance_test_table_3 [id=6bde0a0a52c041ab96301658f5350a71]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:04.886978 14179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56e174b9915546cfbe179435fc8b7685. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:04.886674 14178 tablet_service.cc:1467] Processing CreateTablet for tablet 8a92724f93844232b4c511bf70960e02 (DEFAULT_TABLE table=rebalance_test_table_3 [id=6bde0a0a52c041ab96301658f5350a71]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:04.887351 14178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a92724f93844232b4c511bf70960e02. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:04.889164 14568 tablet_bootstrap.cc:492] T 13842711e9fd4227ba148b04eecdd679 P bf353eb308de425ea6e214b9c6defab3: Bootstrap starting.
I20240503 02:00:04.890163 14568 tablet_bootstrap.cc:654] T 13842711e9fd4227ba148b04eecdd679 P bf353eb308de425ea6e214b9c6defab3: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:04.891269 14568 tablet_bootstrap.cc:492] T 13842711e9fd4227ba148b04eecdd679 P bf353eb308de425ea6e214b9c6defab3: No bootstrap required, opened a new log
I20240503 02:00:04.891345 14568 ts_tablet_manager.cc:1367] T 13842711e9fd4227ba148b04eecdd679 P bf353eb308de425ea6e214b9c6defab3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:04.891710 14568 raft_consensus.cc:348] T 13842711e9fd4227ba148b04eecdd679 P bf353eb308de425ea6e214b9c6defab3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf353eb308de425ea6e214b9c6defab3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36547 } }
I20240503 02:00:04.891803 14568 raft_consensus.cc:374] T 13842711e9fd4227ba148b04eecdd679 P bf353eb308de425ea6e214b9c6defab3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:04.891842 14568 raft_consensus.cc:729] T 13842711e9fd4227ba148b04eecdd679 P bf353eb308de425ea6e214b9c6defab3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf353eb308de425ea6e214b9c6defab3, State: Initialized, Role: FOLLOWER
I20240503 02:00:04.891950 14568 consensus_queue.cc:260] T 13842711e9fd4227ba148b04eecdd679 P bf353eb308de425ea6e214b9c6defab3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf353eb308de425ea6e214b9c6defab3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36547 } }
I20240503 02:00:04.892035 14568 raft_consensus.cc:388] T 13842711e9fd4227ba148b04eecdd679 P bf353eb308de425ea6e214b9c6defab3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:00:04.892063 14568 raft_consensus.cc:482] T 13842711e9fd4227ba148b04eecdd679 P bf353eb308de425ea6e214b9c6defab3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:00:04.892100 14568 raft_consensus.cc:3037] T 13842711e9fd4227ba148b04eecdd679 P bf353eb308de425ea6e214b9c6defab3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:04.893011 14568 raft_consensus.cc:504] T 13842711e9fd4227ba148b04eecdd679 P bf353eb308de425ea6e214b9c6defab3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf353eb308de425ea6e214b9c6defab3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36547 } }
I20240503 02:00:04.893126 14568 leader_election.cc:302] T 13842711e9fd4227ba148b04eecdd679 P bf353eb308de425ea6e214b9c6defab3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bf353eb308de425ea6e214b9c6defab3; no voters:
I20240503 02:00:04.893218 14568 leader_election.cc:288] T 13842711e9fd4227ba148b04eecdd679 P bf353eb308de425ea6e214b9c6defab3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:00:04.893294 14570 raft_consensus.cc:2781] T 13842711e9fd4227ba148b04eecdd679 P bf353eb308de425ea6e214b9c6defab3 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:04.893388 14570 raft_consensus.cc:686] T 13842711e9fd4227ba148b04eecdd679 P bf353eb308de425ea6e214b9c6defab3 [term 1 LEADER]: Becoming Leader. State: Replica: bf353eb308de425ea6e214b9c6defab3, State: Running, Role: LEADER
I20240503 02:00:04.893515 14570 consensus_queue.cc:237] T 13842711e9fd4227ba148b04eecdd679 P bf353eb308de425ea6e214b9c6defab3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf353eb308de425ea6e214b9c6defab3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36547 } }
I20240503 02:00:04.893702 14568 ts_tablet_manager.cc:1398] T 13842711e9fd4227ba148b04eecdd679 P bf353eb308de425ea6e214b9c6defab3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:04.893882 14568 tablet_bootstrap.cc:492] T 56e174b9915546cfbe179435fc8b7685 P bf353eb308de425ea6e214b9c6defab3: Bootstrap starting.
I20240503 02:00:04.894730 14568 tablet_bootstrap.cc:654] T 56e174b9915546cfbe179435fc8b7685 P bf353eb308de425ea6e214b9c6defab3: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:04.895321 13927 catalog_manager.cc:5484] T 13842711e9fd4227ba148b04eecdd679 P bf353eb308de425ea6e214b9c6defab3 reported cstate change: term changed from 0 to 1, leader changed from <none> to bf353eb308de425ea6e214b9c6defab3 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "bf353eb308de425ea6e214b9c6defab3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf353eb308de425ea6e214b9c6defab3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36547 } health_report { overall_health: HEALTHY } } }
I20240503 02:00:04.896073 14568 tablet_bootstrap.cc:492] T 56e174b9915546cfbe179435fc8b7685 P bf353eb308de425ea6e214b9c6defab3: No bootstrap required, opened a new log
I20240503 02:00:04.896173 14568 ts_tablet_manager.cc:1367] T 56e174b9915546cfbe179435fc8b7685 P bf353eb308de425ea6e214b9c6defab3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:04.896591 14568 raft_consensus.cc:348] T 56e174b9915546cfbe179435fc8b7685 P bf353eb308de425ea6e214b9c6defab3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf353eb308de425ea6e214b9c6defab3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36547 } }
I20240503 02:00:04.896718 14568 raft_consensus.cc:374] T 56e174b9915546cfbe179435fc8b7685 P bf353eb308de425ea6e214b9c6defab3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:04.896775 14568 raft_consensus.cc:729] T 56e174b9915546cfbe179435fc8b7685 P bf353eb308de425ea6e214b9c6defab3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf353eb308de425ea6e214b9c6defab3, State: Initialized, Role: FOLLOWER
I20240503 02:00:04.896905 14568 consensus_queue.cc:260] T 56e174b9915546cfbe179435fc8b7685 P bf353eb308de425ea6e214b9c6defab3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf353eb308de425ea6e214b9c6defab3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36547 } }
I20240503 02:00:04.896991 14568 raft_consensus.cc:388] T 56e174b9915546cfbe179435fc8b7685 P bf353eb308de425ea6e214b9c6defab3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:00:04.897027 14568 raft_consensus.cc:482] T 56e174b9915546cfbe179435fc8b7685 P bf353eb308de425ea6e214b9c6defab3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:00:04.897063 14568 raft_consensus.cc:3037] T 56e174b9915546cfbe179435fc8b7685 P bf353eb308de425ea6e214b9c6defab3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:04.897845 14568 raft_consensus.cc:504] T 56e174b9915546cfbe179435fc8b7685 P bf353eb308de425ea6e214b9c6defab3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf353eb308de425ea6e214b9c6defab3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36547 } }
I20240503 02:00:04.897944 14568 leader_election.cc:302] T 56e174b9915546cfbe179435fc8b7685 P bf353eb308de425ea6e214b9c6defab3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bf353eb308de425ea6e214b9c6defab3; no voters:
I20240503 02:00:04.898007 14568 leader_election.cc:288] T 56e174b9915546cfbe179435fc8b7685 P bf353eb308de425ea6e214b9c6defab3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:00:04.898063 14574 raft_consensus.cc:2781] T 56e174b9915546cfbe179435fc8b7685 P bf353eb308de425ea6e214b9c6defab3 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:04.898133 14574 raft_consensus.cc:686] T 56e174b9915546cfbe179435fc8b7685 P bf353eb308de425ea6e214b9c6defab3 [term 1 LEADER]: Becoming Leader. State: Replica: bf353eb308de425ea6e214b9c6defab3, State: Running, Role: LEADER
I20240503 02:00:04.898257 14568 ts_tablet_manager.cc:1398] T 56e174b9915546cfbe179435fc8b7685 P bf353eb308de425ea6e214b9c6defab3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:04.898233 14574 consensus_queue.cc:237] T 56e174b9915546cfbe179435fc8b7685 P bf353eb308de425ea6e214b9c6defab3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf353eb308de425ea6e214b9c6defab3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36547 } }
I20240503 02:00:04.898401 14568 tablet_bootstrap.cc:492] T 8a92724f93844232b4c511bf70960e02 P bf353eb308de425ea6e214b9c6defab3: Bootstrap starting.
I20240503 02:00:04.899312 14568 tablet_bootstrap.cc:654] T 8a92724f93844232b4c511bf70960e02 P bf353eb308de425ea6e214b9c6defab3: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:04.900588 14568 tablet_bootstrap.cc:492] T 8a92724f93844232b4c511bf70960e02 P bf353eb308de425ea6e214b9c6defab3: No bootstrap required, opened a new log
I20240503 02:00:04.900715 14568 ts_tablet_manager.cc:1367] T 8a92724f93844232b4c511bf70960e02 P bf353eb308de425ea6e214b9c6defab3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:04.900789 13927 catalog_manager.cc:5484] T 56e174b9915546cfbe179435fc8b7685 P bf353eb308de425ea6e214b9c6defab3 reported cstate change: term changed from 0 to 1, leader changed from <none> to bf353eb308de425ea6e214b9c6defab3 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "bf353eb308de425ea6e214b9c6defab3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf353eb308de425ea6e214b9c6defab3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36547 } health_report { overall_health: HEALTHY } } }
I20240503 02:00:04.901168 14568 raft_consensus.cc:348] T 8a92724f93844232b4c511bf70960e02 P bf353eb308de425ea6e214b9c6defab3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf353eb308de425ea6e214b9c6defab3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36547 } }
I20240503 02:00:04.901266 14568 raft_consensus.cc:374] T 8a92724f93844232b4c511bf70960e02 P bf353eb308de425ea6e214b9c6defab3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:04.901319 14568 raft_consensus.cc:729] T 8a92724f93844232b4c511bf70960e02 P bf353eb308de425ea6e214b9c6defab3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf353eb308de425ea6e214b9c6defab3, State: Initialized, Role: FOLLOWER
I20240503 02:00:04.901437 14568 consensus_queue.cc:260] T 8a92724f93844232b4c511bf70960e02 P bf353eb308de425ea6e214b9c6defab3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf353eb308de425ea6e214b9c6defab3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36547 } }
I20240503 02:00:04.901510 14568 raft_consensus.cc:388] T 8a92724f93844232b4c511bf70960e02 P bf353eb308de425ea6e214b9c6defab3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:00:04.901531 14568 raft_consensus.cc:482] T 8a92724f93844232b4c511bf70960e02 P bf353eb308de425ea6e214b9c6defab3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:00:04.901576 14568 raft_consensus.cc:3037] T 8a92724f93844232b4c511bf70960e02 P bf353eb308de425ea6e214b9c6defab3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:04.902369 14568 raft_consensus.cc:504] T 8a92724f93844232b4c511bf70960e02 P bf353eb308de425ea6e214b9c6defab3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf353eb308de425ea6e214b9c6defab3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36547 } }
I20240503 02:00:04.902493 14568 leader_election.cc:302] T 8a92724f93844232b4c511bf70960e02 P bf353eb308de425ea6e214b9c6defab3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bf353eb308de425ea6e214b9c6defab3; no voters:
I20240503 02:00:04.902580 14568 leader_election.cc:288] T 8a92724f93844232b4c511bf70960e02 P bf353eb308de425ea6e214b9c6defab3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:00:04.902772 14571 raft_consensus.cc:2781] T 8a92724f93844232b4c511bf70960e02 P bf353eb308de425ea6e214b9c6defab3 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:04.902850 14568 ts_tablet_manager.cc:1398] T 8a92724f93844232b4c511bf70960e02 P bf353eb308de425ea6e214b9c6defab3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:04.902948 14571 raft_consensus.cc:686] T 8a92724f93844232b4c511bf70960e02 P bf353eb308de425ea6e214b9c6defab3 [term 1 LEADER]: Becoming Leader. State: Replica: bf353eb308de425ea6e214b9c6defab3, State: Running, Role: LEADER
I20240503 02:00:04.903132 14571 consensus_queue.cc:237] T 8a92724f93844232b4c511bf70960e02 P bf353eb308de425ea6e214b9c6defab3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf353eb308de425ea6e214b9c6defab3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36547 } }
I20240503 02:00:04.904693 13927 catalog_manager.cc:5484] T 8a92724f93844232b4c511bf70960e02 P bf353eb308de425ea6e214b9c6defab3 reported cstate change: term changed from 0 to 1, leader changed from <none> to bf353eb308de425ea6e214b9c6defab3 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "bf353eb308de425ea6e214b9c6defab3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf353eb308de425ea6e214b9c6defab3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36547 } health_report { overall_health: HEALTHY } } }
I20240503 02:00:05.097122 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task42rE7p/build/debug/bin/kudu
/tmp/dist-test-task42rE7p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.131:45811
--local_ip_for_outbound_sockets=127.0.99.131
--tserver_master_addrs=127.0.99.190:44351
--webserver_port=36555
--webserver_interface=127.0.99.131
--builtin_ntp_servers=127.0.99.148:40197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 02:00:05.210670 14644 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:00:05.210928 14644 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:00:05.210973 14644 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:00:05.213946 14644 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:00:05.214005 14644 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:00:05.214103 14644 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.131
I20240503 02:00:05.217736 14644 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:40197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.131:45811
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.99.131
--webserver_port=36555
--tserver_master_addrs=127.0.99.190:44351
--never_fsync=true
--heap_profile_path=/tmp/kudu.14644
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.131
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 02:00:05.218638 14644 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:00:05.219749 14644 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:00:05.228768 14650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:05.229602 14652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:05.230933 14649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:00:05.232033 14644 server_base.cc:1034] running on GCE node
I20240503 02:00:05.232466 14644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:00:05.233036 14644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:00:05.234205 14644 hybrid_clock.cc:648] HybridClock initialized: now 1714701605234181 us; error 36 us; skew 500 ppm
I20240503 02:00:05.236166 14644 webserver.cc:460] Webserver started at http://127.0.99.131:36555/ using document root <none> and password file <none>
I20240503 02:00:05.236943 14644 fs_manager.cc:359] Metadata directory not provided
I20240503 02:00:05.237049 14644 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:00:05.240468 14644 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240503 02:00:05.242503 14658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:05.243468 14644 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20240503 02:00:05.243571 14644 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "8796d8bfc685436c8889ffa947644a58"
format_stamp: "Formatted at 2024-05-03 02:00:00 on dist-test-slave-dtz3"
I20240503 02:00:05.243944 14644 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:00:05.254995 14644 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:00:05.255515 14644 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:00:05.255661 14644 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:00:05.256129 14644 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:00:05.257001 14644 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:00:05.257050 14644 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:05.257107 14644 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:00:05.257134 14644 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:05.265216 14644 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.131:45811
I20240503 02:00:05.265278 14771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.131:45811 every 8 connection(s)
I20240503 02:00:05.266247 14644 server_base.cc:1166] Dumped server information to /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 02:00:05.273038 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task42rE7p/build/debug/bin/kudu as pid 14644
I20240503 02:00:05.273527 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task42rE7p/build/debug/bin/kudu
/tmp/dist-test-task42rE7p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.132:42443
--local_ip_for_outbound_sockets=127.0.99.132
--tserver_master_addrs=127.0.99.190:44351
--webserver_port=42465
--webserver_interface=127.0.99.132
--builtin_ntp_servers=127.0.99.148:40197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:00:05.274735 14772 heartbeater.cc:347] Connected to a master server at 127.0.99.190:44351
I20240503 02:00:05.274945 14772 heartbeater.cc:464] Registering TS with master...
I20240503 02:00:05.275472 14772 heartbeater.cc:511] Master 127.0.99.190:44351 requested a full tablet report, sending...
I20240503 02:00:05.276546 13927 ts_manager.cc:194] Re-registered known tserver with Master: 8796d8bfc685436c8889ffa947644a58 (127.0.99.131:45811)
I20240503 02:00:05.277108 13927 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.131:52935
W20240503 02:00:05.380105 14775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:00:05.380308 14775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:00:05.380354 14775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:00:05.383401 14775 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:00:05.383464 14775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:00:05.383543 14775 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.132
I20240503 02:00:05.387013 14775 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:40197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.132:42443
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.99.132
--webserver_port=42465
--tserver_master_addrs=127.0.99.190:44351
--never_fsync=true
--heap_profile_path=/tmp/kudu.14775
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.132
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 02:00:05.387838 14775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:00:05.388914 14775 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:00:05.397814 14780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:05.398593 14781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:05.399864 14783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:00:05.400890 14775 server_base.cc:1034] running on GCE node
I20240503 02:00:05.401283 14775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:00:05.401841 14775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:00:05.403007 14775 hybrid_clock.cc:648] HybridClock initialized: now 1714701605402995 us; error 25 us; skew 500 ppm
I20240503 02:00:05.404989 14775 webserver.cc:460] Webserver started at http://127.0.99.132:42465/ using document root <none> and password file <none>
I20240503 02:00:05.405531 14775 fs_manager.cc:359] Metadata directory not provided
I20240503 02:00:05.405606 14775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:00:05.409009 14775 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240503 02:00:05.410871 14789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:05.411788 14775 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:05.411881 14775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "eccecde0730943b8a29023ca7f42d4ff"
format_stamp: "Formatted at 2024-05-03 02:00:00 on dist-test-slave-dtz3"
I20240503 02:00:05.412317 14775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:00:05.446403 14775 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:00:05.446998 14775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:00:05.447186 14775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:00:05.447742 14775 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:00:05.448719 14775 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:00:05.448781 14775 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:05.448838 14775 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:00:05.448876 14775 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:05.457147 14775 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.132:42443
I20240503 02:00:05.457180 14902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.132:42443 every 8 connection(s)
I20240503 02:00:05.458175 14775 server_base.cc:1166] Dumped server information to /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 02:00:05.459579 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task42rE7p/build/debug/bin/kudu as pid 14775
I20240503 02:00:05.467041 14903 heartbeater.cc:347] Connected to a master server at 127.0.99.190:44351
I20240503 02:00:05.467315 14903 heartbeater.cc:464] Registering TS with master...
I20240503 02:00:05.467820 14903 heartbeater.cc:511] Master 127.0.99.190:44351 requested a full tablet report, sending...
I20240503 02:00:05.468957 13927 ts_manager.cc:194] Re-registered known tserver with Master: eccecde0730943b8a29023ca7f42d4ff (127.0.99.132:42443)
I20240503 02:00:05.469712 13927 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.132:45271
I20240503 02:00:05.649750 14049 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:00:05.651118 14178 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:00:05.652321 14706 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:05.652669 14837 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:05.767849 14706 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:05.769536 14837 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:05.769702 14178 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:00:05.777029 14049 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:00:05.860221 14049 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:00:05.860939 14706 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:05.860985 14178 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:00:05.866269 14837 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
66a949c20cbb4391876ef040890466e1 | 127.0.99.190:44351 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.99.148:40197 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
4cd4b11bee2b4878bed3e10d2d76f368 | 127.0.99.129:37199 | HEALTHY | <none> | 6 | 0
8796d8bfc685436c8889ffa947644a58 | 127.0.99.131:45811 | HEALTHY | <none> | 0 | 0
bf353eb308de425ea6e214b9c6defab3 | 127.0.99.130:36547 | HEALTHY | <none> | 7 | 0
eccecde0730943b8a29023ca7f42d4ff | 127.0.99.132:42443 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 4
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 4 server(s) checked
local_ip_for_outbound_sockets | 127.0.99.129 | experimental | 127.0.99.129:37199
local_ip_for_outbound_sockets | 127.0.99.130 | experimental | 127.0.99.130:36547
local_ip_for_outbound_sockets | 127.0.99.131 | experimental | 127.0.99.131:45811
local_ip_for_outbound_sockets | 127.0.99.132 | experimental | 127.0.99.132:42443
never_fsync | true | unsafe,advanced | all 4 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 4 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 4 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked
server_dump_info_format | pb | hidden | all 4 server(s) checked
server_dump_info_path | /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.99.129:37199
server_dump_info_path | /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.0.99.130:36547
server_dump_info_path | /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.99.131:45811
server_dump_info_path | /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.0.99.132:42443
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.99.148:40197 | all 4 server(s) checked
time_source | builtin | all 4 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 5 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 0
Median | 6
Third Quartile | 7
Maximum | 7
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 4
Tables | 5
Tablets | 13
Replicas | 13
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240503 02:00:05.889961 398 log_verifier.cc:126] Checking tablet 13842711e9fd4227ba148b04eecdd679
I20240503 02:00:05.897637 398 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 13842711e9fd4227ba148b04eecdd679
I20240503 02:00:05.897702 398 log_verifier.cc:126] Checking tablet 3a18f1f16b5a42578a713f0c133c8ef7
I20240503 02:00:05.904953 398 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 3a18f1f16b5a42578a713f0c133c8ef7
I20240503 02:00:05.905009 398 log_verifier.cc:126] Checking tablet 47cc87614328411d9a2efba497aa7c81
I20240503 02:00:05.913134 398 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 47cc87614328411d9a2efba497aa7c81
I20240503 02:00:05.913197 398 log_verifier.cc:126] Checking tablet 56be7b02c43a431b8748cf9ee169fa33
I20240503 02:00:05.920581 398 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 56be7b02c43a431b8748cf9ee169fa33
I20240503 02:00:05.920663 398 log_verifier.cc:126] Checking tablet 56e174b9915546cfbe179435fc8b7685
I20240503 02:00:05.928169 398 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 56e174b9915546cfbe179435fc8b7685
I20240503 02:00:05.928236 398 log_verifier.cc:126] Checking tablet 61d98eabfd344a9b8f8895b865848e20
I20240503 02:00:05.936125 398 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 61d98eabfd344a9b8f8895b865848e20
I20240503 02:00:05.936188 398 log_verifier.cc:126] Checking tablet 67434687758243feab59f655a925bce9
I20240503 02:00:05.943487 398 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 67434687758243feab59f655a925bce9
I20240503 02:00:05.943547 398 log_verifier.cc:126] Checking tablet 6b2be0c3787d47788616c682f3492a52
I20240503 02:00:05.950843 398 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 6b2be0c3787d47788616c682f3492a52
I20240503 02:00:05.950904 398 log_verifier.cc:126] Checking tablet 8a92724f93844232b4c511bf70960e02
I20240503 02:00:05.958082 398 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 8a92724f93844232b4c511bf70960e02
I20240503 02:00:05.958148 398 log_verifier.cc:126] Checking tablet 939505a3617342259e1789eb0dd9bb27
I20240503 02:00:05.965508 398 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 939505a3617342259e1789eb0dd9bb27
I20240503 02:00:05.965564 398 log_verifier.cc:126] Checking tablet 9a3118fc5d8740b68dbcfed1056069b2
I20240503 02:00:05.972735 398 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 9a3118fc5d8740b68dbcfed1056069b2
I20240503 02:00:05.972790 398 log_verifier.cc:126] Checking tablet a09a07623e334e3186aa1706b8b24a5d
I20240503 02:00:05.980134 398 log_verifier.cc:177] Verified matching terms for 1 ops in tablet a09a07623e334e3186aa1706b8b24a5d
I20240503 02:00:05.980199 398 log_verifier.cc:126] Checking tablet b215d743523b43e295662f1e351c43c5
I20240503 02:00:05.987469 398 log_verifier.cc:177] Verified matching terms for 1 ops in tablet b215d743523b43e295662f1e351c43c5
I20240503 02:00:05.987964 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task42rE7p/build/debug/bin/kudu with pid 13987
I20240503 02:00:05.994946 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task42rE7p/build/debug/bin/kudu with pid 14118
I20240503 02:00:06.000586 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task42rE7p/build/debug/bin/kudu with pid 14644
I20240503 02:00:06.006211 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task42rE7p/build/debug/bin/kudu with pid 14775
I20240503 02:00:06.011965 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task42rE7p/build/debug/bin/kudu with pid 13896
2024-05-03T02:00:06Z chronyd exiting
[ OK ] RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove/5 (6628 ms)
[----------] 1 test from RebalancerAndSingleReplicaTablets (6628 ms total)
[----------] 2 tests from LocationAwareRebalancingParamTest
[ RUN ] LocationAwareRebalancingParamTest.Rebalance/idx7_rf2_t3_l_1_2_1
I20240503 02:00:06.045632 398 test_util.cc:268] Using random seed: 337627067
I20240503 02:00:06.046417 398 ts_itest-base.cc:112] Starting cluster with:
I20240503 02:00:06.046455 398 ts_itest-base.cc:113] --------------
I20240503 02:00:06.046468 398 ts_itest-base.cc:114] 4 tablet servers
I20240503 02:00:06.046479 398 ts_itest-base.cc:115] 2 replicas per TS
I20240503 02:00:06.046494 398 ts_itest-base.cc:116] --------------
2024-05-03T02:00:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T02:00:06Z Disabled control of system clock
I20240503 02:00:06.054967 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task42rE7p/build/debug/bin/kudu
/tmp/dist-test-task42rE7p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:46163
--webserver_interface=127.0.99.190
--webserver_port=0
--builtin_ntp_servers=127.0.99.148:33691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.99.190:46163
--location_mapping_cmd=/tmp/dist-test-task42rE7p/build/debug/bin/testdata/assign-location.py --state_store=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/location-assignment.state --map /A:1 --map /B:2 --map /C:1
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false
--master_client_location_assignment_enabled=false
--allow_unsafe_replication_factor with env {}
W20240503 02:00:06.161710 15011 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240503 02:00:06.161944 15011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:00:06.161973 15011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:00:06.162040 15011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:00:06.164999 15011 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:00:06.165058 15011 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 02:00:06.165084 15011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:00:06.165103 15011 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 02:00:06.165140 15011 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 02:00:06.168668 15011 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:33691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--location_mapping_cmd=/tmp/dist-test-task42rE7p/build/debug/bin/testdata/assign-location.py --state_store=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/location-assignment.state --map /A:1 --map /B:2 --map /C:1
--ipki_ca_key_size=768
--master_addresses=127.0.99.190:46163
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:46163
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.99.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.15011
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 02:00:06.169503 15011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:00:06.170668 15011 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:00:06.178521 15017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:06.179363 15019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:06.180583 15016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:00:06.181602 15011 server_base.cc:1034] running on GCE node
I20240503 02:00:06.182060 15011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:00:06.182926 15011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:00:06.184072 15011 hybrid_clock.cc:648] HybridClock initialized: now 1714701606184064 us; error 27 us; skew 500 ppm
I20240503 02:00:06.185899 15011 webserver.cc:460] Webserver started at http://127.0.99.190:39569/ using document root <none> and password file <none>
I20240503 02:00:06.186440 15011 fs_manager.cc:359] Metadata directory not provided
I20240503 02:00:06.186497 15011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:00:06.186662 15011 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:00:06.188299 15011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "d2de106e8dac43ca93ab19fc1af5501e"
format_stamp: "Formatted at 2024-05-03 02:00:06 on dist-test-slave-dtz3"
I20240503 02:00:06.188807 15011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "d2de106e8dac43ca93ab19fc1af5501e"
format_stamp: "Formatted at 2024-05-03 02:00:06 on dist-test-slave-dtz3"
I20240503 02:00:06.191963 15011 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:06.194031 15025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:06.194932 15011 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:06.195055 15011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/wal
uuid: "d2de106e8dac43ca93ab19fc1af5501e"
format_stamp: "Formatted at 2024-05-03 02:00:06 on dist-test-slave-dtz3"
I20240503 02:00:06.195128 15011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:00:06.221125 15011 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:00:06.221797 15011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:00:06.221954 15011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:00:06.228340 15011 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.190:46163
I20240503 02:00:06.228363 15077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.190:46163 every 8 connection(s)
I20240503 02:00:06.229255 15011 server_base.cc:1166] Dumped server information to /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 02:00:06.230393 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task42rE7p/build/debug/bin/kudu as pid 15011
I20240503 02:00:06.230553 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 02:00:06.232069 15078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:06.237627 15078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2de106e8dac43ca93ab19fc1af5501e: Bootstrap starting.
I20240503 02:00:06.239580 15078 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d2de106e8dac43ca93ab19fc1af5501e: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:06.240235 15078 log.cc:824] T 00000000000000000000000000000000 P d2de106e8dac43ca93ab19fc1af5501e: Log is configured to *not* fsync() on all Append() calls
I20240503 02:00:06.241748 15078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2de106e8dac43ca93ab19fc1af5501e: No bootstrap required, opened a new log
I20240503 02:00:06.244026 15078 raft_consensus.cc:348] T 00000000000000000000000000000000 P d2de106e8dac43ca93ab19fc1af5501e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2de106e8dac43ca93ab19fc1af5501e" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 46163 } }
I20240503 02:00:06.244191 15078 raft_consensus.cc:374] T 00000000000000000000000000000000 P d2de106e8dac43ca93ab19fc1af5501e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:06.244251 15078 raft_consensus.cc:729] T 00000000000000000000000000000000 P d2de106e8dac43ca93ab19fc1af5501e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2de106e8dac43ca93ab19fc1af5501e, State: Initialized, Role: FOLLOWER
I20240503 02:00:06.244608 15078 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2de106e8dac43ca93ab19fc1af5501e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2de106e8dac43ca93ab19fc1af5501e" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 46163 } }
I20240503 02:00:06.244751 15078 raft_consensus.cc:388] T 00000000000000000000000000000000 P d2de106e8dac43ca93ab19fc1af5501e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:00:06.244808 15078 raft_consensus.cc:482] T 00000000000000000000000000000000 P d2de106e8dac43ca93ab19fc1af5501e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:00:06.244907 15078 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d2de106e8dac43ca93ab19fc1af5501e [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:06.245580 15078 raft_consensus.cc:504] T 00000000000000000000000000000000 P d2de106e8dac43ca93ab19fc1af5501e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2de106e8dac43ca93ab19fc1af5501e" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 46163 } }
I20240503 02:00:06.245905 15078 leader_election.cc:302] T 00000000000000000000000000000000 P d2de106e8dac43ca93ab19fc1af5501e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d2de106e8dac43ca93ab19fc1af5501e; no voters:
I20240503 02:00:06.246152 15078 leader_election.cc:288] T 00000000000000000000000000000000 P d2de106e8dac43ca93ab19fc1af5501e [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:00:06.246233 15083 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d2de106e8dac43ca93ab19fc1af5501e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:06.246819 15083 raft_consensus.cc:686] T 00000000000000000000000000000000 P d2de106e8dac43ca93ab19fc1af5501e [term 1 LEADER]: Becoming Leader. State: Replica: d2de106e8dac43ca93ab19fc1af5501e, State: Running, Role: LEADER
I20240503 02:00:06.247125 15078 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2de106e8dac43ca93ab19fc1af5501e [sys.catalog]: configured and running, proceeding with master startup.
I20240503 02:00:06.247103 15083 consensus_queue.cc:237] T 00000000000000000000000000000000 P d2de106e8dac43ca93ab19fc1af5501e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2de106e8dac43ca93ab19fc1af5501e" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 46163 } }
I20240503 02:00:06.248445 15085 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2de106e8dac43ca93ab19fc1af5501e [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2de106e8dac43ca93ab19fc1af5501e. Latest consensus state: current_term: 1 leader_uuid: "d2de106e8dac43ca93ab19fc1af5501e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2de106e8dac43ca93ab19fc1af5501e" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 46163 } } }
I20240503 02:00:06.248569 15085 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2de106e8dac43ca93ab19fc1af5501e [sys.catalog]: This master's current role is: LEADER
I20240503 02:00:06.248880 15084 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2de106e8dac43ca93ab19fc1af5501e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d2de106e8dac43ca93ab19fc1af5501e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2de106e8dac43ca93ab19fc1af5501e" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 46163 } } }
I20240503 02:00:06.248966 15084 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2de106e8dac43ca93ab19fc1af5501e [sys.catalog]: This master's current role is: LEADER
I20240503 02:00:06.249720 15095 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 02:00:06.252331 15095 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 02:00:06.257318 15095 catalog_manager.cc:1347] Generated new cluster ID: b3f8fc8be2bd4af0bfdf2c3345a24ca6
I20240503 02:00:06.257408 15095 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 02:00:06.285643 15095 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 02:00:06.286592 15095 catalog_manager.cc:1504] Loading token signing keys...
I20240503 02:00:06.295043 15095 catalog_manager.cc:5857] T 00000000000000000000000000000000 P d2de106e8dac43ca93ab19fc1af5501e: Generated new TSK 0
I20240503 02:00:06.295758 15095 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 02:00:06.310622 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task42rE7p/build/debug/bin/kudu
/tmp/dist-test-task42rE7p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.129:0
--local_ip_for_outbound_sockets=127.0.99.129
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:46163
--builtin_ntp_servers=127.0.99.148:33691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 02:00:06.415621 15102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:00:06.415831 15102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:00:06.415884 15102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:00:06.418968 15102 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:00:06.419031 15102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:00:06.419123 15102 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.129
I20240503 02:00:06.422609 15102 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:33691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:46163
--never_fsync=true
--heap_profile_path=/tmp/kudu.15102
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.129
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 02:00:06.423456 15102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:00:06.424512 15102 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:00:06.433646 15110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:06.434470 15107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:06.435322 15108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:00:06.436611 15102 server_base.cc:1034] running on GCE node
I20240503 02:00:06.437031 15102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:00:06.437662 15102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:00:06.438819 15102 hybrid_clock.cc:648] HybridClock initialized: now 1714701606438800 us; error 35 us; skew 500 ppm
I20240503 02:00:06.440583 15102 webserver.cc:460] Webserver started at http://127.0.99.129:44623/ using document root <none> and password file <none>
I20240503 02:00:06.441135 15102 fs_manager.cc:359] Metadata directory not provided
I20240503 02:00:06.441196 15102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:00:06.441390 15102 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:00:06.443009 15102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "5751ea0f088543bcb141b400a6131a1f"
format_stamp: "Formatted at 2024-05-03 02:00:06 on dist-test-slave-dtz3"
I20240503 02:00:06.443477 15102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "5751ea0f088543bcb141b400a6131a1f"
format_stamp: "Formatted at 2024-05-03 02:00:06 on dist-test-slave-dtz3"
I20240503 02:00:06.448006 15102 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240503 02:00:06.451285 15116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:06.452811 15102 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20240503 02:00:06.452935 15102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "5751ea0f088543bcb141b400a6131a1f"
format_stamp: "Formatted at 2024-05-03 02:00:06 on dist-test-slave-dtz3"
I20240503 02:00:06.453043 15102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:00:06.465499 15102 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:00:06.466054 15102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:00:06.466203 15102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:00:06.466699 15102 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:00:06.467553 15102 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:00:06.467602 15102 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:06.467645 15102 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:00:06.467676 15102 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:06.475925 15102 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.129:46041
I20240503 02:00:06.475975 15229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.129:46041 every 8 connection(s)
I20240503 02:00:06.476891 15102 server_base.cc:1166] Dumped server information to /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 02:00:06.485267 15230 heartbeater.cc:347] Connected to a master server at 127.0.99.190:46163
I20240503 02:00:06.485493 15230 heartbeater.cc:464] Registering TS with master...
I20240503 02:00:06.485992 15230 heartbeater.cc:511] Master 127.0.99.190:46163 requested a full tablet report, sending...
I20240503 02:00:06.486447 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task42rE7p/build/debug/bin/kudu as pid 15102
I20240503 02:00:06.486574 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 02:00:06.491140 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task42rE7p/build/debug/bin/kudu
/tmp/dist-test-task42rE7p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.130:0
--local_ip_for_outbound_sockets=127.0.99.130
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:46163
--builtin_ntp_servers=127.0.99.148:33691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:00:06.536187 15042 ts_manager.cc:194] Registered new tserver with Master: 5751ea0f088543bcb141b400a6131a1f (127.0.99.129:46041)
I20240503 02:00:06.538074 15042 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.129:34101
W20240503 02:00:06.612965 15234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:00:06.613176 15234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:00:06.613240 15234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:00:06.616117 15234 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:00:06.616186 15234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:00:06.616271 15234 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.130
I20240503 02:00:06.619768 15234 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:33691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:46163
--never_fsync=true
--heap_profile_path=/tmp/kudu.15234
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.130
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 02:00:06.620635 15234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:00:06.621728 15234 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:00:06.631655 15240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:06.632514 15239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:06.633332 15242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:00:06.634668 15234 server_base.cc:1034] running on GCE node
I20240503 02:00:06.635030 15234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:00:06.635681 15234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:00:06.636844 15234 hybrid_clock.cc:648] HybridClock initialized: now 1714701606636823 us; error 35 us; skew 500 ppm
I20240503 02:00:06.638595 15234 webserver.cc:460] Webserver started at http://127.0.99.130:34995/ using document root <none> and password file <none>
I20240503 02:00:06.639137 15234 fs_manager.cc:359] Metadata directory not provided
I20240503 02:00:06.639191 15234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:00:06.639369 15234 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:00:06.641084 15234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "7932a656946f4546a91738584eb612e2"
format_stamp: "Formatted at 2024-05-03 02:00:06 on dist-test-slave-dtz3"
I20240503 02:00:06.641549 15234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "7932a656946f4546a91738584eb612e2"
format_stamp: "Formatted at 2024-05-03 02:00:06 on dist-test-slave-dtz3"
I20240503 02:00:06.644981 15234 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240503 02:00:06.647437 15248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:06.648690 15234 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:06.648800 15234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "7932a656946f4546a91738584eb612e2"
format_stamp: "Formatted at 2024-05-03 02:00:06 on dist-test-slave-dtz3"
I20240503 02:00:06.648895 15234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:00:06.667382 15234 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:00:06.667980 15234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:00:06.668130 15234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:00:06.668706 15234 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:00:06.669610 15234 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:00:06.669661 15234 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:06.669720 15234 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:00:06.669747 15234 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:06.679584 15234 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.130:46271
I20240503 02:00:06.679630 15361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.130:46271 every 8 connection(s)
I20240503 02:00:06.680619 15234 server_base.cc:1166] Dumped server information to /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 02:00:06.687729 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task42rE7p/build/debug/bin/kudu as pid 15234
I20240503 02:00:06.687898 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 02:00:06.690389 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task42rE7p/build/debug/bin/kudu
/tmp/dist-test-task42rE7p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.131:0
--local_ip_for_outbound_sockets=127.0.99.131
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:46163
--builtin_ntp_servers=127.0.99.148:33691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:00:06.692395 15362 heartbeater.cc:347] Connected to a master server at 127.0.99.190:46163
I20240503 02:00:06.692729 15362 heartbeater.cc:464] Registering TS with master...
I20240503 02:00:06.693485 15362 heartbeater.cc:511] Master 127.0.99.190:46163 requested a full tablet report, sending...
I20240503 02:00:06.746513 15042 ts_manager.cc:194] Registered new tserver with Master: 7932a656946f4546a91738584eb612e2 (127.0.99.130:46271)
I20240503 02:00:06.747191 15042 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.130:49653
W20240503 02:00:06.816035 15365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:00:06.816288 15365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:00:06.816363 15365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:00:06.819438 15365 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:00:06.819511 15365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:00:06.819608 15365 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.131
I20240503 02:00:06.823225 15365 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:33691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:46163
--never_fsync=true
--heap_profile_path=/tmp/kudu.15365
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.131
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 02:00:06.824127 15365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:00:06.825240 15365 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:00:06.834522 15371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:06.835821 15374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:06.836740 15372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:00:06.837805 15365 server_base.cc:1034] running on GCE node
I20240503 02:00:06.838243 15365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:00:06.838944 15365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:00:06.840116 15365 hybrid_clock.cc:648] HybridClock initialized: now 1714701606840103 us; error 30 us; skew 500 ppm
I20240503 02:00:06.842046 15365 webserver.cc:460] Webserver started at http://127.0.99.131:37705/ using document root <none> and password file <none>
I20240503 02:00:06.842600 15365 fs_manager.cc:359] Metadata directory not provided
I20240503 02:00:06.842660 15365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:00:06.842836 15365 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:00:06.844489 15365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "3f852460ab584dfb937c35f39f19f394"
format_stamp: "Formatted at 2024-05-03 02:00:06 on dist-test-slave-dtz3"
I20240503 02:00:06.844980 15365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "3f852460ab584dfb937c35f39f19f394"
format_stamp: "Formatted at 2024-05-03 02:00:06 on dist-test-slave-dtz3"
I20240503 02:00:06.848636 15365 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240503 02:00:06.850935 15380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:06.852049 15365 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:06.852149 15365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "3f852460ab584dfb937c35f39f19f394"
format_stamp: "Formatted at 2024-05-03 02:00:06 on dist-test-slave-dtz3"
I20240503 02:00:06.852254 15365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:00:06.875505 15365 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:00:06.876103 15365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:00:06.876266 15365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:00:06.876807 15365 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:00:06.877678 15365 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:00:06.877727 15365 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:06.877768 15365 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:00:06.877795 15365 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:06.886101 15365 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.131:34755
I20240503 02:00:06.886119 15493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.131:34755 every 8 connection(s)
I20240503 02:00:06.887024 15365 server_base.cc:1166] Dumped server information to /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 02:00:06.895592 15494 heartbeater.cc:347] Connected to a master server at 127.0.99.190:46163
I20240503 02:00:06.895797 15494 heartbeater.cc:464] Registering TS with master...
I20240503 02:00:06.896276 15494 heartbeater.cc:511] Master 127.0.99.190:46163 requested a full tablet report, sending...
I20240503 02:00:06.897197 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task42rE7p/build/debug/bin/kudu as pid 15365
I20240503 02:00:06.897300 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 02:00:06.902654 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task42rE7p/build/debug/bin/kudu
/tmp/dist-test-task42rE7p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.132:0
--local_ip_for_outbound_sockets=127.0.99.132
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:46163
--builtin_ntp_servers=127.0.99.148:33691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:00:06.933324 15042 ts_manager.cc:194] Registered new tserver with Master: 3f852460ab584dfb937c35f39f19f394 (127.0.99.131:34755)
I20240503 02:00:06.934032 15042 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.131:41057
W20240503 02:00:07.007918 15498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:00:07.008122 15498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:00:07.008190 15498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:00:07.011075 15498 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:00:07.011138 15498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:00:07.011221 15498 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.132
I20240503 02:00:07.014639 15498 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:33691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:46163
--never_fsync=true
--heap_profile_path=/tmp/kudu.15498
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.132
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 02:00:07.015455 15498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:00:07.016477 15498 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:00:07.025266 15503 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:07.026401 15504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:07.027352 15506 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:00:07.028339 15498 server_base.cc:1034] running on GCE node
I20240503 02:00:07.028820 15498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:00:07.029534 15498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:00:07.030705 15498 hybrid_clock.cc:648] HybridClock initialized: now 1714701607030694 us; error 28 us; skew 500 ppm
I20240503 02:00:07.032506 15498 webserver.cc:460] Webserver started at http://127.0.99.132:46643/ using document root <none> and password file <none>
I20240503 02:00:07.033082 15498 fs_manager.cc:359] Metadata directory not provided
I20240503 02:00:07.033149 15498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:00:07.033335 15498 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:00:07.035007 15498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "17eb0a665dc648d9a5ad2c62930861c4"
format_stamp: "Formatted at 2024-05-03 02:00:07 on dist-test-slave-dtz3"
I20240503 02:00:07.035477 15498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "17eb0a665dc648d9a5ad2c62930861c4"
format_stamp: "Formatted at 2024-05-03 02:00:07 on dist-test-slave-dtz3"
I20240503 02:00:07.038846 15498 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20240503 02:00:07.040979 15512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:07.041927 15498 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:07.042042 15498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "17eb0a665dc648d9a5ad2c62930861c4"
format_stamp: "Formatted at 2024-05-03 02:00:07 on dist-test-slave-dtz3"
I20240503 02:00:07.042124 15498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:00:07.062434 15498 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:00:07.063061 15498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:00:07.063223 15498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:00:07.063757 15498 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:00:07.064631 15498 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:00:07.064716 15498 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:07.064785 15498 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:00:07.064810 15498 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:07.072923 15498 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.132:33559
I20240503 02:00:07.072963 15625 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.132:33559 every 8 connection(s)
I20240503 02:00:07.073833 15498 server_base.cc:1166] Dumped server information to /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 02:00:07.078200 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task42rE7p/build/debug/bin/kudu as pid 15498
I20240503 02:00:07.078339 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 02:00:07.082789 15626 heartbeater.cc:347] Connected to a master server at 127.0.99.190:46163
I20240503 02:00:07.083011 15626 heartbeater.cc:464] Registering TS with master...
I20240503 02:00:07.083514 15626 heartbeater.cc:511] Master 127.0.99.190:46163 requested a full tablet report, sending...
I20240503 02:00:07.120216 15042 ts_manager.cc:194] Registered new tserver with Master: 17eb0a665dc648d9a5ad2c62930861c4 (127.0.99.132:33559)
I20240503 02:00:07.120925 15042 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.132:60221
I20240503 02:00:07.128136 398 external_mini_cluster.cc:930] 4 TS(s) registered with all masters
I20240503 02:00:07.137080 398 external_mini_cluster.cc:930] 4 TS(s) registered with all masters
I20240503 02:00:07.137154 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task42rE7p/build/debug/bin/kudu with pid 15234
I20240503 02:00:07.142577 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task42rE7p/build/debug/bin/kudu with pid 15365
I20240503 02:00:07.541045 15230 heartbeater.cc:503] Master 127.0.99.190:46163 was elected leader, sending a full tablet report...
I20240503 02:00:08.123276 15626 heartbeater.cc:503] Master 127.0.99.190:46163 was elected leader, sending a full tablet report...
I20240503 02:00:10.900094 15042 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59506:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240503 02:00:10.903841 15042 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_0 in case of a server failure: 3 tablet servers would be needed, 2 are available. Consider bringing up more tablet servers.
I20240503 02:00:10.920595 15559 tablet_service.cc:1467] Processing CreateTablet for tablet 84425c87d5964b4bb5c5ecfb72360e07 (DEFAULT_TABLE table=rebalance_test_table_0 [id=b9f4cdc190fd486bb9d37c53db387db4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:10.920598 15164 tablet_service.cc:1467] Processing CreateTablet for tablet 14fabb00ac3149688d216d542b60679e (DEFAULT_TABLE table=rebalance_test_table_0 [id=b9f4cdc190fd486bb9d37c53db387db4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:10.920786 15162 tablet_service.cc:1467] Processing CreateTablet for tablet 4438db9442024011bd8d5632a03d34b0 (DEFAULT_TABLE table=rebalance_test_table_0 [id=b9f4cdc190fd486bb9d37c53db387db4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:10.921267 15560 tablet_service.cc:1467] Processing CreateTablet for tablet 14fabb00ac3149688d216d542b60679e (DEFAULT_TABLE table=rebalance_test_table_0 [id=b9f4cdc190fd486bb9d37c53db387db4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:10.920595 15558 tablet_service.cc:1467] Processing CreateTablet for tablet 4438db9442024011bd8d5632a03d34b0 (DEFAULT_TABLE table=rebalance_test_table_0 [id=b9f4cdc190fd486bb9d37c53db387db4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:10.921767 15559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84425c87d5964b4bb5c5ecfb72360e07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:10.921767 15164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14fabb00ac3149688d216d542b60679e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:10.920598 15163 tablet_service.cc:1467] Processing CreateTablet for tablet 84425c87d5964b4bb5c5ecfb72360e07 (DEFAULT_TABLE table=rebalance_test_table_0 [id=b9f4cdc190fd486bb9d37c53db387db4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:10.922109 15163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84425c87d5964b4bb5c5ecfb72360e07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:10.922137 15560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14fabb00ac3149688d216d542b60679e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:10.922947 15162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4438db9442024011bd8d5632a03d34b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:10.923422 15558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4438db9442024011bd8d5632a03d34b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:10.928939 15644 tablet_bootstrap.cc:492] T 4438db9442024011bd8d5632a03d34b0 P 17eb0a665dc648d9a5ad2c62930861c4: Bootstrap starting.
I20240503 02:00:10.931520 15644 tablet_bootstrap.cc:654] T 4438db9442024011bd8d5632a03d34b0 P 17eb0a665dc648d9a5ad2c62930861c4: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:10.932598 15643 tablet_bootstrap.cc:492] T 4438db9442024011bd8d5632a03d34b0 P 5751ea0f088543bcb141b400a6131a1f: Bootstrap starting.
I20240503 02:00:10.932727 15644 log.cc:824] T 4438db9442024011bd8d5632a03d34b0 P 17eb0a665dc648d9a5ad2c62930861c4: Log is configured to *not* fsync() on all Append() calls
I20240503 02:00:10.934535 15644 tablet_bootstrap.cc:492] T 4438db9442024011bd8d5632a03d34b0 P 17eb0a665dc648d9a5ad2c62930861c4: No bootstrap required, opened a new log
I20240503 02:00:10.934605 15643 tablet_bootstrap.cc:654] T 4438db9442024011bd8d5632a03d34b0 P 5751ea0f088543bcb141b400a6131a1f: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:10.934722 15644 ts_tablet_manager.cc:1367] T 4438db9442024011bd8d5632a03d34b0 P 17eb0a665dc648d9a5ad2c62930861c4: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240503 02:00:10.935379 15643 log.cc:824] T 4438db9442024011bd8d5632a03d34b0 P 5751ea0f088543bcb141b400a6131a1f: Log is configured to *not* fsync() on all Append() calls
I20240503 02:00:10.937026 15643 tablet_bootstrap.cc:492] T 4438db9442024011bd8d5632a03d34b0 P 5751ea0f088543bcb141b400a6131a1f: No bootstrap required, opened a new log
I20240503 02:00:10.937188 15643 ts_tablet_manager.cc:1367] T 4438db9442024011bd8d5632a03d34b0 P 5751ea0f088543bcb141b400a6131a1f: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20240503 02:00:10.938184 15644 raft_consensus.cc:348] T 4438db9442024011bd8d5632a03d34b0 P 17eb0a665dc648d9a5ad2c62930861c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } } peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } }
I20240503 02:00:10.938405 15644 raft_consensus.cc:374] T 4438db9442024011bd8d5632a03d34b0 P 17eb0a665dc648d9a5ad2c62930861c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:10.938459 15644 raft_consensus.cc:729] T 4438db9442024011bd8d5632a03d34b0 P 17eb0a665dc648d9a5ad2c62930861c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17eb0a665dc648d9a5ad2c62930861c4, State: Initialized, Role: FOLLOWER
I20240503 02:00:10.938946 15644 consensus_queue.cc:260] T 4438db9442024011bd8d5632a03d34b0 P 17eb0a665dc648d9a5ad2c62930861c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } } peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } }
I20240503 02:00:10.939851 15644 ts_tablet_manager.cc:1398] T 4438db9442024011bd8d5632a03d34b0 P 17eb0a665dc648d9a5ad2c62930861c4: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20240503 02:00:10.940207 15644 tablet_bootstrap.cc:492] T 84425c87d5964b4bb5c5ecfb72360e07 P 17eb0a665dc648d9a5ad2c62930861c4: Bootstrap starting.
I20240503 02:00:10.940606 15643 raft_consensus.cc:348] T 4438db9442024011bd8d5632a03d34b0 P 5751ea0f088543bcb141b400a6131a1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } } peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } }
I20240503 02:00:10.940910 15643 raft_consensus.cc:374] T 4438db9442024011bd8d5632a03d34b0 P 5751ea0f088543bcb141b400a6131a1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:10.941027 15643 raft_consensus.cc:729] T 4438db9442024011bd8d5632a03d34b0 P 5751ea0f088543bcb141b400a6131a1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5751ea0f088543bcb141b400a6131a1f, State: Initialized, Role: FOLLOWER
I20240503 02:00:10.941210 15644 tablet_bootstrap.cc:654] T 84425c87d5964b4bb5c5ecfb72360e07 P 17eb0a665dc648d9a5ad2c62930861c4: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:10.941594 15643 consensus_queue.cc:260] T 4438db9442024011bd8d5632a03d34b0 P 5751ea0f088543bcb141b400a6131a1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } } peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } }
I20240503 02:00:10.942351 15644 tablet_bootstrap.cc:492] T 84425c87d5964b4bb5c5ecfb72360e07 P 17eb0a665dc648d9a5ad2c62930861c4: No bootstrap required, opened a new log
I20240503 02:00:10.942435 15643 ts_tablet_manager.cc:1398] T 4438db9442024011bd8d5632a03d34b0 P 5751ea0f088543bcb141b400a6131a1f: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240503 02:00:10.942451 15644 ts_tablet_manager.cc:1367] T 84425c87d5964b4bb5c5ecfb72360e07 P 17eb0a665dc648d9a5ad2c62930861c4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:10.942678 15643 tablet_bootstrap.cc:492] T 84425c87d5964b4bb5c5ecfb72360e07 P 5751ea0f088543bcb141b400a6131a1f: Bootstrap starting.
I20240503 02:00:10.942974 15644 raft_consensus.cc:348] T 84425c87d5964b4bb5c5ecfb72360e07 P 17eb0a665dc648d9a5ad2c62930861c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } } peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } }
I20240503 02:00:10.943094 15644 raft_consensus.cc:374] T 84425c87d5964b4bb5c5ecfb72360e07 P 17eb0a665dc648d9a5ad2c62930861c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:10.943132 15644 raft_consensus.cc:729] T 84425c87d5964b4bb5c5ecfb72360e07 P 17eb0a665dc648d9a5ad2c62930861c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17eb0a665dc648d9a5ad2c62930861c4, State: Initialized, Role: FOLLOWER
I20240503 02:00:10.943245 15644 consensus_queue.cc:260] T 84425c87d5964b4bb5c5ecfb72360e07 P 17eb0a665dc648d9a5ad2c62930861c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } } peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } }
I20240503 02:00:10.943486 15644 ts_tablet_manager.cc:1398] T 84425c87d5964b4bb5c5ecfb72360e07 P 17eb0a665dc648d9a5ad2c62930861c4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:10.943607 15643 tablet_bootstrap.cc:654] T 84425c87d5964b4bb5c5ecfb72360e07 P 5751ea0f088543bcb141b400a6131a1f: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:10.943640 15644 tablet_bootstrap.cc:492] T 14fabb00ac3149688d216d542b60679e P 17eb0a665dc648d9a5ad2c62930861c4: Bootstrap starting.
I20240503 02:00:10.944500 15644 tablet_bootstrap.cc:654] T 14fabb00ac3149688d216d542b60679e P 17eb0a665dc648d9a5ad2c62930861c4: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:10.945657 15643 tablet_bootstrap.cc:492] T 84425c87d5964b4bb5c5ecfb72360e07 P 5751ea0f088543bcb141b400a6131a1f: No bootstrap required, opened a new log
I20240503 02:00:10.945744 15643 ts_tablet_manager.cc:1367] T 84425c87d5964b4bb5c5ecfb72360e07 P 5751ea0f088543bcb141b400a6131a1f: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240503 02:00:10.945852 15644 tablet_bootstrap.cc:492] T 14fabb00ac3149688d216d542b60679e P 17eb0a665dc648d9a5ad2c62930861c4: No bootstrap required, opened a new log
I20240503 02:00:10.945940 15644 ts_tablet_manager.cc:1367] T 14fabb00ac3149688d216d542b60679e P 17eb0a665dc648d9a5ad2c62930861c4: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 02:00:10.946323 15643 raft_consensus.cc:348] T 84425c87d5964b4bb5c5ecfb72360e07 P 5751ea0f088543bcb141b400a6131a1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } } peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } }
I20240503 02:00:10.946362 15644 raft_consensus.cc:348] T 14fabb00ac3149688d216d542b60679e P 17eb0a665dc648d9a5ad2c62930861c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } } peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } }
I20240503 02:00:10.946399 15643 raft_consensus.cc:374] T 84425c87d5964b4bb5c5ecfb72360e07 P 5751ea0f088543bcb141b400a6131a1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:10.946431 15644 raft_consensus.cc:374] T 14fabb00ac3149688d216d542b60679e P 17eb0a665dc648d9a5ad2c62930861c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:10.946431 15643 raft_consensus.cc:729] T 84425c87d5964b4bb5c5ecfb72360e07 P 5751ea0f088543bcb141b400a6131a1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5751ea0f088543bcb141b400a6131a1f, State: Initialized, Role: FOLLOWER
I20240503 02:00:10.946476 15644 raft_consensus.cc:729] T 14fabb00ac3149688d216d542b60679e P 17eb0a665dc648d9a5ad2c62930861c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17eb0a665dc648d9a5ad2c62930861c4, State: Initialized, Role: FOLLOWER
I20240503 02:00:10.946561 15643 consensus_queue.cc:260] T 84425c87d5964b4bb5c5ecfb72360e07 P 5751ea0f088543bcb141b400a6131a1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } } peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } }
I20240503 02:00:10.946596 15644 consensus_queue.cc:260] T 14fabb00ac3149688d216d542b60679e P 17eb0a665dc648d9a5ad2c62930861c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } } peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } }
I20240503 02:00:10.946841 15643 ts_tablet_manager.cc:1398] T 84425c87d5964b4bb5c5ecfb72360e07 P 5751ea0f088543bcb141b400a6131a1f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:10.946952 15644 ts_tablet_manager.cc:1398] T 14fabb00ac3149688d216d542b60679e P 17eb0a665dc648d9a5ad2c62930861c4: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:10.946985 15643 tablet_bootstrap.cc:492] T 14fabb00ac3149688d216d542b60679e P 5751ea0f088543bcb141b400a6131a1f: Bootstrap starting.
I20240503 02:00:10.947990 15643 tablet_bootstrap.cc:654] T 14fabb00ac3149688d216d542b60679e P 5751ea0f088543bcb141b400a6131a1f: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:10.949390 15643 tablet_bootstrap.cc:492] T 14fabb00ac3149688d216d542b60679e P 5751ea0f088543bcb141b400a6131a1f: No bootstrap required, opened a new log
I20240503 02:00:10.949470 15643 ts_tablet_manager.cc:1367] T 14fabb00ac3149688d216d542b60679e P 5751ea0f088543bcb141b400a6131a1f: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240503 02:00:10.949899 15643 raft_consensus.cc:348] T 14fabb00ac3149688d216d542b60679e P 5751ea0f088543bcb141b400a6131a1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } } peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } }
I20240503 02:00:10.949987 15643 raft_consensus.cc:374] T 14fabb00ac3149688d216d542b60679e P 5751ea0f088543bcb141b400a6131a1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:10.950017 15643 raft_consensus.cc:729] T 14fabb00ac3149688d216d542b60679e P 5751ea0f088543bcb141b400a6131a1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5751ea0f088543bcb141b400a6131a1f, State: Initialized, Role: FOLLOWER
I20240503 02:00:10.950129 15643 consensus_queue.cc:260] T 14fabb00ac3149688d216d542b60679e P 5751ea0f088543bcb141b400a6131a1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } } peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } }
I20240503 02:00:10.950428 15643 ts_tablet_manager.cc:1398] T 14fabb00ac3149688d216d542b60679e P 5751ea0f088543bcb141b400a6131a1f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:10.975239 15649 raft_consensus.cc:482] T 14fabb00ac3149688d216d542b60679e P 5751ea0f088543bcb141b400a6131a1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:10.975452 15649 raft_consensus.cc:504] T 14fabb00ac3149688d216d542b60679e P 5751ea0f088543bcb141b400a6131a1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } } peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } }
I20240503 02:00:10.976614 15649 leader_election.cc:288] T 14fabb00ac3149688d216d542b60679e P 5751ea0f088543bcb141b400a6131a1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 17eb0a665dc648d9a5ad2c62930861c4 (127.0.99.132:33559)
W20240503 02:00:10.981120 15231 tablet.cc:2369] T 14fabb00ac3149688d216d542b60679e P 5751ea0f088543bcb141b400a6131a1f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:00:10.981088 15580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14fabb00ac3149688d216d542b60679e" candidate_uuid: "5751ea0f088543bcb141b400a6131a1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17eb0a665dc648d9a5ad2c62930861c4" is_pre_election: true
I20240503 02:00:10.981428 15580 raft_consensus.cc:2454] T 14fabb00ac3149688d216d542b60679e P 17eb0a665dc648d9a5ad2c62930861c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5751ea0f088543bcb141b400a6131a1f in term 0.
I20240503 02:00:10.981868 15120 leader_election.cc:302] T 14fabb00ac3149688d216d542b60679e P 5751ea0f088543bcb141b400a6131a1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 17eb0a665dc648d9a5ad2c62930861c4, 5751ea0f088543bcb141b400a6131a1f; no voters:
I20240503 02:00:10.982093 15649 raft_consensus.cc:2781] T 14fabb00ac3149688d216d542b60679e P 5751ea0f088543bcb141b400a6131a1f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:10.982159 15649 raft_consensus.cc:482] T 14fabb00ac3149688d216d542b60679e P 5751ea0f088543bcb141b400a6131a1f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:10.982203 15649 raft_consensus.cc:3037] T 14fabb00ac3149688d216d542b60679e P 5751ea0f088543bcb141b400a6131a1f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:10.983182 15649 raft_consensus.cc:504] T 14fabb00ac3149688d216d542b60679e P 5751ea0f088543bcb141b400a6131a1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } } peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } }
I20240503 02:00:10.983424 15649 leader_election.cc:288] T 14fabb00ac3149688d216d542b60679e P 5751ea0f088543bcb141b400a6131a1f [CANDIDATE]: Term 1 election: Requested vote from peers 17eb0a665dc648d9a5ad2c62930861c4 (127.0.99.132:33559)
I20240503 02:00:10.983789 15580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14fabb00ac3149688d216d542b60679e" candidate_uuid: "5751ea0f088543bcb141b400a6131a1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17eb0a665dc648d9a5ad2c62930861c4"
I20240503 02:00:10.983886 15580 raft_consensus.cc:3037] T 14fabb00ac3149688d216d542b60679e P 17eb0a665dc648d9a5ad2c62930861c4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:10.984827 15580 raft_consensus.cc:2454] T 14fabb00ac3149688d216d542b60679e P 17eb0a665dc648d9a5ad2c62930861c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5751ea0f088543bcb141b400a6131a1f in term 1.
I20240503 02:00:10.985074 15120 leader_election.cc:302] T 14fabb00ac3149688d216d542b60679e P 5751ea0f088543bcb141b400a6131a1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 17eb0a665dc648d9a5ad2c62930861c4, 5751ea0f088543bcb141b400a6131a1f; no voters:
I20240503 02:00:10.985217 15649 raft_consensus.cc:2781] T 14fabb00ac3149688d216d542b60679e P 5751ea0f088543bcb141b400a6131a1f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:10.985430 15649 raft_consensus.cc:686] T 14fabb00ac3149688d216d542b60679e P 5751ea0f088543bcb141b400a6131a1f [term 1 LEADER]: Becoming Leader. State: Replica: 5751ea0f088543bcb141b400a6131a1f, State: Running, Role: LEADER
I20240503 02:00:10.985738 15649 consensus_queue.cc:237] T 14fabb00ac3149688d216d542b60679e P 5751ea0f088543bcb141b400a6131a1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } } peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } }
I20240503 02:00:10.988888 15040 catalog_manager.cc:5484] T 14fabb00ac3149688d216d542b60679e P 5751ea0f088543bcb141b400a6131a1f reported cstate change: term changed from 0 to 1, leader changed from <none> to 5751ea0f088543bcb141b400a6131a1f (127.0.99.129). New cstate: current_term: 1 leader_uuid: "5751ea0f088543bcb141b400a6131a1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } health_report { overall_health: UNKNOWN } } }
W20240503 02:00:11.078383 15627 tablet.cc:2369] T 14fabb00ac3149688d216d542b60679e P 17eb0a665dc648d9a5ad2c62930861c4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:00:11.137893 15649 raft_consensus.cc:482] T 4438db9442024011bd8d5632a03d34b0 P 5751ea0f088543bcb141b400a6131a1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:11.138085 15649 raft_consensus.cc:504] T 4438db9442024011bd8d5632a03d34b0 P 5751ea0f088543bcb141b400a6131a1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } } peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } }
I20240503 02:00:11.138547 15649 leader_election.cc:288] T 4438db9442024011bd8d5632a03d34b0 P 5751ea0f088543bcb141b400a6131a1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 17eb0a665dc648d9a5ad2c62930861c4 (127.0.99.132:33559)
I20240503 02:00:11.139222 15580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4438db9442024011bd8d5632a03d34b0" candidate_uuid: "5751ea0f088543bcb141b400a6131a1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17eb0a665dc648d9a5ad2c62930861c4" is_pre_election: true
I20240503 02:00:11.139385 15580 raft_consensus.cc:2454] T 4438db9442024011bd8d5632a03d34b0 P 17eb0a665dc648d9a5ad2c62930861c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5751ea0f088543bcb141b400a6131a1f in term 0.
I20240503 02:00:11.139678 15120 leader_election.cc:302] T 4438db9442024011bd8d5632a03d34b0 P 5751ea0f088543bcb141b400a6131a1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 17eb0a665dc648d9a5ad2c62930861c4, 5751ea0f088543bcb141b400a6131a1f; no voters:
I20240503 02:00:11.139845 15649 raft_consensus.cc:2781] T 4438db9442024011bd8d5632a03d34b0 P 5751ea0f088543bcb141b400a6131a1f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:11.139905 15649 raft_consensus.cc:482] T 4438db9442024011bd8d5632a03d34b0 P 5751ea0f088543bcb141b400a6131a1f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:11.139941 15649 raft_consensus.cc:3037] T 4438db9442024011bd8d5632a03d34b0 P 5751ea0f088543bcb141b400a6131a1f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:11.140952 15649 raft_consensus.cc:504] T 4438db9442024011bd8d5632a03d34b0 P 5751ea0f088543bcb141b400a6131a1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } } peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } }
I20240503 02:00:11.141214 15649 leader_election.cc:288] T 4438db9442024011bd8d5632a03d34b0 P 5751ea0f088543bcb141b400a6131a1f [CANDIDATE]: Term 1 election: Requested vote from peers 17eb0a665dc648d9a5ad2c62930861c4 (127.0.99.132:33559)
I20240503 02:00:11.141588 15580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4438db9442024011bd8d5632a03d34b0" candidate_uuid: "5751ea0f088543bcb141b400a6131a1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17eb0a665dc648d9a5ad2c62930861c4"
I20240503 02:00:11.141701 15580 raft_consensus.cc:3037] T 4438db9442024011bd8d5632a03d34b0 P 17eb0a665dc648d9a5ad2c62930861c4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:11.142577 15580 raft_consensus.cc:2454] T 4438db9442024011bd8d5632a03d34b0 P 17eb0a665dc648d9a5ad2c62930861c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5751ea0f088543bcb141b400a6131a1f in term 1.
I20240503 02:00:11.142839 15120 leader_election.cc:302] T 4438db9442024011bd8d5632a03d34b0 P 5751ea0f088543bcb141b400a6131a1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 17eb0a665dc648d9a5ad2c62930861c4, 5751ea0f088543bcb141b400a6131a1f; no voters:
I20240503 02:00:11.143003 15649 raft_consensus.cc:2781] T 4438db9442024011bd8d5632a03d34b0 P 5751ea0f088543bcb141b400a6131a1f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:11.143085 15649 raft_consensus.cc:686] T 4438db9442024011bd8d5632a03d34b0 P 5751ea0f088543bcb141b400a6131a1f [term 1 LEADER]: Becoming Leader. State: Replica: 5751ea0f088543bcb141b400a6131a1f, State: Running, Role: LEADER
I20240503 02:00:11.143254 15649 consensus_queue.cc:237] T 4438db9442024011bd8d5632a03d34b0 P 5751ea0f088543bcb141b400a6131a1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } } peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } }
I20240503 02:00:11.145018 15040 catalog_manager.cc:5484] T 4438db9442024011bd8d5632a03d34b0 P 5751ea0f088543bcb141b400a6131a1f reported cstate change: term changed from 0 to 1, leader changed from <none> to 5751ea0f088543bcb141b400a6131a1f (127.0.99.129). New cstate: current_term: 1 leader_uuid: "5751ea0f088543bcb141b400a6131a1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:11.270722 15649 raft_consensus.cc:482] T 84425c87d5964b4bb5c5ecfb72360e07 P 5751ea0f088543bcb141b400a6131a1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:11.270893 15649 raft_consensus.cc:504] T 84425c87d5964b4bb5c5ecfb72360e07 P 5751ea0f088543bcb141b400a6131a1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } } peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } }
I20240503 02:00:11.271368 15649 leader_election.cc:288] T 84425c87d5964b4bb5c5ecfb72360e07 P 5751ea0f088543bcb141b400a6131a1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 17eb0a665dc648d9a5ad2c62930861c4 (127.0.99.132:33559)
I20240503 02:00:11.271976 15580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84425c87d5964b4bb5c5ecfb72360e07" candidate_uuid: "5751ea0f088543bcb141b400a6131a1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17eb0a665dc648d9a5ad2c62930861c4" is_pre_election: true
I20240503 02:00:11.272157 15580 raft_consensus.cc:2454] T 84425c87d5964b4bb5c5ecfb72360e07 P 17eb0a665dc648d9a5ad2c62930861c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5751ea0f088543bcb141b400a6131a1f in term 0.
I20240503 02:00:11.272531 15120 leader_election.cc:302] T 84425c87d5964b4bb5c5ecfb72360e07 P 5751ea0f088543bcb141b400a6131a1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 17eb0a665dc648d9a5ad2c62930861c4, 5751ea0f088543bcb141b400a6131a1f; no voters:
I20240503 02:00:11.272810 15649 raft_consensus.cc:2781] T 84425c87d5964b4bb5c5ecfb72360e07 P 5751ea0f088543bcb141b400a6131a1f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:11.272876 15649 raft_consensus.cc:482] T 84425c87d5964b4bb5c5ecfb72360e07 P 5751ea0f088543bcb141b400a6131a1f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:11.272902 15649 raft_consensus.cc:3037] T 84425c87d5964b4bb5c5ecfb72360e07 P 5751ea0f088543bcb141b400a6131a1f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:11.273800 15649 raft_consensus.cc:504] T 84425c87d5964b4bb5c5ecfb72360e07 P 5751ea0f088543bcb141b400a6131a1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } } peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } }
I20240503 02:00:11.274097 15649 leader_election.cc:288] T 84425c87d5964b4bb5c5ecfb72360e07 P 5751ea0f088543bcb141b400a6131a1f [CANDIDATE]: Term 1 election: Requested vote from peers 17eb0a665dc648d9a5ad2c62930861c4 (127.0.99.132:33559)
I20240503 02:00:11.274513 15580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84425c87d5964b4bb5c5ecfb72360e07" candidate_uuid: "5751ea0f088543bcb141b400a6131a1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17eb0a665dc648d9a5ad2c62930861c4"
I20240503 02:00:11.274637 15580 raft_consensus.cc:3037] T 84425c87d5964b4bb5c5ecfb72360e07 P 17eb0a665dc648d9a5ad2c62930861c4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:11.275475 15580 raft_consensus.cc:2454] T 84425c87d5964b4bb5c5ecfb72360e07 P 17eb0a665dc648d9a5ad2c62930861c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5751ea0f088543bcb141b400a6131a1f in term 1.
I20240503 02:00:11.275790 15120 leader_election.cc:302] T 84425c87d5964b4bb5c5ecfb72360e07 P 5751ea0f088543bcb141b400a6131a1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 17eb0a665dc648d9a5ad2c62930861c4, 5751ea0f088543bcb141b400a6131a1f; no voters:
I20240503 02:00:11.276080 15649 raft_consensus.cc:2781] T 84425c87d5964b4bb5c5ecfb72360e07 P 5751ea0f088543bcb141b400a6131a1f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:11.276212 15649 raft_consensus.cc:686] T 84425c87d5964b4bb5c5ecfb72360e07 P 5751ea0f088543bcb141b400a6131a1f [term 1 LEADER]: Becoming Leader. State: Replica: 5751ea0f088543bcb141b400a6131a1f, State: Running, Role: LEADER
I20240503 02:00:11.276366 15649 consensus_queue.cc:237] T 84425c87d5964b4bb5c5ecfb72360e07 P 5751ea0f088543bcb141b400a6131a1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } } peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } }
I20240503 02:00:11.278345 15040 catalog_manager.cc:5484] T 84425c87d5964b4bb5c5ecfb72360e07 P 5751ea0f088543bcb141b400a6131a1f reported cstate change: term changed from 0 to 1, leader changed from <none> to 5751ea0f088543bcb141b400a6131a1f (127.0.99.129). New cstate: current_term: 1 leader_uuid: "5751ea0f088543bcb141b400a6131a1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } health_report { overall_health: HEALTHY } } }
I20240503 02:00:11.389772 15649 consensus_queue.cc:1035] T 14fabb00ac3149688d216d542b60679e P 5751ea0f088543bcb141b400a6131a1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:00:11.488175 15658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:00:11.488440 15658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:00:11.491376 15658 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240503 02:00:11.588318 15580 raft_consensus.cc:1261] T 4438db9442024011bd8d5632a03d34b0 P 17eb0a665dc648d9a5ad2c62930861c4 [term 1 FOLLOWER]: Refusing update from remote peer 5751ea0f088543bcb141b400a6131a1f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:11.588457 15579 raft_consensus.cc:1261] T 84425c87d5964b4bb5c5ecfb72360e07 P 17eb0a665dc648d9a5ad2c62930861c4 [term 1 FOLLOWER]: Refusing update from remote peer 5751ea0f088543bcb141b400a6131a1f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:11.589027 15654 consensus_queue.cc:1035] T 4438db9442024011bd8d5632a03d34b0 P 5751ea0f088543bcb141b400a6131a1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:11.589026 15649 consensus_queue.cc:1035] T 84425c87d5964b4bb5c5ecfb72360e07 P 5751ea0f088543bcb141b400a6131a1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:11.606034 15661 mvcc.cc:205] Tried to move back new op lower bound from 7023417801058742272 to 7023417799243763712. Current Snapshot: MvccSnapshot[applied={T|T < 7023417801058742272}]
INSERT report
rows total: 2000
time total: 101.523 ms
time per row: 0.0507616 ms
I20240503 02:00:11.638530 15042 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59506:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240503 02:00:11.639128 15042 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_1 in case of a server failure: 3 tablet servers would be needed, 2 are available. Consider bringing up more tablet servers.
I20240503 02:00:11.644526 15560 tablet_service.cc:1467] Processing CreateTablet for tablet a567f7530d30404faa9ebe967138e0b4 (DEFAULT_TABLE table=rebalance_test_table_1 [id=174f433588294c61856c430b17973e89]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:11.644932 15560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a567f7530d30404faa9ebe967138e0b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:11.645267 15163 tablet_service.cc:1467] Processing CreateTablet for tablet 595c6713bec0492882c1e58225dcce42 (DEFAULT_TABLE table=rebalance_test_table_1 [id=174f433588294c61856c430b17973e89]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:11.645591 15163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 595c6713bec0492882c1e58225dcce42. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:11.647169 15702 tablet_bootstrap.cc:492] T a567f7530d30404faa9ebe967138e0b4 P 17eb0a665dc648d9a5ad2c62930861c4: Bootstrap starting.
I20240503 02:00:11.647936 15703 tablet_bootstrap.cc:492] T 595c6713bec0492882c1e58225dcce42 P 5751ea0f088543bcb141b400a6131a1f: Bootstrap starting.
I20240503 02:00:11.648150 15702 tablet_bootstrap.cc:654] T a567f7530d30404faa9ebe967138e0b4 P 17eb0a665dc648d9a5ad2c62930861c4: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:11.648113 15164 tablet_service.cc:1467] Processing CreateTablet for tablet 7244e26e54694bd48373e8634b4daf93 (DEFAULT_TABLE table=rebalance_test_table_1 [id=174f433588294c61856c430b17973e89]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:11.648339 15164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7244e26e54694bd48373e8634b4daf93. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:11.648964 15703 tablet_bootstrap.cc:654] T 595c6713bec0492882c1e58225dcce42 P 5751ea0f088543bcb141b400a6131a1f: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:11.649240 15702 tablet_bootstrap.cc:492] T a567f7530d30404faa9ebe967138e0b4 P 17eb0a665dc648d9a5ad2c62930861c4: No bootstrap required, opened a new log
I20240503 02:00:11.649329 15702 ts_tablet_manager.cc:1367] T a567f7530d30404faa9ebe967138e0b4 P 17eb0a665dc648d9a5ad2c62930861c4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:00:11.649698 15702 raft_consensus.cc:348] T a567f7530d30404faa9ebe967138e0b4 P 17eb0a665dc648d9a5ad2c62930861c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } } peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } }
I20240503 02:00:11.649797 15702 raft_consensus.cc:374] T a567f7530d30404faa9ebe967138e0b4 P 17eb0a665dc648d9a5ad2c62930861c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:11.649834 15702 raft_consensus.cc:729] T a567f7530d30404faa9ebe967138e0b4 P 17eb0a665dc648d9a5ad2c62930861c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17eb0a665dc648d9a5ad2c62930861c4, State: Initialized, Role: FOLLOWER
I20240503 02:00:11.649966 15702 consensus_queue.cc:260] T a567f7530d30404faa9ebe967138e0b4 P 17eb0a665dc648d9a5ad2c62930861c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } } peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } }
I20240503 02:00:11.650192 15702 ts_tablet_manager.cc:1398] T a567f7530d30404faa9ebe967138e0b4 P 17eb0a665dc648d9a5ad2c62930861c4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:11.650586 15162 tablet_service.cc:1467] Processing CreateTablet for tablet a567f7530d30404faa9ebe967138e0b4 (DEFAULT_TABLE table=rebalance_test_table_1 [id=174f433588294c61856c430b17973e89]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:11.650760 15162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a567f7530d30404faa9ebe967138e0b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:11.652762 15560 tablet_service.cc:1467] Processing CreateTablet for tablet 7244e26e54694bd48373e8634b4daf93 (DEFAULT_TABLE table=rebalance_test_table_1 [id=174f433588294c61856c430b17973e89]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:11.652922 15559 tablet_service.cc:1467] Processing CreateTablet for tablet 595c6713bec0492882c1e58225dcce42 (DEFAULT_TABLE table=rebalance_test_table_1 [id=174f433588294c61856c430b17973e89]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:11.653034 15560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7244e26e54694bd48373e8634b4daf93. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:11.653208 15559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 595c6713bec0492882c1e58225dcce42. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:11.653796 15703 tablet_bootstrap.cc:492] T 595c6713bec0492882c1e58225dcce42 P 5751ea0f088543bcb141b400a6131a1f: No bootstrap required, opened a new log
I20240503 02:00:11.653888 15703 ts_tablet_manager.cc:1367] T 595c6713bec0492882c1e58225dcce42 P 5751ea0f088543bcb141b400a6131a1f: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 02:00:11.654393 15703 raft_consensus.cc:348] T 595c6713bec0492882c1e58225dcce42 P 5751ea0f088543bcb141b400a6131a1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } } peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } }
I20240503 02:00:11.654517 15703 raft_consensus.cc:374] T 595c6713bec0492882c1e58225dcce42 P 5751ea0f088543bcb141b400a6131a1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:11.654556 15703 raft_consensus.cc:729] T 595c6713bec0492882c1e58225dcce42 P 5751ea0f088543bcb141b400a6131a1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5751ea0f088543bcb141b400a6131a1f, State: Initialized, Role: FOLLOWER
I20240503 02:00:11.654711 15703 consensus_queue.cc:260] T 595c6713bec0492882c1e58225dcce42 P 5751ea0f088543bcb141b400a6131a1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } } peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } }
I20240503 02:00:11.654984 15703 ts_tablet_manager.cc:1398] T 595c6713bec0492882c1e58225dcce42 P 5751ea0f088543bcb141b400a6131a1f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:11.654978 15702 tablet_bootstrap.cc:492] T 7244e26e54694bd48373e8634b4daf93 P 17eb0a665dc648d9a5ad2c62930861c4: Bootstrap starting.
I20240503 02:00:11.655174 15703 tablet_bootstrap.cc:492] T 7244e26e54694bd48373e8634b4daf93 P 5751ea0f088543bcb141b400a6131a1f: Bootstrap starting.
I20240503 02:00:11.655874 15702 tablet_bootstrap.cc:654] T 7244e26e54694bd48373e8634b4daf93 P 17eb0a665dc648d9a5ad2c62930861c4: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:11.656085 15703 tablet_bootstrap.cc:654] T 7244e26e54694bd48373e8634b4daf93 P 5751ea0f088543bcb141b400a6131a1f: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:11.657011 15702 tablet_bootstrap.cc:492] T 7244e26e54694bd48373e8634b4daf93 P 17eb0a665dc648d9a5ad2c62930861c4: No bootstrap required, opened a new log
I20240503 02:00:11.657116 15702 ts_tablet_manager.cc:1367] T 7244e26e54694bd48373e8634b4daf93 P 17eb0a665dc648d9a5ad2c62930861c4: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 02:00:11.657330 15703 tablet_bootstrap.cc:492] T 7244e26e54694bd48373e8634b4daf93 P 5751ea0f088543bcb141b400a6131a1f: No bootstrap required, opened a new log
I20240503 02:00:11.657413 15703 ts_tablet_manager.cc:1367] T 7244e26e54694bd48373e8634b4daf93 P 5751ea0f088543bcb141b400a6131a1f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:11.657557 15702 raft_consensus.cc:348] T 7244e26e54694bd48373e8634b4daf93 P 17eb0a665dc648d9a5ad2c62930861c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } } peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } }
I20240503 02:00:11.657641 15702 raft_consensus.cc:374] T 7244e26e54694bd48373e8634b4daf93 P 17eb0a665dc648d9a5ad2c62930861c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:11.657692 15702 raft_consensus.cc:729] T 7244e26e54694bd48373e8634b4daf93 P 17eb0a665dc648d9a5ad2c62930861c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17eb0a665dc648d9a5ad2c62930861c4, State: Initialized, Role: FOLLOWER
I20240503 02:00:11.657815 15702 consensus_queue.cc:260] T 7244e26e54694bd48373e8634b4daf93 P 17eb0a665dc648d9a5ad2c62930861c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } } peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } }
I20240503 02:00:11.657840 15703 raft_consensus.cc:348] T 7244e26e54694bd48373e8634b4daf93 P 5751ea0f088543bcb141b400a6131a1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } } peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } }
I20240503 02:00:11.657955 15703 raft_consensus.cc:374] T 7244e26e54694bd48373e8634b4daf93 P 5751ea0f088543bcb141b400a6131a1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:11.658000 15703 raft_consensus.cc:729] T 7244e26e54694bd48373e8634b4daf93 P 5751ea0f088543bcb141b400a6131a1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5751ea0f088543bcb141b400a6131a1f, State: Initialized, Role: FOLLOWER
I20240503 02:00:11.658100 15702 ts_tablet_manager.cc:1398] T 7244e26e54694bd48373e8634b4daf93 P 17eb0a665dc648d9a5ad2c62930861c4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:11.658124 15703 consensus_queue.cc:260] T 7244e26e54694bd48373e8634b4daf93 P 5751ea0f088543bcb141b400a6131a1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } } peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } }
I20240503 02:00:11.658267 15702 tablet_bootstrap.cc:492] T 595c6713bec0492882c1e58225dcce42 P 17eb0a665dc648d9a5ad2c62930861c4: Bootstrap starting.
I20240503 02:00:11.658391 15703 ts_tablet_manager.cc:1398] T 7244e26e54694bd48373e8634b4daf93 P 5751ea0f088543bcb141b400a6131a1f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:11.658564 15703 tablet_bootstrap.cc:492] T a567f7530d30404faa9ebe967138e0b4 P 5751ea0f088543bcb141b400a6131a1f: Bootstrap starting.
I20240503 02:00:11.659145 15702 tablet_bootstrap.cc:654] T 595c6713bec0492882c1e58225dcce42 P 17eb0a665dc648d9a5ad2c62930861c4: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:11.659457 15703 tablet_bootstrap.cc:654] T a567f7530d30404faa9ebe967138e0b4 P 5751ea0f088543bcb141b400a6131a1f: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:11.660609 15702 tablet_bootstrap.cc:492] T 595c6713bec0492882c1e58225dcce42 P 17eb0a665dc648d9a5ad2c62930861c4: No bootstrap required, opened a new log
I20240503 02:00:11.660809 15702 ts_tablet_manager.cc:1367] T 595c6713bec0492882c1e58225dcce42 P 17eb0a665dc648d9a5ad2c62930861c4: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240503 02:00:11.661237 15702 raft_consensus.cc:348] T 595c6713bec0492882c1e58225dcce42 P 17eb0a665dc648d9a5ad2c62930861c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } } peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } }
I20240503 02:00:11.661335 15702 raft_consensus.cc:374] T 595c6713bec0492882c1e58225dcce42 P 17eb0a665dc648d9a5ad2c62930861c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:11.661375 15702 raft_consensus.cc:729] T 595c6713bec0492882c1e58225dcce42 P 17eb0a665dc648d9a5ad2c62930861c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17eb0a665dc648d9a5ad2c62930861c4, State: Initialized, Role: FOLLOWER
I20240503 02:00:11.661414 15703 tablet_bootstrap.cc:492] T a567f7530d30404faa9ebe967138e0b4 P 5751ea0f088543bcb141b400a6131a1f: No bootstrap required, opened a new log
I20240503 02:00:11.661504 15703 ts_tablet_manager.cc:1367] T a567f7530d30404faa9ebe967138e0b4 P 5751ea0f088543bcb141b400a6131a1f: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 02:00:11.661513 15702 consensus_queue.cc:260] T 595c6713bec0492882c1e58225dcce42 P 17eb0a665dc648d9a5ad2c62930861c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } } peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } }
I20240503 02:00:11.661767 15702 ts_tablet_manager.cc:1398] T 595c6713bec0492882c1e58225dcce42 P 17eb0a665dc648d9a5ad2c62930861c4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:11.661932 15703 raft_consensus.cc:348] T a567f7530d30404faa9ebe967138e0b4 P 5751ea0f088543bcb141b400a6131a1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } } peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } }
I20240503 02:00:11.662039 15703 raft_consensus.cc:374] T a567f7530d30404faa9ebe967138e0b4 P 5751ea0f088543bcb141b400a6131a1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:11.662074 15703 raft_consensus.cc:729] T a567f7530d30404faa9ebe967138e0b4 P 5751ea0f088543bcb141b400a6131a1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5751ea0f088543bcb141b400a6131a1f, State: Initialized, Role: FOLLOWER
I20240503 02:00:11.662176 15703 consensus_queue.cc:260] T a567f7530d30404faa9ebe967138e0b4 P 5751ea0f088543bcb141b400a6131a1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } } peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } }
I20240503 02:00:11.662448 15703 ts_tablet_manager.cc:1398] T a567f7530d30404faa9ebe967138e0b4 P 5751ea0f088543bcb141b400a6131a1f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:11.737970 15647 raft_consensus.cc:482] T 595c6713bec0492882c1e58225dcce42 P 17eb0a665dc648d9a5ad2c62930861c4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:11.738168 15647 raft_consensus.cc:504] T 595c6713bec0492882c1e58225dcce42 P 17eb0a665dc648d9a5ad2c62930861c4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } } peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } }
I20240503 02:00:11.739581 15647 leader_election.cc:288] T 595c6713bec0492882c1e58225dcce42 P 17eb0a665dc648d9a5ad2c62930861c4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5751ea0f088543bcb141b400a6131a1f (127.0.99.129:46041)
I20240503 02:00:11.743697 15184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "595c6713bec0492882c1e58225dcce42" candidate_uuid: "17eb0a665dc648d9a5ad2c62930861c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5751ea0f088543bcb141b400a6131a1f" is_pre_election: true
I20240503 02:00:11.743969 15184 raft_consensus.cc:2454] T 595c6713bec0492882c1e58225dcce42 P 5751ea0f088543bcb141b400a6131a1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17eb0a665dc648d9a5ad2c62930861c4 in term 0.
I20240503 02:00:11.744369 15513 leader_election.cc:302] T 595c6713bec0492882c1e58225dcce42 P 17eb0a665dc648d9a5ad2c62930861c4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 17eb0a665dc648d9a5ad2c62930861c4, 5751ea0f088543bcb141b400a6131a1f; no voters:
I20240503 02:00:11.744616 15647 raft_consensus.cc:2781] T 595c6713bec0492882c1e58225dcce42 P 17eb0a665dc648d9a5ad2c62930861c4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:11.744709 15647 raft_consensus.cc:482] T 595c6713bec0492882c1e58225dcce42 P 17eb0a665dc648d9a5ad2c62930861c4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:11.744764 15647 raft_consensus.cc:3037] T 595c6713bec0492882c1e58225dcce42 P 17eb0a665dc648d9a5ad2c62930861c4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:11.745683 15647 raft_consensus.cc:504] T 595c6713bec0492882c1e58225dcce42 P 17eb0a665dc648d9a5ad2c62930861c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } } peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } }
I20240503 02:00:11.745993 15647 leader_election.cc:288] T 595c6713bec0492882c1e58225dcce42 P 17eb0a665dc648d9a5ad2c62930861c4 [CANDIDATE]: Term 1 election: Requested vote from peers 5751ea0f088543bcb141b400a6131a1f (127.0.99.129:46041)
I20240503 02:00:11.746352 15184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "595c6713bec0492882c1e58225dcce42" candidate_uuid: "17eb0a665dc648d9a5ad2c62930861c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5751ea0f088543bcb141b400a6131a1f"
I20240503 02:00:11.746460 15184 raft_consensus.cc:3037] T 595c6713bec0492882c1e58225dcce42 P 5751ea0f088543bcb141b400a6131a1f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:11.747473 15184 raft_consensus.cc:2454] T 595c6713bec0492882c1e58225dcce42 P 5751ea0f088543bcb141b400a6131a1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17eb0a665dc648d9a5ad2c62930861c4 in term 1.
I20240503 02:00:11.747774 15513 leader_election.cc:302] T 595c6713bec0492882c1e58225dcce42 P 17eb0a665dc648d9a5ad2c62930861c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 17eb0a665dc648d9a5ad2c62930861c4, 5751ea0f088543bcb141b400a6131a1f; no voters:
I20240503 02:00:11.747936 15647 raft_consensus.cc:2781] T 595c6713bec0492882c1e58225dcce42 P 17eb0a665dc648d9a5ad2c62930861c4 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:11.748176 15647 raft_consensus.cc:686] T 595c6713bec0492882c1e58225dcce42 P 17eb0a665dc648d9a5ad2c62930861c4 [term 1 LEADER]: Becoming Leader. State: Replica: 17eb0a665dc648d9a5ad2c62930861c4, State: Running, Role: LEADER
I20240503 02:00:11.748497 15647 consensus_queue.cc:237] T 595c6713bec0492882c1e58225dcce42 P 17eb0a665dc648d9a5ad2c62930861c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } } peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } }
I20240503 02:00:11.750792 15041 catalog_manager.cc:5484] T 595c6713bec0492882c1e58225dcce42 P 17eb0a665dc648d9a5ad2c62930861c4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 17eb0a665dc648d9a5ad2c62930861c4 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "17eb0a665dc648d9a5ad2c62930861c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:11.826824 15649 raft_consensus.cc:482] T 7244e26e54694bd48373e8634b4daf93 P 5751ea0f088543bcb141b400a6131a1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:11.827032 15649 raft_consensus.cc:504] T 7244e26e54694bd48373e8634b4daf93 P 5751ea0f088543bcb141b400a6131a1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } } peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } }
I20240503 02:00:11.827534 15649 leader_election.cc:288] T 7244e26e54694bd48373e8634b4daf93 P 5751ea0f088543bcb141b400a6131a1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 17eb0a665dc648d9a5ad2c62930861c4 (127.0.99.132:33559)
I20240503 02:00:11.828210 15580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7244e26e54694bd48373e8634b4daf93" candidate_uuid: "5751ea0f088543bcb141b400a6131a1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17eb0a665dc648d9a5ad2c62930861c4" is_pre_election: true
I20240503 02:00:11.828406 15580 raft_consensus.cc:2454] T 7244e26e54694bd48373e8634b4daf93 P 17eb0a665dc648d9a5ad2c62930861c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5751ea0f088543bcb141b400a6131a1f in term 0.
I20240503 02:00:11.828771 15120 leader_election.cc:302] T 7244e26e54694bd48373e8634b4daf93 P 5751ea0f088543bcb141b400a6131a1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 17eb0a665dc648d9a5ad2c62930861c4, 5751ea0f088543bcb141b400a6131a1f; no voters:
I20240503 02:00:11.829000 15649 raft_consensus.cc:2781] T 7244e26e54694bd48373e8634b4daf93 P 5751ea0f088543bcb141b400a6131a1f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:11.829078 15649 raft_consensus.cc:482] T 7244e26e54694bd48373e8634b4daf93 P 5751ea0f088543bcb141b400a6131a1f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:11.829108 15649 raft_consensus.cc:3037] T 7244e26e54694bd48373e8634b4daf93 P 5751ea0f088543bcb141b400a6131a1f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:11.829979 15649 raft_consensus.cc:504] T 7244e26e54694bd48373e8634b4daf93 P 5751ea0f088543bcb141b400a6131a1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } } peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } }
I20240503 02:00:11.830353 15649 leader_election.cc:288] T 7244e26e54694bd48373e8634b4daf93 P 5751ea0f088543bcb141b400a6131a1f [CANDIDATE]: Term 1 election: Requested vote from peers 17eb0a665dc648d9a5ad2c62930861c4 (127.0.99.132:33559)
I20240503 02:00:11.830710 15580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7244e26e54694bd48373e8634b4daf93" candidate_uuid: "5751ea0f088543bcb141b400a6131a1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17eb0a665dc648d9a5ad2c62930861c4"
I20240503 02:00:11.830845 15580 raft_consensus.cc:3037] T 7244e26e54694bd48373e8634b4daf93 P 17eb0a665dc648d9a5ad2c62930861c4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:11.831568 15580 raft_consensus.cc:2454] T 7244e26e54694bd48373e8634b4daf93 P 17eb0a665dc648d9a5ad2c62930861c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5751ea0f088543bcb141b400a6131a1f in term 1.
I20240503 02:00:11.831885 15120 leader_election.cc:302] T 7244e26e54694bd48373e8634b4daf93 P 5751ea0f088543bcb141b400a6131a1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 17eb0a665dc648d9a5ad2c62930861c4, 5751ea0f088543bcb141b400a6131a1f; no voters:
I20240503 02:00:11.832087 15649 raft_consensus.cc:2781] T 7244e26e54694bd48373e8634b4daf93 P 5751ea0f088543bcb141b400a6131a1f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:11.832199 15649 raft_consensus.cc:686] T 7244e26e54694bd48373e8634b4daf93 P 5751ea0f088543bcb141b400a6131a1f [term 1 LEADER]: Becoming Leader. State: Replica: 5751ea0f088543bcb141b400a6131a1f, State: Running, Role: LEADER
I20240503 02:00:11.832360 15649 consensus_queue.cc:237] T 7244e26e54694bd48373e8634b4daf93 P 5751ea0f088543bcb141b400a6131a1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } } peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } }
I20240503 02:00:11.834208 15041 catalog_manager.cc:5484] T 7244e26e54694bd48373e8634b4daf93 P 5751ea0f088543bcb141b400a6131a1f reported cstate change: term changed from 0 to 1, leader changed from <none> to 5751ea0f088543bcb141b400a6131a1f (127.0.99.129). New cstate: current_term: 1 leader_uuid: "5751ea0f088543bcb141b400a6131a1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:11.907759 15647 raft_consensus.cc:482] T a567f7530d30404faa9ebe967138e0b4 P 17eb0a665dc648d9a5ad2c62930861c4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:11.907945 15647 raft_consensus.cc:504] T a567f7530d30404faa9ebe967138e0b4 P 17eb0a665dc648d9a5ad2c62930861c4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } } peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } }
I20240503 02:00:11.908469 15647 leader_election.cc:288] T a567f7530d30404faa9ebe967138e0b4 P 17eb0a665dc648d9a5ad2c62930861c4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5751ea0f088543bcb141b400a6131a1f (127.0.99.129:46041)
I20240503 02:00:11.909107 15184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a567f7530d30404faa9ebe967138e0b4" candidate_uuid: "17eb0a665dc648d9a5ad2c62930861c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5751ea0f088543bcb141b400a6131a1f" is_pre_election: true
I20240503 02:00:11.909262 15184 raft_consensus.cc:2454] T a567f7530d30404faa9ebe967138e0b4 P 5751ea0f088543bcb141b400a6131a1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17eb0a665dc648d9a5ad2c62930861c4 in term 0.
I20240503 02:00:11.909574 15513 leader_election.cc:302] T a567f7530d30404faa9ebe967138e0b4 P 17eb0a665dc648d9a5ad2c62930861c4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 17eb0a665dc648d9a5ad2c62930861c4, 5751ea0f088543bcb141b400a6131a1f; no voters:
I20240503 02:00:11.909739 15647 raft_consensus.cc:2781] T a567f7530d30404faa9ebe967138e0b4 P 17eb0a665dc648d9a5ad2c62930861c4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:11.909797 15647 raft_consensus.cc:482] T a567f7530d30404faa9ebe967138e0b4 P 17eb0a665dc648d9a5ad2c62930861c4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:11.909823 15647 raft_consensus.cc:3037] T a567f7530d30404faa9ebe967138e0b4 P 17eb0a665dc648d9a5ad2c62930861c4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:11.910764 15647 raft_consensus.cc:504] T a567f7530d30404faa9ebe967138e0b4 P 17eb0a665dc648d9a5ad2c62930861c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } } peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } }
I20240503 02:00:11.911017 15647 leader_election.cc:288] T a567f7530d30404faa9ebe967138e0b4 P 17eb0a665dc648d9a5ad2c62930861c4 [CANDIDATE]: Term 1 election: Requested vote from peers 5751ea0f088543bcb141b400a6131a1f (127.0.99.129:46041)
I20240503 02:00:11.911479 15184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a567f7530d30404faa9ebe967138e0b4" candidate_uuid: "17eb0a665dc648d9a5ad2c62930861c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5751ea0f088543bcb141b400a6131a1f"
I20240503 02:00:11.911674 15184 raft_consensus.cc:3037] T a567f7530d30404faa9ebe967138e0b4 P 5751ea0f088543bcb141b400a6131a1f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:11.912539 15184 raft_consensus.cc:2454] T a567f7530d30404faa9ebe967138e0b4 P 5751ea0f088543bcb141b400a6131a1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17eb0a665dc648d9a5ad2c62930861c4 in term 1.
I20240503 02:00:11.912890 15513 leader_election.cc:302] T a567f7530d30404faa9ebe967138e0b4 P 17eb0a665dc648d9a5ad2c62930861c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 17eb0a665dc648d9a5ad2c62930861c4, 5751ea0f088543bcb141b400a6131a1f; no voters:
I20240503 02:00:11.913038 15647 raft_consensus.cc:2781] T a567f7530d30404faa9ebe967138e0b4 P 17eb0a665dc648d9a5ad2c62930861c4 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:11.913121 15647 raft_consensus.cc:686] T a567f7530d30404faa9ebe967138e0b4 P 17eb0a665dc648d9a5ad2c62930861c4 [term 1 LEADER]: Becoming Leader. State: Replica: 17eb0a665dc648d9a5ad2c62930861c4, State: Running, Role: LEADER
I20240503 02:00:11.913275 15647 consensus_queue.cc:237] T a567f7530d30404faa9ebe967138e0b4 P 17eb0a665dc648d9a5ad2c62930861c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } } peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } }
I20240503 02:00:11.915072 15041 catalog_manager.cc:5484] T a567f7530d30404faa9ebe967138e0b4 P 17eb0a665dc648d9a5ad2c62930861c4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 17eb0a665dc648d9a5ad2c62930861c4 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "17eb0a665dc648d9a5ad2c62930861c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } health_report { overall_health: UNKNOWN } } }
W20240503 02:00:12.059394 15715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:00:12.059713 15715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:00:12.062877 15715 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240503 02:00:12.143623 15580 raft_consensus.cc:1261] T 7244e26e54694bd48373e8634b4daf93 P 17eb0a665dc648d9a5ad2c62930861c4 [term 1 FOLLOWER]: Refusing update from remote peer 5751ea0f088543bcb141b400a6131a1f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:12.144060 15649 consensus_queue.cc:1035] T 7244e26e54694bd48373e8634b4daf93 P 5751ea0f088543bcb141b400a6131a1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:12.145547 15184 raft_consensus.cc:1261] T 595c6713bec0492882c1e58225dcce42 P 5751ea0f088543bcb141b400a6131a1f [term 1 FOLLOWER]: Refusing update from remote peer 17eb0a665dc648d9a5ad2c62930861c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:12.146884 15184 raft_consensus.cc:1261] T a567f7530d30404faa9ebe967138e0b4 P 5751ea0f088543bcb141b400a6131a1f [term 1 FOLLOWER]: Refusing update from remote peer 17eb0a665dc648d9a5ad2c62930861c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:12.147055 15711 consensus_queue.cc:1035] T 595c6713bec0492882c1e58225dcce42 P 17eb0a665dc648d9a5ad2c62930861c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:12.148705 15647 consensus_queue.cc:1035] T a567f7530d30404faa9ebe967138e0b4 P 17eb0a665dc648d9a5ad2c62930861c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:12.156577 15755 mvcc.cc:205] Tried to move back new op lower bound from 7023417803344658432 to 7023417802397630464. Current Snapshot: MvccSnapshot[applied={T|T < 7023417803344658432}]
INSERT report
rows total: 2000
time total: 84.345 ms
time per row: 0.0421725 ms
I20240503 02:00:12.193799 15038 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59506:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240503 02:00:12.194453 15038 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_2 in case of a server failure: 3 tablet servers would be needed, 2 are available. Consider bringing up more tablet servers.
I20240503 02:00:12.199751 15162 tablet_service.cc:1467] Processing CreateTablet for tablet a14de0ce61c74ac5ae8bdcfb403e7e5d (DEFAULT_TABLE table=rebalance_test_table_2 [id=1fdb6fbbdfc14fe7bf1064da99b35275]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:12.199839 15559 tablet_service.cc:1467] Processing CreateTablet for tablet a14de0ce61c74ac5ae8bdcfb403e7e5d (DEFAULT_TABLE table=rebalance_test_table_2 [id=1fdb6fbbdfc14fe7bf1064da99b35275]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:12.200145 15162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a14de0ce61c74ac5ae8bdcfb403e7e5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:12.200187 15559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a14de0ce61c74ac5ae8bdcfb403e7e5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:12.200309 15558 tablet_service.cc:1467] Processing CreateTablet for tablet f357df7d66a646fc817b19b28306a5a0 (DEFAULT_TABLE table=rebalance_test_table_2 [id=1fdb6fbbdfc14fe7bf1064da99b35275]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:12.200359 15560 tablet_service.cc:1467] Processing CreateTablet for tablet f72aa769aa8f465ba4469dfc34ae35c1 (DEFAULT_TABLE table=rebalance_test_table_2 [id=1fdb6fbbdfc14fe7bf1064da99b35275]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:12.200534 15558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f357df7d66a646fc817b19b28306a5a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:12.200922 15164 tablet_service.cc:1467] Processing CreateTablet for tablet f72aa769aa8f465ba4469dfc34ae35c1 (DEFAULT_TABLE table=rebalance_test_table_2 [id=1fdb6fbbdfc14fe7bf1064da99b35275]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:12.201130 15164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f72aa769aa8f465ba4469dfc34ae35c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:12.202579 15764 tablet_bootstrap.cc:492] T a14de0ce61c74ac5ae8bdcfb403e7e5d P 5751ea0f088543bcb141b400a6131a1f: Bootstrap starting.
I20240503 02:00:12.202839 15765 tablet_bootstrap.cc:492] T f357df7d66a646fc817b19b28306a5a0 P 17eb0a665dc648d9a5ad2c62930861c4: Bootstrap starting.
I20240503 02:00:12.203354 15560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f72aa769aa8f465ba4469dfc34ae35c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:12.203544 15764 tablet_bootstrap.cc:654] T a14de0ce61c74ac5ae8bdcfb403e7e5d P 5751ea0f088543bcb141b400a6131a1f: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:12.203964 15765 tablet_bootstrap.cc:654] T f357df7d66a646fc817b19b28306a5a0 P 17eb0a665dc648d9a5ad2c62930861c4: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:12.204821 15764 tablet_bootstrap.cc:492] T a14de0ce61c74ac5ae8bdcfb403e7e5d P 5751ea0f088543bcb141b400a6131a1f: No bootstrap required, opened a new log
I20240503 02:00:12.204979 15764 ts_tablet_manager.cc:1367] T a14de0ce61c74ac5ae8bdcfb403e7e5d P 5751ea0f088543bcb141b400a6131a1f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:12.205138 15164 tablet_service.cc:1467] Processing CreateTablet for tablet f357df7d66a646fc817b19b28306a5a0 (DEFAULT_TABLE table=rebalance_test_table_2 [id=1fdb6fbbdfc14fe7bf1064da99b35275]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:12.205394 15164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f357df7d66a646fc817b19b28306a5a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:12.205487 15764 raft_consensus.cc:348] T a14de0ce61c74ac5ae8bdcfb403e7e5d P 5751ea0f088543bcb141b400a6131a1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } } peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } }
I20240503 02:00:12.205698 15764 raft_consensus.cc:374] T a14de0ce61c74ac5ae8bdcfb403e7e5d P 5751ea0f088543bcb141b400a6131a1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:12.205821 15764 raft_consensus.cc:729] T a14de0ce61c74ac5ae8bdcfb403e7e5d P 5751ea0f088543bcb141b400a6131a1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5751ea0f088543bcb141b400a6131a1f, State: Initialized, Role: FOLLOWER
I20240503 02:00:12.206032 15764 consensus_queue.cc:260] T a14de0ce61c74ac5ae8bdcfb403e7e5d P 5751ea0f088543bcb141b400a6131a1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } } peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } }
I20240503 02:00:12.206286 15765 tablet_bootstrap.cc:492] T f357df7d66a646fc817b19b28306a5a0 P 17eb0a665dc648d9a5ad2c62930861c4: No bootstrap required, opened a new log
I20240503 02:00:12.206375 15765 ts_tablet_manager.cc:1367] T f357df7d66a646fc817b19b28306a5a0 P 17eb0a665dc648d9a5ad2c62930861c4: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:00:12.206375 15764 ts_tablet_manager.cc:1398] T a14de0ce61c74ac5ae8bdcfb403e7e5d P 5751ea0f088543bcb141b400a6131a1f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:12.206820 15765 raft_consensus.cc:348] T f357df7d66a646fc817b19b28306a5a0 P 17eb0a665dc648d9a5ad2c62930861c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } } peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } }
I20240503 02:00:12.206933 15765 raft_consensus.cc:374] T f357df7d66a646fc817b19b28306a5a0 P 17eb0a665dc648d9a5ad2c62930861c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:12.206971 15765 raft_consensus.cc:729] T f357df7d66a646fc817b19b28306a5a0 P 17eb0a665dc648d9a5ad2c62930861c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17eb0a665dc648d9a5ad2c62930861c4, State: Initialized, Role: FOLLOWER
I20240503 02:00:12.207173 15765 consensus_queue.cc:260] T f357df7d66a646fc817b19b28306a5a0 P 17eb0a665dc648d9a5ad2c62930861c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } } peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } }
I20240503 02:00:12.207267 15764 tablet_bootstrap.cc:492] T f72aa769aa8f465ba4469dfc34ae35c1 P 5751ea0f088543bcb141b400a6131a1f: Bootstrap starting.
I20240503 02:00:12.207623 15765 ts_tablet_manager.cc:1398] T f357df7d66a646fc817b19b28306a5a0 P 17eb0a665dc648d9a5ad2c62930861c4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:12.207849 15765 tablet_bootstrap.cc:492] T a14de0ce61c74ac5ae8bdcfb403e7e5d P 17eb0a665dc648d9a5ad2c62930861c4: Bootstrap starting.
I20240503 02:00:12.208796 15764 tablet_bootstrap.cc:654] T f72aa769aa8f465ba4469dfc34ae35c1 P 5751ea0f088543bcb141b400a6131a1f: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:12.208791 15765 tablet_bootstrap.cc:654] T a14de0ce61c74ac5ae8bdcfb403e7e5d P 17eb0a665dc648d9a5ad2c62930861c4: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:12.209965 15765 tablet_bootstrap.cc:492] T a14de0ce61c74ac5ae8bdcfb403e7e5d P 17eb0a665dc648d9a5ad2c62930861c4: No bootstrap required, opened a new log
I20240503 02:00:12.210080 15765 ts_tablet_manager.cc:1367] T a14de0ce61c74ac5ae8bdcfb403e7e5d P 17eb0a665dc648d9a5ad2c62930861c4: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 02:00:12.210275 15764 tablet_bootstrap.cc:492] T f72aa769aa8f465ba4469dfc34ae35c1 P 5751ea0f088543bcb141b400a6131a1f: No bootstrap required, opened a new log
I20240503 02:00:12.210361 15764 ts_tablet_manager.cc:1367] T f72aa769aa8f465ba4469dfc34ae35c1 P 5751ea0f088543bcb141b400a6131a1f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:12.210485 15765 raft_consensus.cc:348] T a14de0ce61c74ac5ae8bdcfb403e7e5d P 17eb0a665dc648d9a5ad2c62930861c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } } peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } }
I20240503 02:00:12.210561 15765 raft_consensus.cc:374] T a14de0ce61c74ac5ae8bdcfb403e7e5d P 17eb0a665dc648d9a5ad2c62930861c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:12.210595 15765 raft_consensus.cc:729] T a14de0ce61c74ac5ae8bdcfb403e7e5d P 17eb0a665dc648d9a5ad2c62930861c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17eb0a665dc648d9a5ad2c62930861c4, State: Initialized, Role: FOLLOWER
I20240503 02:00:12.210703 15765 consensus_queue.cc:260] T a14de0ce61c74ac5ae8bdcfb403e7e5d P 17eb0a665dc648d9a5ad2c62930861c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } } peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } }
I20240503 02:00:12.210839 15764 raft_consensus.cc:348] T f72aa769aa8f465ba4469dfc34ae35c1 P 5751ea0f088543bcb141b400a6131a1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } } peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } }
I20240503 02:00:12.210933 15765 ts_tablet_manager.cc:1398] T a14de0ce61c74ac5ae8bdcfb403e7e5d P 17eb0a665dc648d9a5ad2c62930861c4: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:12.210953 15764 raft_consensus.cc:374] T f72aa769aa8f465ba4469dfc34ae35c1 P 5751ea0f088543bcb141b400a6131a1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:12.211016 15764 raft_consensus.cc:729] T f72aa769aa8f465ba4469dfc34ae35c1 P 5751ea0f088543bcb141b400a6131a1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5751ea0f088543bcb141b400a6131a1f, State: Initialized, Role: FOLLOWER
I20240503 02:00:12.211120 15765 tablet_bootstrap.cc:492] T f72aa769aa8f465ba4469dfc34ae35c1 P 17eb0a665dc648d9a5ad2c62930861c4: Bootstrap starting.
I20240503 02:00:12.211145 15764 consensus_queue.cc:260] T f72aa769aa8f465ba4469dfc34ae35c1 P 5751ea0f088543bcb141b400a6131a1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } } peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } }
I20240503 02:00:12.211835 15764 ts_tablet_manager.cc:1398] T f72aa769aa8f465ba4469dfc34ae35c1 P 5751ea0f088543bcb141b400a6131a1f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:12.211941 15765 tablet_bootstrap.cc:654] T f72aa769aa8f465ba4469dfc34ae35c1 P 17eb0a665dc648d9a5ad2c62930861c4: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:12.212036 15764 tablet_bootstrap.cc:492] T f357df7d66a646fc817b19b28306a5a0 P 5751ea0f088543bcb141b400a6131a1f: Bootstrap starting.
I20240503 02:00:12.212945 15764 tablet_bootstrap.cc:654] T f357df7d66a646fc817b19b28306a5a0 P 5751ea0f088543bcb141b400a6131a1f: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:12.213722 15765 tablet_bootstrap.cc:492] T f72aa769aa8f465ba4469dfc34ae35c1 P 17eb0a665dc648d9a5ad2c62930861c4: No bootstrap required, opened a new log
I20240503 02:00:12.213852 15765 ts_tablet_manager.cc:1367] T f72aa769aa8f465ba4469dfc34ae35c1 P 17eb0a665dc648d9a5ad2c62930861c4: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 02:00:12.214270 15765 raft_consensus.cc:348] T f72aa769aa8f465ba4469dfc34ae35c1 P 17eb0a665dc648d9a5ad2c62930861c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } } peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } }
I20240503 02:00:12.214355 15765 raft_consensus.cc:374] T f72aa769aa8f465ba4469dfc34ae35c1 P 17eb0a665dc648d9a5ad2c62930861c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:12.214390 15765 raft_consensus.cc:729] T f72aa769aa8f465ba4469dfc34ae35c1 P 17eb0a665dc648d9a5ad2c62930861c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17eb0a665dc648d9a5ad2c62930861c4, State: Initialized, Role: FOLLOWER
I20240503 02:00:12.214514 15765 consensus_queue.cc:260] T f72aa769aa8f465ba4469dfc34ae35c1 P 17eb0a665dc648d9a5ad2c62930861c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } } peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } }
I20240503 02:00:12.214751 15765 ts_tablet_manager.cc:1398] T f72aa769aa8f465ba4469dfc34ae35c1 P 17eb0a665dc648d9a5ad2c62930861c4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:12.216215 15764 tablet_bootstrap.cc:492] T f357df7d66a646fc817b19b28306a5a0 P 5751ea0f088543bcb141b400a6131a1f: No bootstrap required, opened a new log
I20240503 02:00:12.216312 15764 ts_tablet_manager.cc:1367] T f357df7d66a646fc817b19b28306a5a0 P 5751ea0f088543bcb141b400a6131a1f: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:00:12.216761 15764 raft_consensus.cc:348] T f357df7d66a646fc817b19b28306a5a0 P 5751ea0f088543bcb141b400a6131a1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } } peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } }
I20240503 02:00:12.216862 15764 raft_consensus.cc:374] T f357df7d66a646fc817b19b28306a5a0 P 5751ea0f088543bcb141b400a6131a1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:12.216894 15764 raft_consensus.cc:729] T f357df7d66a646fc817b19b28306a5a0 P 5751ea0f088543bcb141b400a6131a1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5751ea0f088543bcb141b400a6131a1f, State: Initialized, Role: FOLLOWER
I20240503 02:00:12.217043 15764 consensus_queue.cc:260] T f357df7d66a646fc817b19b28306a5a0 P 5751ea0f088543bcb141b400a6131a1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } } peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } }
I20240503 02:00:12.217375 15764 ts_tablet_manager.cc:1398] T f357df7d66a646fc817b19b28306a5a0 P 5751ea0f088543bcb141b400a6131a1f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:12.271934 15748 raft_consensus.cc:482] T f357df7d66a646fc817b19b28306a5a0 P 17eb0a665dc648d9a5ad2c62930861c4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:12.272070 15748 raft_consensus.cc:504] T f357df7d66a646fc817b19b28306a5a0 P 17eb0a665dc648d9a5ad2c62930861c4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } } peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } }
I20240503 02:00:12.272457 15748 leader_election.cc:288] T f357df7d66a646fc817b19b28306a5a0 P 17eb0a665dc648d9a5ad2c62930861c4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5751ea0f088543bcb141b400a6131a1f (127.0.99.129:46041)
I20240503 02:00:12.272950 15184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f357df7d66a646fc817b19b28306a5a0" candidate_uuid: "17eb0a665dc648d9a5ad2c62930861c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5751ea0f088543bcb141b400a6131a1f" is_pre_election: true
I20240503 02:00:12.273118 15184 raft_consensus.cc:2454] T f357df7d66a646fc817b19b28306a5a0 P 5751ea0f088543bcb141b400a6131a1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17eb0a665dc648d9a5ad2c62930861c4 in term 0.
I20240503 02:00:12.273447 15513 leader_election.cc:302] T f357df7d66a646fc817b19b28306a5a0 P 17eb0a665dc648d9a5ad2c62930861c4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 17eb0a665dc648d9a5ad2c62930861c4, 5751ea0f088543bcb141b400a6131a1f; no voters:
I20240503 02:00:12.273650 15748 raft_consensus.cc:2781] T f357df7d66a646fc817b19b28306a5a0 P 17eb0a665dc648d9a5ad2c62930861c4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:12.273711 15748 raft_consensus.cc:482] T f357df7d66a646fc817b19b28306a5a0 P 17eb0a665dc648d9a5ad2c62930861c4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:12.273739 15748 raft_consensus.cc:3037] T f357df7d66a646fc817b19b28306a5a0 P 17eb0a665dc648d9a5ad2c62930861c4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:12.274592 15748 raft_consensus.cc:504] T f357df7d66a646fc817b19b28306a5a0 P 17eb0a665dc648d9a5ad2c62930861c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } } peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } }
I20240503 02:00:12.274847 15748 leader_election.cc:288] T f357df7d66a646fc817b19b28306a5a0 P 17eb0a665dc648d9a5ad2c62930861c4 [CANDIDATE]: Term 1 election: Requested vote from peers 5751ea0f088543bcb141b400a6131a1f (127.0.99.129:46041)
I20240503 02:00:12.275210 15184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f357df7d66a646fc817b19b28306a5a0" candidate_uuid: "17eb0a665dc648d9a5ad2c62930861c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5751ea0f088543bcb141b400a6131a1f"
I20240503 02:00:12.275312 15184 raft_consensus.cc:3037] T f357df7d66a646fc817b19b28306a5a0 P 5751ea0f088543bcb141b400a6131a1f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:12.276072 15184 raft_consensus.cc:2454] T f357df7d66a646fc817b19b28306a5a0 P 5751ea0f088543bcb141b400a6131a1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17eb0a665dc648d9a5ad2c62930861c4 in term 1.
I20240503 02:00:12.276345 15513 leader_election.cc:302] T f357df7d66a646fc817b19b28306a5a0 P 17eb0a665dc648d9a5ad2c62930861c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 17eb0a665dc648d9a5ad2c62930861c4, 5751ea0f088543bcb141b400a6131a1f; no voters:
I20240503 02:00:12.276507 15748 raft_consensus.cc:2781] T f357df7d66a646fc817b19b28306a5a0 P 17eb0a665dc648d9a5ad2c62930861c4 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:12.276583 15748 raft_consensus.cc:686] T f357df7d66a646fc817b19b28306a5a0 P 17eb0a665dc648d9a5ad2c62930861c4 [term 1 LEADER]: Becoming Leader. State: Replica: 17eb0a665dc648d9a5ad2c62930861c4, State: Running, Role: LEADER
I20240503 02:00:12.276731 15748 consensus_queue.cc:237] T f357df7d66a646fc817b19b28306a5a0 P 17eb0a665dc648d9a5ad2c62930861c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } } peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } }
I20240503 02:00:12.278206 15034 catalog_manager.cc:5484] T f357df7d66a646fc817b19b28306a5a0 P 17eb0a665dc648d9a5ad2c62930861c4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 17eb0a665dc648d9a5ad2c62930861c4 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "17eb0a665dc648d9a5ad2c62930861c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:12.346544 15748 raft_consensus.cc:482] T a14de0ce61c74ac5ae8bdcfb403e7e5d P 17eb0a665dc648d9a5ad2c62930861c4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:12.346688 15748 raft_consensus.cc:504] T a14de0ce61c74ac5ae8bdcfb403e7e5d P 17eb0a665dc648d9a5ad2c62930861c4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } } peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } }
I20240503 02:00:12.347079 15748 leader_election.cc:288] T a14de0ce61c74ac5ae8bdcfb403e7e5d P 17eb0a665dc648d9a5ad2c62930861c4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5751ea0f088543bcb141b400a6131a1f (127.0.99.129:46041)
I20240503 02:00:12.347586 15184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a14de0ce61c74ac5ae8bdcfb403e7e5d" candidate_uuid: "17eb0a665dc648d9a5ad2c62930861c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5751ea0f088543bcb141b400a6131a1f" is_pre_election: true
I20240503 02:00:12.347755 15184 raft_consensus.cc:2454] T a14de0ce61c74ac5ae8bdcfb403e7e5d P 5751ea0f088543bcb141b400a6131a1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17eb0a665dc648d9a5ad2c62930861c4 in term 0.
I20240503 02:00:12.348080 15513 leader_election.cc:302] T a14de0ce61c74ac5ae8bdcfb403e7e5d P 17eb0a665dc648d9a5ad2c62930861c4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 17eb0a665dc648d9a5ad2c62930861c4, 5751ea0f088543bcb141b400a6131a1f; no voters:
I20240503 02:00:12.348238 15748 raft_consensus.cc:2781] T a14de0ce61c74ac5ae8bdcfb403e7e5d P 17eb0a665dc648d9a5ad2c62930861c4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:12.348299 15748 raft_consensus.cc:482] T a14de0ce61c74ac5ae8bdcfb403e7e5d P 17eb0a665dc648d9a5ad2c62930861c4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:12.348347 15748 raft_consensus.cc:3037] T a14de0ce61c74ac5ae8bdcfb403e7e5d P 17eb0a665dc648d9a5ad2c62930861c4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:12.349323 15748 raft_consensus.cc:504] T a14de0ce61c74ac5ae8bdcfb403e7e5d P 17eb0a665dc648d9a5ad2c62930861c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } } peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } }
I20240503 02:00:12.349573 15748 leader_election.cc:288] T a14de0ce61c74ac5ae8bdcfb403e7e5d P 17eb0a665dc648d9a5ad2c62930861c4 [CANDIDATE]: Term 1 election: Requested vote from peers 5751ea0f088543bcb141b400a6131a1f (127.0.99.129:46041)
I20240503 02:00:12.349937 15184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a14de0ce61c74ac5ae8bdcfb403e7e5d" candidate_uuid: "17eb0a665dc648d9a5ad2c62930861c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5751ea0f088543bcb141b400a6131a1f"
I20240503 02:00:12.350040 15184 raft_consensus.cc:3037] T a14de0ce61c74ac5ae8bdcfb403e7e5d P 5751ea0f088543bcb141b400a6131a1f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:12.350800 15184 raft_consensus.cc:2454] T a14de0ce61c74ac5ae8bdcfb403e7e5d P 5751ea0f088543bcb141b400a6131a1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17eb0a665dc648d9a5ad2c62930861c4 in term 1.
I20240503 02:00:12.351053 15513 leader_election.cc:302] T a14de0ce61c74ac5ae8bdcfb403e7e5d P 17eb0a665dc648d9a5ad2c62930861c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 17eb0a665dc648d9a5ad2c62930861c4, 5751ea0f088543bcb141b400a6131a1f; no voters:
I20240503 02:00:12.351219 15748 raft_consensus.cc:2781] T a14de0ce61c74ac5ae8bdcfb403e7e5d P 17eb0a665dc648d9a5ad2c62930861c4 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:12.351310 15748 raft_consensus.cc:686] T a14de0ce61c74ac5ae8bdcfb403e7e5d P 17eb0a665dc648d9a5ad2c62930861c4 [term 1 LEADER]: Becoming Leader. State: Replica: 17eb0a665dc648d9a5ad2c62930861c4, State: Running, Role: LEADER
I20240503 02:00:12.351464 15748 consensus_queue.cc:237] T a14de0ce61c74ac5ae8bdcfb403e7e5d P 17eb0a665dc648d9a5ad2c62930861c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } } peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } }
I20240503 02:00:12.353080 15034 catalog_manager.cc:5484] T a14de0ce61c74ac5ae8bdcfb403e7e5d P 17eb0a665dc648d9a5ad2c62930861c4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 17eb0a665dc648d9a5ad2c62930861c4 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "17eb0a665dc648d9a5ad2c62930861c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:12.393872 15763 raft_consensus.cc:482] T f72aa769aa8f465ba4469dfc34ae35c1 P 5751ea0f088543bcb141b400a6131a1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:12.394013 15763 raft_consensus.cc:504] T f72aa769aa8f465ba4469dfc34ae35c1 P 5751ea0f088543bcb141b400a6131a1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } } peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } }
I20240503 02:00:12.394464 15763 leader_election.cc:288] T f72aa769aa8f465ba4469dfc34ae35c1 P 5751ea0f088543bcb141b400a6131a1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 17eb0a665dc648d9a5ad2c62930861c4 (127.0.99.132:33559)
I20240503 02:00:12.394882 15580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f72aa769aa8f465ba4469dfc34ae35c1" candidate_uuid: "5751ea0f088543bcb141b400a6131a1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17eb0a665dc648d9a5ad2c62930861c4" is_pre_election: true
I20240503 02:00:12.395010 15580 raft_consensus.cc:2454] T f72aa769aa8f465ba4469dfc34ae35c1 P 17eb0a665dc648d9a5ad2c62930861c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5751ea0f088543bcb141b400a6131a1f in term 0.
I20240503 02:00:12.395327 15120 leader_election.cc:302] T f72aa769aa8f465ba4469dfc34ae35c1 P 5751ea0f088543bcb141b400a6131a1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 17eb0a665dc648d9a5ad2c62930861c4, 5751ea0f088543bcb141b400a6131a1f; no voters:
I20240503 02:00:12.395546 15763 raft_consensus.cc:2781] T f72aa769aa8f465ba4469dfc34ae35c1 P 5751ea0f088543bcb141b400a6131a1f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:12.395604 15763 raft_consensus.cc:482] T f72aa769aa8f465ba4469dfc34ae35c1 P 5751ea0f088543bcb141b400a6131a1f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:12.395643 15763 raft_consensus.cc:3037] T f72aa769aa8f465ba4469dfc34ae35c1 P 5751ea0f088543bcb141b400a6131a1f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:12.396533 15763 raft_consensus.cc:504] T f72aa769aa8f465ba4469dfc34ae35c1 P 5751ea0f088543bcb141b400a6131a1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } } peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } }
I20240503 02:00:12.396797 15763 leader_election.cc:288] T f72aa769aa8f465ba4469dfc34ae35c1 P 5751ea0f088543bcb141b400a6131a1f [CANDIDATE]: Term 1 election: Requested vote from peers 17eb0a665dc648d9a5ad2c62930861c4 (127.0.99.132:33559)
I20240503 02:00:12.397163 15580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f72aa769aa8f465ba4469dfc34ae35c1" candidate_uuid: "5751ea0f088543bcb141b400a6131a1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17eb0a665dc648d9a5ad2c62930861c4"
I20240503 02:00:12.397254 15580 raft_consensus.cc:3037] T f72aa769aa8f465ba4469dfc34ae35c1 P 17eb0a665dc648d9a5ad2c62930861c4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:12.398010 15580 raft_consensus.cc:2454] T f72aa769aa8f465ba4469dfc34ae35c1 P 17eb0a665dc648d9a5ad2c62930861c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5751ea0f088543bcb141b400a6131a1f in term 1.
I20240503 02:00:12.398308 15120 leader_election.cc:302] T f72aa769aa8f465ba4469dfc34ae35c1 P 5751ea0f088543bcb141b400a6131a1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 17eb0a665dc648d9a5ad2c62930861c4, 5751ea0f088543bcb141b400a6131a1f; no voters:
I20240503 02:00:12.398489 15763 raft_consensus.cc:2781] T f72aa769aa8f465ba4469dfc34ae35c1 P 5751ea0f088543bcb141b400a6131a1f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:12.398597 15763 raft_consensus.cc:686] T f72aa769aa8f465ba4469dfc34ae35c1 P 5751ea0f088543bcb141b400a6131a1f [term 1 LEADER]: Becoming Leader. State: Replica: 5751ea0f088543bcb141b400a6131a1f, State: Running, Role: LEADER
I20240503 02:00:12.398742 15763 consensus_queue.cc:237] T f72aa769aa8f465ba4469dfc34ae35c1 P 5751ea0f088543bcb141b400a6131a1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } } peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } }
I20240503 02:00:12.400280 15034 catalog_manager.cc:5484] T f72aa769aa8f465ba4469dfc34ae35c1 P 5751ea0f088543bcb141b400a6131a1f reported cstate change: term changed from 0 to 1, leader changed from <none> to 5751ea0f088543bcb141b400a6131a1f (127.0.99.129). New cstate: current_term: 1 leader_uuid: "5751ea0f088543bcb141b400a6131a1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } health_report { overall_health: UNKNOWN } } }
W20240503 02:00:12.544168 15775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:00:12.544402 15775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:00:12.547736 15775 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240503 02:00:12.621843 15184 raft_consensus.cc:1261] T a14de0ce61c74ac5ae8bdcfb403e7e5d P 5751ea0f088543bcb141b400a6131a1f [term 1 FOLLOWER]: Refusing update from remote peer 17eb0a665dc648d9a5ad2c62930861c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:12.622139 15183 raft_consensus.cc:1261] T f357df7d66a646fc817b19b28306a5a0 P 5751ea0f088543bcb141b400a6131a1f [term 1 FOLLOWER]: Refusing update from remote peer 17eb0a665dc648d9a5ad2c62930861c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:12.622581 15711 consensus_queue.cc:1035] T a14de0ce61c74ac5ae8bdcfb403e7e5d P 17eb0a665dc648d9a5ad2c62930861c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:12.622853 15711 consensus_queue.cc:1035] T f357df7d66a646fc817b19b28306a5a0 P 17eb0a665dc648d9a5ad2c62930861c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:12.624451 15580 raft_consensus.cc:1261] T f72aa769aa8f465ba4469dfc34ae35c1 P 17eb0a665dc648d9a5ad2c62930861c4 [term 1 FOLLOWER]: Refusing update from remote peer 5751ea0f088543bcb141b400a6131a1f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:12.625348 15763 consensus_queue.cc:1035] T f72aa769aa8f465ba4469dfc34ae35c1 P 5751ea0f088543bcb141b400a6131a1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 80.9827 ms
time per row: 0.0404914 ms
I20240503 02:00:12.668802 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task42rE7p/build/debug/bin/kudu
/tmp/dist-test-task42rE7p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.130:46271
--local_ip_for_outbound_sockets=127.0.99.130
--tserver_master_addrs=127.0.99.190:46163
--webserver_port=34995
--webserver_interface=127.0.99.130
--builtin_ntp_servers=127.0.99.148:33691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 02:00:12.772375 15805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:00:12.772572 15805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:00:12.772614 15805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:00:12.775563 15805 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:00:12.775625 15805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:00:12.775722 15805 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.130
I20240503 02:00:12.779165 15805 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:33691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.130:46271
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.99.130
--webserver_port=34995
--tserver_master_addrs=127.0.99.190:46163
--never_fsync=true
--heap_profile_path=/tmp/kudu.15805
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.130
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 02:00:12.779973 15805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:00:12.781069 15805 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 02:00:12.790293 15805 server_base.cc:1034] running on GCE node
W20240503 02:00:12.791136 15813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:12.791919 15810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:12.793210 15811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:00:12.793699 15805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:00:12.794293 15805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:00:12.795473 15805 hybrid_clock.cc:648] HybridClock initialized: now 1714701612795449 us; error 37 us; skew 500 ppm
I20240503 02:00:12.797370 15805 webserver.cc:460] Webserver started at http://127.0.99.130:34995/ using document root <none> and password file <none>
I20240503 02:00:12.797904 15805 fs_manager.cc:359] Metadata directory not provided
I20240503 02:00:12.797977 15805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:00:12.801430 15805 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20240503 02:00:12.803268 15819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:12.804277 15805 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:12.804428 15805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "7932a656946f4546a91738584eb612e2"
format_stamp: "Formatted at 2024-05-03 02:00:06 on dist-test-slave-dtz3"
I20240503 02:00:12.804875 15805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:00:12.817469 15805 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:00:12.818001 15805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:00:12.818156 15805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:00:12.818645 15805 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:00:12.819497 15805 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:00:12.819545 15805 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:12.819598 15805 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:00:12.819628 15805 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:12.828420 15805 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.130:46271
I20240503 02:00:12.828465 15932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.130:46271 every 8 connection(s)
I20240503 02:00:12.829452 15805 server_base.cc:1166] Dumped server information to /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 02:00:12.833909 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task42rE7p/build/debug/bin/kudu as pid 15805
I20240503 02:00:12.834316 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task42rE7p/build/debug/bin/kudu
/tmp/dist-test-task42rE7p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.131:34755
--local_ip_for_outbound_sockets=127.0.99.131
--tserver_master_addrs=127.0.99.190:46163
--webserver_port=37705
--webserver_interface=127.0.99.131
--builtin_ntp_servers=127.0.99.148:33691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:00:12.838470 15933 heartbeater.cc:347] Connected to a master server at 127.0.99.190:46163
I20240503 02:00:12.838689 15933 heartbeater.cc:464] Registering TS with master...
I20240503 02:00:12.839314 15933 heartbeater.cc:511] Master 127.0.99.190:46163 requested a full tablet report, sending...
I20240503 02:00:12.840699 15034 ts_manager.cc:194] Re-registered known tserver with Master: 7932a656946f4546a91738584eb612e2 (127.0.99.130:46271)
I20240503 02:00:12.841534 15034 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.130:60839
W20240503 02:00:12.942432 15936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:00:12.942651 15936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:00:12.942692 15936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:00:12.945686 15936 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:00:12.945762 15936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:00:12.945832 15936 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.131
I20240503 02:00:12.949599 15936 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:33691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.131:34755
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.99.131
--webserver_port=37705
--tserver_master_addrs=127.0.99.190:46163
--never_fsync=true
--heap_profile_path=/tmp/kudu.15936
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.131
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 02:00:12.950438 15936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:00:12.951474 15936 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:00:12.960901 15942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:12.962075 15944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:00:12.963228 15936 server_base.cc:1034] running on GCE node
W20240503 02:00:12.964051 15941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:00:12.964542 15936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:00:12.965205 15936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:00:12.966383 15936 hybrid_clock.cc:648] HybridClock initialized: now 1714701612966358 us; error 40 us; skew 500 ppm
I20240503 02:00:12.968273 15936 webserver.cc:460] Webserver started at http://127.0.99.131:37705/ using document root <none> and password file <none>
I20240503 02:00:12.968865 15936 fs_manager.cc:359] Metadata directory not provided
I20240503 02:00:12.968945 15936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:00:12.972121 15936 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240503 02:00:12.974083 15950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:12.975082 15936 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:12.975178 15936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "3f852460ab584dfb937c35f39f19f394"
format_stamp: "Formatted at 2024-05-03 02:00:06 on dist-test-slave-dtz3"
I20240503 02:00:12.975649 15936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:00:13.033916 15936 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:00:13.034543 15936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:00:13.034699 15936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:00:13.035249 15936 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:00:13.036159 15936 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:00:13.036209 15936 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:13.036269 15936 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:00:13.036296 15936 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:13.044745 15936 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.131:34755
I20240503 02:00:13.044802 16063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.131:34755 every 8 connection(s)
I20240503 02:00:13.045835 15936 server_base.cc:1166] Dumped server information to /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 02:00:13.050740 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task42rE7p/build/debug/bin/kudu as pid 15936
I20240503 02:00:13.055351 16064 heartbeater.cc:347] Connected to a master server at 127.0.99.190:46163
I20240503 02:00:13.055562 16064 heartbeater.cc:464] Registering TS with master...
I20240503 02:00:13.056108 16064 heartbeater.cc:511] Master 127.0.99.190:46163 requested a full tablet report, sending...
I20240503 02:00:13.057171 15034 ts_manager.cc:194] Re-registered known tserver with Master: 3f852460ab584dfb937c35f39f19f394 (127.0.99.131:34755)
I20240503 02:00:13.057833 15034 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.131:40177
I20240503 02:00:13.234628 15164 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:00:13.235184 15998 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:13.237622 15560 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:00:13.252846 15867 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:13.329818 15998 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:13.330601 15560 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:00:13.330950 15164 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:00:13.331823 15867 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:13.429176 15998 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:13.430224 15164 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:00:13.430361 15560 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:00:13.430940 15867 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:13.514286 15867 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:13.514501 15164 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:00:13.514694 15560 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:00:13.514964 15998 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:13.551445 15580 consensus_queue.cc:237] T f357df7d66a646fc817b19b28306a5a0 P 17eb0a665dc648d9a5ad2c62930861c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } } peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } attrs { replace: true } } peers { permanent_uuid: "3f852460ab584dfb937c35f39f19f394" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34755 } attrs { promote: true } }
I20240503 02:00:13.552946 15184 raft_consensus.cc:1261] T f357df7d66a646fc817b19b28306a5a0 P 5751ea0f088543bcb141b400a6131a1f [term 1 FOLLOWER]: Refusing update from remote peer 17eb0a665dc648d9a5ad2c62930861c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:13.553373 16205 consensus_queue.cc:1035] T f357df7d66a646fc817b19b28306a5a0 P 17eb0a665dc648d9a5ad2c62930861c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:13.554972 15804 raft_consensus.cc:2932] T f357df7d66a646fc817b19b28306a5a0 P 17eb0a665dc648d9a5ad2c62930861c4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3f852460ab584dfb937c35f39f19f394 (127.0.99.131) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } } peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } attrs { replace: true } } peers { permanent_uuid: "3f852460ab584dfb937c35f39f19f394" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34755 } attrs { promote: true } } }
I20240503 02:00:13.555377 15184 raft_consensus.cc:2932] T f357df7d66a646fc817b19b28306a5a0 P 5751ea0f088543bcb141b400a6131a1f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3f852460ab584dfb937c35f39f19f394 (127.0.99.131) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } } peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } attrs { replace: true } } peers { permanent_uuid: "3f852460ab584dfb937c35f39f19f394" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34755 } attrs { promote: true } } }
W20240503 02:00:13.556962 15515 consensus_peers.cc:485] T f357df7d66a646fc817b19b28306a5a0 P 17eb0a665dc648d9a5ad2c62930861c4 -> Peer 3f852460ab584dfb937c35f39f19f394 (127.0.99.131:34755): Couldn't send request to peer 3f852460ab584dfb937c35f39f19f394. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f357df7d66a646fc817b19b28306a5a0. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:13.557991 15039 catalog_manager.cc:5484] T f357df7d66a646fc817b19b28306a5a0 P 5751ea0f088543bcb141b400a6131a1f reported cstate change: config changed from index -1 to 4, NON_VOTER 3f852460ab584dfb937c35f39f19f394 (127.0.99.131) added. New cstate: current_term: 1 leader_uuid: "17eb0a665dc648d9a5ad2c62930861c4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } } peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } attrs { replace: true } } peers { permanent_uuid: "3f852460ab584dfb937c35f39f19f394" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34755 } attrs { promote: true } } }
I20240503 02:00:13.564632 15184 consensus_queue.cc:237] T 7244e26e54694bd48373e8634b4daf93 P 5751ea0f088543bcb141b400a6131a1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } } peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } attrs { replace: true } } peers { permanent_uuid: "7932a656946f4546a91738584eb612e2" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46271 } attrs { promote: true } }
I20240503 02:00:13.565936 15580 raft_consensus.cc:1261] T 7244e26e54694bd48373e8634b4daf93 P 17eb0a665dc648d9a5ad2c62930861c4 [term 1 FOLLOWER]: Refusing update from remote peer 5751ea0f088543bcb141b400a6131a1f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:13.566365 16217 consensus_queue.cc:1035] T 7244e26e54694bd48373e8634b4daf93 P 5751ea0f088543bcb141b400a6131a1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:13.567948 15763 raft_consensus.cc:2932] T 7244e26e54694bd48373e8634b4daf93 P 5751ea0f088543bcb141b400a6131a1f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 7932a656946f4546a91738584eb612e2 (127.0.99.130) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } } peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } attrs { replace: true } } peers { permanent_uuid: "7932a656946f4546a91738584eb612e2" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46271 } attrs { promote: true } } }
I20240503 02:00:13.568455 15580 raft_consensus.cc:2932] T 7244e26e54694bd48373e8634b4daf93 P 17eb0a665dc648d9a5ad2c62930861c4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 7932a656946f4546a91738584eb612e2 (127.0.99.130) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } } peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } attrs { replace: true } } peers { permanent_uuid: "7932a656946f4546a91738584eb612e2" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46271 } attrs { promote: true } } }
W20240503 02:00:13.570017 15117 consensus_peers.cc:485] T 7244e26e54694bd48373e8634b4daf93 P 5751ea0f088543bcb141b400a6131a1f -> Peer 7932a656946f4546a91738584eb612e2 (127.0.99.130:46271): Couldn't send request to peer 7932a656946f4546a91738584eb612e2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7244e26e54694bd48373e8634b4daf93. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:13.570762 15038 catalog_manager.cc:5484] T 7244e26e54694bd48373e8634b4daf93 P 5751ea0f088543bcb141b400a6131a1f reported cstate change: config changed from index -1 to 4, NON_VOTER 7932a656946f4546a91738584eb612e2 (127.0.99.130) added. New cstate: current_term: 1 leader_uuid: "5751ea0f088543bcb141b400a6131a1f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7932a656946f4546a91738584eb612e2" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46271 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:13.577060 15580 consensus_queue.cc:237] T 595c6713bec0492882c1e58225dcce42 P 17eb0a665dc648d9a5ad2c62930861c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } } peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } attrs { replace: true } } peers { permanent_uuid: "3f852460ab584dfb937c35f39f19f394" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34755 } attrs { promote: true } }
I20240503 02:00:13.578217 15184 raft_consensus.cc:1261] T 595c6713bec0492882c1e58225dcce42 P 5751ea0f088543bcb141b400a6131a1f [term 1 FOLLOWER]: Refusing update from remote peer 17eb0a665dc648d9a5ad2c62930861c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 02:00:13.578410 15515 consensus_peers.cc:485] T 595c6713bec0492882c1e58225dcce42 P 17eb0a665dc648d9a5ad2c62930861c4 -> Peer 3f852460ab584dfb937c35f39f19f394 (127.0.99.131:34755): Couldn't send request to peer 3f852460ab584dfb937c35f39f19f394. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 595c6713bec0492882c1e58225dcce42. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:13.578682 15804 consensus_queue.cc:1035] T 595c6713bec0492882c1e58225dcce42 P 17eb0a665dc648d9a5ad2c62930861c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:13.580235 16205 raft_consensus.cc:2932] T 595c6713bec0492882c1e58225dcce42 P 17eb0a665dc648d9a5ad2c62930861c4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3f852460ab584dfb937c35f39f19f394 (127.0.99.131) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } } peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } attrs { replace: true } } peers { permanent_uuid: "3f852460ab584dfb937c35f39f19f394" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34755 } attrs { promote: true } } }
I20240503 02:00:13.580780 15184 raft_consensus.cc:2932] T 595c6713bec0492882c1e58225dcce42 P 5751ea0f088543bcb141b400a6131a1f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3f852460ab584dfb937c35f39f19f394 (127.0.99.131) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } } peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } attrs { replace: true } } peers { permanent_uuid: "3f852460ab584dfb937c35f39f19f394" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34755 } attrs { promote: true } } }
I20240503 02:00:13.582613 15034 catalog_manager.cc:5484] T 595c6713bec0492882c1e58225dcce42 P 17eb0a665dc648d9a5ad2c62930861c4 reported cstate change: config changed from index -1 to 4, NON_VOTER 3f852460ab584dfb937c35f39f19f394 (127.0.99.131) added. New cstate: current_term: 1 leader_uuid: "17eb0a665dc648d9a5ad2c62930861c4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3f852460ab584dfb937c35f39f19f394" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34755 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:13.589607 15184 consensus_queue.cc:237] T f72aa769aa8f465ba4469dfc34ae35c1 P 5751ea0f088543bcb141b400a6131a1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } } peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } attrs { replace: true } } peers { permanent_uuid: "7932a656946f4546a91738584eb612e2" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46271 } attrs { promote: true } }
I20240503 02:00:13.590801 15580 raft_consensus.cc:1261] T f72aa769aa8f465ba4469dfc34ae35c1 P 17eb0a665dc648d9a5ad2c62930861c4 [term 1 FOLLOWER]: Refusing update from remote peer 5751ea0f088543bcb141b400a6131a1f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 02:00:13.590899 15117 consensus_peers.cc:485] T f72aa769aa8f465ba4469dfc34ae35c1 P 5751ea0f088543bcb141b400a6131a1f -> Peer 7932a656946f4546a91738584eb612e2 (127.0.99.130:46271): Couldn't send request to peer 7932a656946f4546a91738584eb612e2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f72aa769aa8f465ba4469dfc34ae35c1. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:13.591194 16220 consensus_queue.cc:1035] T f72aa769aa8f465ba4469dfc34ae35c1 P 5751ea0f088543bcb141b400a6131a1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:13.592581 15763 raft_consensus.cc:2932] T f72aa769aa8f465ba4469dfc34ae35c1 P 5751ea0f088543bcb141b400a6131a1f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 7932a656946f4546a91738584eb612e2 (127.0.99.130) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } } peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } attrs { replace: true } } peers { permanent_uuid: "7932a656946f4546a91738584eb612e2" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46271 } attrs { promote: true } } }
I20240503 02:00:13.593008 15580 raft_consensus.cc:2932] T f72aa769aa8f465ba4469dfc34ae35c1 P 17eb0a665dc648d9a5ad2c62930861c4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 7932a656946f4546a91738584eb612e2 (127.0.99.130) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } } peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } attrs { replace: true } } peers { permanent_uuid: "7932a656946f4546a91738584eb612e2" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46271 } attrs { promote: true } } }
I20240503 02:00:13.594980 15038 catalog_manager.cc:5484] T f72aa769aa8f465ba4469dfc34ae35c1 P 5751ea0f088543bcb141b400a6131a1f reported cstate change: config changed from index -1 to 4, NON_VOTER 7932a656946f4546a91738584eb612e2 (127.0.99.130) added. New cstate: current_term: 1 leader_uuid: "5751ea0f088543bcb141b400a6131a1f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7932a656946f4546a91738584eb612e2" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46271 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:13.601771 15184 consensus_queue.cc:237] T 4438db9442024011bd8d5632a03d34b0 P 5751ea0f088543bcb141b400a6131a1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } attrs { replace: true } } peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } } peers { permanent_uuid: "7932a656946f4546a91738584eb612e2" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46271 } attrs { promote: true } }
I20240503 02:00:13.602926 15580 raft_consensus.cc:1261] T 4438db9442024011bd8d5632a03d34b0 P 17eb0a665dc648d9a5ad2c62930861c4 [term 1 FOLLOWER]: Refusing update from remote peer 5751ea0f088543bcb141b400a6131a1f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 02:00:13.603219 15117 consensus_peers.cc:485] T 4438db9442024011bd8d5632a03d34b0 P 5751ea0f088543bcb141b400a6131a1f -> Peer 7932a656946f4546a91738584eb612e2 (127.0.99.130:46271): Couldn't send request to peer 7932a656946f4546a91738584eb612e2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4438db9442024011bd8d5632a03d34b0. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:13.603371 15763 consensus_queue.cc:1035] T 4438db9442024011bd8d5632a03d34b0 P 5751ea0f088543bcb141b400a6131a1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:13.605098 16220 raft_consensus.cc:2932] T 4438db9442024011bd8d5632a03d34b0 P 5751ea0f088543bcb141b400a6131a1f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 7932a656946f4546a91738584eb612e2 (127.0.99.130) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } attrs { replace: true } } peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } } peers { permanent_uuid: "7932a656946f4546a91738584eb612e2" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46271 } attrs { promote: true } } }
I20240503 02:00:13.605561 15580 raft_consensus.cc:2932] T 4438db9442024011bd8d5632a03d34b0 P 17eb0a665dc648d9a5ad2c62930861c4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 7932a656946f4546a91738584eb612e2 (127.0.99.130) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } attrs { replace: true } } peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } } peers { permanent_uuid: "7932a656946f4546a91738584eb612e2" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46271 } attrs { promote: true } } }
I20240503 02:00:13.608029 15034 catalog_manager.cc:5484] T 4438db9442024011bd8d5632a03d34b0 P 17eb0a665dc648d9a5ad2c62930861c4 reported cstate change: config changed from index -1 to 4, NON_VOTER 7932a656946f4546a91738584eb612e2 (127.0.99.130) added. New cstate: current_term: 1 leader_uuid: "5751ea0f088543bcb141b400a6131a1f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } attrs { replace: true } } peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } } peers { permanent_uuid: "7932a656946f4546a91738584eb612e2" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46271 } attrs { promote: true } } }
I20240503 02:00:13.614147 15184 consensus_queue.cc:237] T 84425c87d5964b4bb5c5ecfb72360e07 P 5751ea0f088543bcb141b400a6131a1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } attrs { replace: true } } peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } } peers { permanent_uuid: "3f852460ab584dfb937c35f39f19f394" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34755 } attrs { promote: true } }
I20240503 02:00:13.615502 15580 raft_consensus.cc:1261] T 84425c87d5964b4bb5c5ecfb72360e07 P 17eb0a665dc648d9a5ad2c62930861c4 [term 1 FOLLOWER]: Refusing update from remote peer 5751ea0f088543bcb141b400a6131a1f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:13.615885 15763 consensus_queue.cc:1035] T 84425c87d5964b4bb5c5ecfb72360e07 P 5751ea0f088543bcb141b400a6131a1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:13.617683 16220 raft_consensus.cc:2932] T 84425c87d5964b4bb5c5ecfb72360e07 P 5751ea0f088543bcb141b400a6131a1f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3f852460ab584dfb937c35f39f19f394 (127.0.99.131) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } attrs { replace: true } } peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } } peers { permanent_uuid: "3f852460ab584dfb937c35f39f19f394" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34755 } attrs { promote: true } } }
I20240503 02:00:13.618153 15580 raft_consensus.cc:2932] T 84425c87d5964b4bb5c5ecfb72360e07 P 17eb0a665dc648d9a5ad2c62930861c4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3f852460ab584dfb937c35f39f19f394 (127.0.99.131) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } attrs { replace: true } } peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } } peers { permanent_uuid: "3f852460ab584dfb937c35f39f19f394" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34755 } attrs { promote: true } } }
W20240503 02:00:13.619601 15119 consensus_peers.cc:485] T 84425c87d5964b4bb5c5ecfb72360e07 P 5751ea0f088543bcb141b400a6131a1f -> Peer 3f852460ab584dfb937c35f39f19f394 (127.0.99.131:34755): Couldn't send request to peer 3f852460ab584dfb937c35f39f19f394. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 84425c87d5964b4bb5c5ecfb72360e07. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:13.620441 15038 catalog_manager.cc:5484] T 84425c87d5964b4bb5c5ecfb72360e07 P 17eb0a665dc648d9a5ad2c62930861c4 reported cstate change: config changed from index -1 to 4, NON_VOTER 3f852460ab584dfb937c35f39f19f394 (127.0.99.131) added. New cstate: current_term: 1 leader_uuid: "5751ea0f088543bcb141b400a6131a1f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } attrs { replace: true } } peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } } peers { permanent_uuid: "3f852460ab584dfb937c35f39f19f394" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34755 } attrs { promote: true } } }
I20240503 02:00:13.843465 15933 heartbeater.cc:503] Master 127.0.99.190:46163 was elected leader, sending a full tablet report...
I20240503 02:00:14.020128 16308 ts_tablet_manager.cc:902] T f357df7d66a646fc817b19b28306a5a0 P 3f852460ab584dfb937c35f39f19f394: Initiating tablet copy from peer 17eb0a665dc648d9a5ad2c62930861c4 (127.0.99.132:33559)
I20240503 02:00:14.020901 16308 tablet_copy_client.cc:308] T f357df7d66a646fc817b19b28306a5a0 P 3f852460ab584dfb937c35f39f19f394: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:33559
I20240503 02:00:14.024310 15600 tablet_copy_service.cc:140] P 17eb0a665dc648d9a5ad2c62930861c4: Received BeginTabletCopySession request for tablet f357df7d66a646fc817b19b28306a5a0 from peer 3f852460ab584dfb937c35f39f19f394 ({username='slave'} at 127.0.99.131:49287)
I20240503 02:00:14.024439 15600 tablet_copy_service.cc:161] P 17eb0a665dc648d9a5ad2c62930861c4: Beginning new tablet copy session on tablet f357df7d66a646fc817b19b28306a5a0 from peer 3f852460ab584dfb937c35f39f19f394 at {username='slave'} at 127.0.99.131:49287: session id = 3f852460ab584dfb937c35f39f19f394-f357df7d66a646fc817b19b28306a5a0
I20240503 02:00:14.025615 15600 tablet_copy_source_session.cc:205] T f357df7d66a646fc817b19b28306a5a0 P 17eb0a665dc648d9a5ad2c62930861c4: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:14.028024 16314 ts_tablet_manager.cc:902] T 595c6713bec0492882c1e58225dcce42 P 3f852460ab584dfb937c35f39f19f394: Initiating tablet copy from peer 17eb0a665dc648d9a5ad2c62930861c4 (127.0.99.132:33559)
I20240503 02:00:14.028496 16314 tablet_copy_client.cc:308] T 595c6713bec0492882c1e58225dcce42 P 3f852460ab584dfb937c35f39f19f394: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:33559
I20240503 02:00:14.028906 15600 tablet_copy_service.cc:140] P 17eb0a665dc648d9a5ad2c62930861c4: Received BeginTabletCopySession request for tablet 595c6713bec0492882c1e58225dcce42 from peer 3f852460ab584dfb937c35f39f19f394 ({username='slave'} at 127.0.99.131:49287)
I20240503 02:00:14.028997 15600 tablet_copy_service.cc:161] P 17eb0a665dc648d9a5ad2c62930861c4: Beginning new tablet copy session on tablet 595c6713bec0492882c1e58225dcce42 from peer 3f852460ab584dfb937c35f39f19f394 at {username='slave'} at 127.0.99.131:49287: session id = 3f852460ab584dfb937c35f39f19f394-595c6713bec0492882c1e58225dcce42
I20240503 02:00:14.029953 15600 tablet_copy_source_session.cc:205] T 595c6713bec0492882c1e58225dcce42 P 17eb0a665dc648d9a5ad2c62930861c4: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:14.030864 16308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f357df7d66a646fc817b19b28306a5a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:14.031225 16314 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 595c6713bec0492882c1e58225dcce42. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:14.036765 16314 tablet_copy_client.cc:783] T 595c6713bec0492882c1e58225dcce42 P 3f852460ab584dfb937c35f39f19f394: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:14.036831 16308 tablet_copy_client.cc:783] T f357df7d66a646fc817b19b28306a5a0 P 3f852460ab584dfb937c35f39f19f394: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:14.036940 16314 tablet_copy_client.cc:647] T 595c6713bec0492882c1e58225dcce42 P 3f852460ab584dfb937c35f39f19f394: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:14.036986 16308 tablet_copy_client.cc:647] T f357df7d66a646fc817b19b28306a5a0 P 3f852460ab584dfb937c35f39f19f394: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:14.038483 16314 tablet_copy_client.cc:523] T 595c6713bec0492882c1e58225dcce42 P 3f852460ab584dfb937c35f39f19f394: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:14.038484 16308 tablet_copy_client.cc:523] T f357df7d66a646fc817b19b28306a5a0 P 3f852460ab584dfb937c35f39f19f394: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:14.039870 16314 tablet_bootstrap.cc:492] T 595c6713bec0492882c1e58225dcce42 P 3f852460ab584dfb937c35f39f19f394: Bootstrap starting.
I20240503 02:00:14.039870 16308 tablet_bootstrap.cc:492] T f357df7d66a646fc817b19b28306a5a0 P 3f852460ab584dfb937c35f39f19f394: Bootstrap starting.
I20240503 02:00:14.044957 16308 log.cc:824] T f357df7d66a646fc817b19b28306a5a0 P 3f852460ab584dfb937c35f39f19f394: Log is configured to *not* fsync() on all Append() calls
I20240503 02:00:14.053822 16321 ts_tablet_manager.cc:902] T f72aa769aa8f465ba4469dfc34ae35c1 P 7932a656946f4546a91738584eb612e2: Initiating tablet copy from peer 5751ea0f088543bcb141b400a6131a1f (127.0.99.129:46041)
I20240503 02:00:14.054587 16321 tablet_copy_client.cc:308] T f72aa769aa8f465ba4469dfc34ae35c1 P 7932a656946f4546a91738584eb612e2: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:46041
I20240503 02:00:14.059680 15204 tablet_copy_service.cc:140] P 5751ea0f088543bcb141b400a6131a1f: Received BeginTabletCopySession request for tablet f72aa769aa8f465ba4469dfc34ae35c1 from peer 7932a656946f4546a91738584eb612e2 ({username='slave'} at 127.0.99.130:34095)
I20240503 02:00:14.059762 16064 heartbeater.cc:503] Master 127.0.99.190:46163 was elected leader, sending a full tablet report...
I20240503 02:00:14.059829 15204 tablet_copy_service.cc:161] P 5751ea0f088543bcb141b400a6131a1f: Beginning new tablet copy session on tablet f72aa769aa8f465ba4469dfc34ae35c1 from peer 7932a656946f4546a91738584eb612e2 at {username='slave'} at 127.0.99.130:34095: session id = 7932a656946f4546a91738584eb612e2-f72aa769aa8f465ba4469dfc34ae35c1
I20240503 02:00:14.061277 15204 tablet_copy_source_session.cc:205] T f72aa769aa8f465ba4469dfc34ae35c1 P 5751ea0f088543bcb141b400a6131a1f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:14.064716 16321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f72aa769aa8f465ba4469dfc34ae35c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:14.069680 16321 tablet_copy_client.cc:783] T f72aa769aa8f465ba4469dfc34ae35c1 P 7932a656946f4546a91738584eb612e2: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:14.069759 16314 tablet_bootstrap.cc:492] T 595c6713bec0492882c1e58225dcce42 P 3f852460ab584dfb937c35f39f19f394: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:14.069869 16321 tablet_copy_client.cc:647] T f72aa769aa8f465ba4469dfc34ae35c1 P 7932a656946f4546a91738584eb612e2: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:14.070089 16314 tablet_bootstrap.cc:492] T 595c6713bec0492882c1e58225dcce42 P 3f852460ab584dfb937c35f39f19f394: Bootstrap complete.
I20240503 02:00:14.070379 16314 ts_tablet_manager.cc:1367] T 595c6713bec0492882c1e58225dcce42 P 3f852460ab584dfb937c35f39f19f394: Time spent bootstrapping tablet: real 0.031s user 0.029s sys 0.000s
I20240503 02:00:14.071651 16321 tablet_copy_client.cc:523] T f72aa769aa8f465ba4469dfc34ae35c1 P 7932a656946f4546a91738584eb612e2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:14.071938 16308 tablet_bootstrap.cc:492] T f357df7d66a646fc817b19b28306a5a0 P 3f852460ab584dfb937c35f39f19f394: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:14.072167 16308 tablet_bootstrap.cc:492] T f357df7d66a646fc817b19b28306a5a0 P 3f852460ab584dfb937c35f39f19f394: Bootstrap complete.
I20240503 02:00:14.072304 16308 ts_tablet_manager.cc:1367] T f357df7d66a646fc817b19b28306a5a0 P 3f852460ab584dfb937c35f39f19f394: Time spent bootstrapping tablet: real 0.033s user 0.028s sys 0.000s
I20240503 02:00:14.072748 16308 raft_consensus.cc:348] T f357df7d66a646fc817b19b28306a5a0 P 3f852460ab584dfb937c35f39f19f394 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } } peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } attrs { replace: true } } peers { permanent_uuid: "3f852460ab584dfb937c35f39f19f394" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34755 } attrs { promote: true } }
I20240503 02:00:14.072727 16314 raft_consensus.cc:348] T 595c6713bec0492882c1e58225dcce42 P 3f852460ab584dfb937c35f39f19f394 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } } peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } attrs { replace: true } } peers { permanent_uuid: "3f852460ab584dfb937c35f39f19f394" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34755 } attrs { promote: true } }
I20240503 02:00:14.073005 16308 raft_consensus.cc:729] T f357df7d66a646fc817b19b28306a5a0 P 3f852460ab584dfb937c35f39f19f394 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3f852460ab584dfb937c35f39f19f394, State: Initialized, Role: LEARNER
I20240503 02:00:14.073005 16314 raft_consensus.cc:729] T 595c6713bec0492882c1e58225dcce42 P 3f852460ab584dfb937c35f39f19f394 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3f852460ab584dfb937c35f39f19f394, State: Initialized, Role: LEARNER
I20240503 02:00:14.073335 16321 tablet_bootstrap.cc:492] T f72aa769aa8f465ba4469dfc34ae35c1 P 7932a656946f4546a91738584eb612e2: Bootstrap starting.
I20240503 02:00:14.073364 16308 consensus_queue.cc:260] T f357df7d66a646fc817b19b28306a5a0 P 3f852460ab584dfb937c35f39f19f394 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } } peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } attrs { replace: true } } peers { permanent_uuid: "3f852460ab584dfb937c35f39f19f394" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34755 } attrs { promote: true } }
I20240503 02:00:14.073364 16314 consensus_queue.cc:260] T 595c6713bec0492882c1e58225dcce42 P 3f852460ab584dfb937c35f39f19f394 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } } peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } attrs { replace: true } } peers { permanent_uuid: "3f852460ab584dfb937c35f39f19f394" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34755 } attrs { promote: true } }
I20240503 02:00:14.074183 16314 ts_tablet_manager.cc:1398] T 595c6713bec0492882c1e58225dcce42 P 3f852460ab584dfb937c35f39f19f394: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20240503 02:00:14.074183 16308 ts_tablet_manager.cc:1398] T f357df7d66a646fc817b19b28306a5a0 P 3f852460ab584dfb937c35f39f19f394: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 02:00:14.074956 15600 tablet_copy_service.cc:342] P 17eb0a665dc648d9a5ad2c62930861c4: Request end of tablet copy session 3f852460ab584dfb937c35f39f19f394-f357df7d66a646fc817b19b28306a5a0 received from {username='slave'} at 127.0.99.131:49287
I20240503 02:00:14.075034 15600 tablet_copy_service.cc:433] P 17eb0a665dc648d9a5ad2c62930861c4: ending tablet copy session 3f852460ab584dfb937c35f39f19f394-f357df7d66a646fc817b19b28306a5a0 on tablet f357df7d66a646fc817b19b28306a5a0 with peer 3f852460ab584dfb937c35f39f19f394
I20240503 02:00:14.075268 15599 tablet_copy_service.cc:342] P 17eb0a665dc648d9a5ad2c62930861c4: Request end of tablet copy session 3f852460ab584dfb937c35f39f19f394-595c6713bec0492882c1e58225dcce42 received from {username='slave'} at 127.0.99.131:49287
I20240503 02:00:14.075325 15599 tablet_copy_service.cc:433] P 17eb0a665dc648d9a5ad2c62930861c4: ending tablet copy session 3f852460ab584dfb937c35f39f19f394-595c6713bec0492882c1e58225dcce42 on tablet 595c6713bec0492882c1e58225dcce42 with peer 3f852460ab584dfb937c35f39f19f394
I20240503 02:00:14.079228 16321 log.cc:824] T f72aa769aa8f465ba4469dfc34ae35c1 P 7932a656946f4546a91738584eb612e2: Log is configured to *not* fsync() on all Append() calls
I20240503 02:00:14.103166 16321 tablet_bootstrap.cc:492] T f72aa769aa8f465ba4469dfc34ae35c1 P 7932a656946f4546a91738584eb612e2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:14.103542 16321 tablet_bootstrap.cc:492] T f72aa769aa8f465ba4469dfc34ae35c1 P 7932a656946f4546a91738584eb612e2: Bootstrap complete.
I20240503 02:00:14.103864 16321 ts_tablet_manager.cc:1367] T f72aa769aa8f465ba4469dfc34ae35c1 P 7932a656946f4546a91738584eb612e2: Time spent bootstrapping tablet: real 0.031s user 0.028s sys 0.000s
I20240503 02:00:14.106418 16321 raft_consensus.cc:348] T f72aa769aa8f465ba4469dfc34ae35c1 P 7932a656946f4546a91738584eb612e2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } } peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } attrs { replace: true } } peers { permanent_uuid: "7932a656946f4546a91738584eb612e2" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46271 } attrs { promote: true } }
I20240503 02:00:14.106621 16321 raft_consensus.cc:729] T f72aa769aa8f465ba4469dfc34ae35c1 P 7932a656946f4546a91738584eb612e2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7932a656946f4546a91738584eb612e2, State: Initialized, Role: LEARNER
I20240503 02:00:14.106995 16321 consensus_queue.cc:260] T f72aa769aa8f465ba4469dfc34ae35c1 P 7932a656946f4546a91738584eb612e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } } peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } attrs { replace: true } } peers { permanent_uuid: "7932a656946f4546a91738584eb612e2" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46271 } attrs { promote: true } }
I20240503 02:00:14.108003 16321 ts_tablet_manager.cc:1398] T f72aa769aa8f465ba4469dfc34ae35c1 P 7932a656946f4546a91738584eb612e2: Time spent starting tablet: real 0.004s user 0.007s sys 0.000s
I20240503 02:00:14.108636 15204 tablet_copy_service.cc:342] P 5751ea0f088543bcb141b400a6131a1f: Request end of tablet copy session 7932a656946f4546a91738584eb612e2-f72aa769aa8f465ba4469dfc34ae35c1 received from {username='slave'} at 127.0.99.130:34095
I20240503 02:00:14.108749 15204 tablet_copy_service.cc:433] P 5751ea0f088543bcb141b400a6131a1f: ending tablet copy session 7932a656946f4546a91738584eb612e2-f72aa769aa8f465ba4469dfc34ae35c1 on tablet f72aa769aa8f465ba4469dfc34ae35c1 with peer 7932a656946f4546a91738584eb612e2
I20240503 02:00:14.137204 16321 ts_tablet_manager.cc:902] T 4438db9442024011bd8d5632a03d34b0 P 7932a656946f4546a91738584eb612e2: Initiating tablet copy from peer 5751ea0f088543bcb141b400a6131a1f (127.0.99.129:46041)
I20240503 02:00:14.137614 16321 tablet_copy_client.cc:308] T 4438db9442024011bd8d5632a03d34b0 P 7932a656946f4546a91738584eb612e2: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:46041
I20240503 02:00:14.138062 15204 tablet_copy_service.cc:140] P 5751ea0f088543bcb141b400a6131a1f: Received BeginTabletCopySession request for tablet 4438db9442024011bd8d5632a03d34b0 from peer 7932a656946f4546a91738584eb612e2 ({username='slave'} at 127.0.99.130:34095)
I20240503 02:00:14.138154 15204 tablet_copy_service.cc:161] P 5751ea0f088543bcb141b400a6131a1f: Beginning new tablet copy session on tablet 4438db9442024011bd8d5632a03d34b0 from peer 7932a656946f4546a91738584eb612e2 at {username='slave'} at 127.0.99.130:34095: session id = 7932a656946f4546a91738584eb612e2-4438db9442024011bd8d5632a03d34b0
I20240503 02:00:14.138852 15204 tablet_copy_source_session.cc:205] T 4438db9442024011bd8d5632a03d34b0 P 5751ea0f088543bcb141b400a6131a1f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:14.139600 16321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4438db9442024011bd8d5632a03d34b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:14.141471 16321 tablet_copy_client.cc:783] T 4438db9442024011bd8d5632a03d34b0 P 7932a656946f4546a91738584eb612e2: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:14.141615 16321 tablet_copy_client.cc:647] T 4438db9442024011bd8d5632a03d34b0 P 7932a656946f4546a91738584eb612e2: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:14.142738 16321 tablet_copy_client.cc:523] T 4438db9442024011bd8d5632a03d34b0 P 7932a656946f4546a91738584eb612e2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:14.143692 16321 tablet_bootstrap.cc:492] T 4438db9442024011bd8d5632a03d34b0 P 7932a656946f4546a91738584eb612e2: Bootstrap starting.
I20240503 02:00:14.160377 16321 tablet_bootstrap.cc:492] T 4438db9442024011bd8d5632a03d34b0 P 7932a656946f4546a91738584eb612e2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:14.160614 16321 tablet_bootstrap.cc:492] T 4438db9442024011bd8d5632a03d34b0 P 7932a656946f4546a91738584eb612e2: Bootstrap complete.
I20240503 02:00:14.160766 16321 ts_tablet_manager.cc:1367] T 4438db9442024011bd8d5632a03d34b0 P 7932a656946f4546a91738584eb612e2: Time spent bootstrapping tablet: real 0.017s user 0.010s sys 0.008s
I20240503 02:00:14.161239 16321 raft_consensus.cc:348] T 4438db9442024011bd8d5632a03d34b0 P 7932a656946f4546a91738584eb612e2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } attrs { replace: true } } peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } } peers { permanent_uuid: "7932a656946f4546a91738584eb612e2" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46271 } attrs { promote: true } }
I20240503 02:00:14.161350 16321 raft_consensus.cc:729] T 4438db9442024011bd8d5632a03d34b0 P 7932a656946f4546a91738584eb612e2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7932a656946f4546a91738584eb612e2, State: Initialized, Role: LEARNER
I20240503 02:00:14.161448 16321 consensus_queue.cc:260] T 4438db9442024011bd8d5632a03d34b0 P 7932a656946f4546a91738584eb612e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } attrs { replace: true } } peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } } peers { permanent_uuid: "7932a656946f4546a91738584eb612e2" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46271 } attrs { promote: true } }
I20240503 02:00:14.161666 16321 ts_tablet_manager.cc:1398] T 4438db9442024011bd8d5632a03d34b0 P 7932a656946f4546a91738584eb612e2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:14.162225 15204 tablet_copy_service.cc:342] P 5751ea0f088543bcb141b400a6131a1f: Request end of tablet copy session 7932a656946f4546a91738584eb612e2-4438db9442024011bd8d5632a03d34b0 received from {username='slave'} at 127.0.99.130:34095
I20240503 02:00:14.162310 15204 tablet_copy_service.cc:433] P 5751ea0f088543bcb141b400a6131a1f: ending tablet copy session 7932a656946f4546a91738584eb612e2-4438db9442024011bd8d5632a03d34b0 on tablet 4438db9442024011bd8d5632a03d34b0 with peer 7932a656946f4546a91738584eb612e2
I20240503 02:00:14.173848 16314 ts_tablet_manager.cc:902] T 84425c87d5964b4bb5c5ecfb72360e07 P 3f852460ab584dfb937c35f39f19f394: Initiating tablet copy from peer 5751ea0f088543bcb141b400a6131a1f (127.0.99.129:46041)
I20240503 02:00:14.174307 16314 tablet_copy_client.cc:308] T 84425c87d5964b4bb5c5ecfb72360e07 P 3f852460ab584dfb937c35f39f19f394: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:46041
I20240503 02:00:14.177515 15204 tablet_copy_service.cc:140] P 5751ea0f088543bcb141b400a6131a1f: Received BeginTabletCopySession request for tablet 84425c87d5964b4bb5c5ecfb72360e07 from peer 3f852460ab584dfb937c35f39f19f394 ({username='slave'} at 127.0.99.131:34477)
I20240503 02:00:14.177593 15204 tablet_copy_service.cc:161] P 5751ea0f088543bcb141b400a6131a1f: Beginning new tablet copy session on tablet 84425c87d5964b4bb5c5ecfb72360e07 from peer 3f852460ab584dfb937c35f39f19f394 at {username='slave'} at 127.0.99.131:34477: session id = 3f852460ab584dfb937c35f39f19f394-84425c87d5964b4bb5c5ecfb72360e07
I20240503 02:00:14.178225 15204 tablet_copy_source_session.cc:205] T 84425c87d5964b4bb5c5ecfb72360e07 P 5751ea0f088543bcb141b400a6131a1f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:14.178985 16314 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84425c87d5964b4bb5c5ecfb72360e07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:14.180622 16314 tablet_copy_client.cc:783] T 84425c87d5964b4bb5c5ecfb72360e07 P 3f852460ab584dfb937c35f39f19f394: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:14.180806 16314 tablet_copy_client.cc:647] T 84425c87d5964b4bb5c5ecfb72360e07 P 3f852460ab584dfb937c35f39f19f394: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:14.181808 16314 tablet_copy_client.cc:523] T 84425c87d5964b4bb5c5ecfb72360e07 P 3f852460ab584dfb937c35f39f19f394: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:14.182699 16314 tablet_bootstrap.cc:492] T 84425c87d5964b4bb5c5ecfb72360e07 P 3f852460ab584dfb937c35f39f19f394: Bootstrap starting.
I20240503 02:00:14.190577 16321 ts_tablet_manager.cc:902] T 7244e26e54694bd48373e8634b4daf93 P 7932a656946f4546a91738584eb612e2: Initiating tablet copy from peer 5751ea0f088543bcb141b400a6131a1f (127.0.99.129:46041)
I20240503 02:00:14.190990 16321 tablet_copy_client.cc:308] T 7244e26e54694bd48373e8634b4daf93 P 7932a656946f4546a91738584eb612e2: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:46041
I20240503 02:00:14.191416 15204 tablet_copy_service.cc:140] P 5751ea0f088543bcb141b400a6131a1f: Received BeginTabletCopySession request for tablet 7244e26e54694bd48373e8634b4daf93 from peer 7932a656946f4546a91738584eb612e2 ({username='slave'} at 127.0.99.130:34095)
I20240503 02:00:14.191509 15204 tablet_copy_service.cc:161] P 5751ea0f088543bcb141b400a6131a1f: Beginning new tablet copy session on tablet 7244e26e54694bd48373e8634b4daf93 from peer 7932a656946f4546a91738584eb612e2 at {username='slave'} at 127.0.99.130:34095: session id = 7932a656946f4546a91738584eb612e2-7244e26e54694bd48373e8634b4daf93
I20240503 02:00:14.192145 15204 tablet_copy_source_session.cc:205] T 7244e26e54694bd48373e8634b4daf93 P 5751ea0f088543bcb141b400a6131a1f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:14.192853 16321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7244e26e54694bd48373e8634b4daf93. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:14.194396 16321 tablet_copy_client.cc:783] T 7244e26e54694bd48373e8634b4daf93 P 7932a656946f4546a91738584eb612e2: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:14.194538 16321 tablet_copy_client.cc:647] T 7244e26e54694bd48373e8634b4daf93 P 7932a656946f4546a91738584eb612e2: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:14.195611 16321 tablet_copy_client.cc:523] T 7244e26e54694bd48373e8634b4daf93 P 7932a656946f4546a91738584eb612e2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:14.196576 16321 tablet_bootstrap.cc:492] T 7244e26e54694bd48373e8634b4daf93 P 7932a656946f4546a91738584eb612e2: Bootstrap starting.
I20240503 02:00:14.198807 16314 tablet_bootstrap.cc:492] T 84425c87d5964b4bb5c5ecfb72360e07 P 3f852460ab584dfb937c35f39f19f394: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:14.199051 16314 tablet_bootstrap.cc:492] T 84425c87d5964b4bb5c5ecfb72360e07 P 3f852460ab584dfb937c35f39f19f394: Bootstrap complete.
I20240503 02:00:14.199225 16314 ts_tablet_manager.cc:1367] T 84425c87d5964b4bb5c5ecfb72360e07 P 3f852460ab584dfb937c35f39f19f394: Time spent bootstrapping tablet: real 0.017s user 0.013s sys 0.002s
I20240503 02:00:14.199697 16314 raft_consensus.cc:348] T 84425c87d5964b4bb5c5ecfb72360e07 P 3f852460ab584dfb937c35f39f19f394 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } attrs { replace: true } } peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } } peers { permanent_uuid: "3f852460ab584dfb937c35f39f19f394" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34755 } attrs { promote: true } }
I20240503 02:00:14.199801 16314 raft_consensus.cc:729] T 84425c87d5964b4bb5c5ecfb72360e07 P 3f852460ab584dfb937c35f39f19f394 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3f852460ab584dfb937c35f39f19f394, State: Initialized, Role: LEARNER
I20240503 02:00:14.199896 16314 consensus_queue.cc:260] T 84425c87d5964b4bb5c5ecfb72360e07 P 3f852460ab584dfb937c35f39f19f394 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } attrs { replace: true } } peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } } peers { permanent_uuid: "3f852460ab584dfb937c35f39f19f394" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34755 } attrs { promote: true } }
I20240503 02:00:14.200156 16314 ts_tablet_manager.cc:1398] T 84425c87d5964b4bb5c5ecfb72360e07 P 3f852460ab584dfb937c35f39f19f394: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:14.200721 15204 tablet_copy_service.cc:342] P 5751ea0f088543bcb141b400a6131a1f: Request end of tablet copy session 3f852460ab584dfb937c35f39f19f394-84425c87d5964b4bb5c5ecfb72360e07 received from {username='slave'} at 127.0.99.131:34477
I20240503 02:00:14.200804 15204 tablet_copy_service.cc:433] P 5751ea0f088543bcb141b400a6131a1f: ending tablet copy session 3f852460ab584dfb937c35f39f19f394-84425c87d5964b4bb5c5ecfb72360e07 on tablet 84425c87d5964b4bb5c5ecfb72360e07 with peer 3f852460ab584dfb937c35f39f19f394
I20240503 02:00:14.213172 16321 tablet_bootstrap.cc:492] T 7244e26e54694bd48373e8634b4daf93 P 7932a656946f4546a91738584eb612e2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:14.213394 16321 tablet_bootstrap.cc:492] T 7244e26e54694bd48373e8634b4daf93 P 7932a656946f4546a91738584eb612e2: Bootstrap complete.
I20240503 02:00:14.213542 16321 ts_tablet_manager.cc:1367] T 7244e26e54694bd48373e8634b4daf93 P 7932a656946f4546a91738584eb612e2: Time spent bootstrapping tablet: real 0.017s user 0.017s sys 0.000s
I20240503 02:00:14.213867 16321 raft_consensus.cc:348] T 7244e26e54694bd48373e8634b4daf93 P 7932a656946f4546a91738584eb612e2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } } peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } attrs { replace: true } } peers { permanent_uuid: "7932a656946f4546a91738584eb612e2" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46271 } attrs { promote: true } }
I20240503 02:00:14.213958 16321 raft_consensus.cc:729] T 7244e26e54694bd48373e8634b4daf93 P 7932a656946f4546a91738584eb612e2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7932a656946f4546a91738584eb612e2, State: Initialized, Role: LEARNER
I20240503 02:00:14.214025 16321 consensus_queue.cc:260] T 7244e26e54694bd48373e8634b4daf93 P 7932a656946f4546a91738584eb612e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } } peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } attrs { replace: true } } peers { permanent_uuid: "7932a656946f4546a91738584eb612e2" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46271 } attrs { promote: true } }
I20240503 02:00:14.214247 16321 ts_tablet_manager.cc:1398] T 7244e26e54694bd48373e8634b4daf93 P 7932a656946f4546a91738584eb612e2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:14.214723 15204 tablet_copy_service.cc:342] P 5751ea0f088543bcb141b400a6131a1f: Request end of tablet copy session 7932a656946f4546a91738584eb612e2-7244e26e54694bd48373e8634b4daf93 received from {username='slave'} at 127.0.99.130:34095
I20240503 02:00:14.214793 15204 tablet_copy_service.cc:433] P 5751ea0f088543bcb141b400a6131a1f: ending tablet copy session 7932a656946f4546a91738584eb612e2-7244e26e54694bd48373e8634b4daf93 on tablet 7244e26e54694bd48373e8634b4daf93 with peer 7932a656946f4546a91738584eb612e2
I20240503 02:00:14.405897 16018 raft_consensus.cc:1203] T f357df7d66a646fc817b19b28306a5a0 P 3f852460ab584dfb937c35f39f19f394 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:00:14.559046 16018 raft_consensus.cc:1203] T 595c6713bec0492882c1e58225dcce42 P 3f852460ab584dfb937c35f39f19f394 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:00:14.651794 15887 raft_consensus.cc:1203] T 7244e26e54694bd48373e8634b4daf93 P 7932a656946f4546a91738584eb612e2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:00:14.656581 15887 raft_consensus.cc:1203] T f72aa769aa8f465ba4469dfc34ae35c1 P 7932a656946f4546a91738584eb612e2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:00:14.666043 15887 raft_consensus.cc:1203] T 4438db9442024011bd8d5632a03d34b0 P 7932a656946f4546a91738584eb612e2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:00:14.787158 16018 raft_consensus.cc:1203] T 84425c87d5964b4bb5c5ecfb72360e07 P 3f852460ab584dfb937c35f39f19f394 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:00:14.954427 16406 raft_consensus.cc:1050] T f357df7d66a646fc817b19b28306a5a0 P 17eb0a665dc648d9a5ad2c62930861c4: attempting to promote NON_VOTER 3f852460ab584dfb937c35f39f19f394 to VOTER
I20240503 02:00:14.954905 16406 consensus_queue.cc:237] T f357df7d66a646fc817b19b28306a5a0 P 17eb0a665dc648d9a5ad2c62930861c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } } peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } attrs { replace: true } } peers { permanent_uuid: "3f852460ab584dfb937c35f39f19f394" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34755 } attrs { promote: false } }
I20240503 02:00:14.956132 15184 raft_consensus.cc:1261] T f357df7d66a646fc817b19b28306a5a0 P 5751ea0f088543bcb141b400a6131a1f [term 1 FOLLOWER]: Refusing update from remote peer 17eb0a665dc648d9a5ad2c62930861c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:14.956218 16018 raft_consensus.cc:1261] T f357df7d66a646fc817b19b28306a5a0 P 3f852460ab584dfb937c35f39f19f394 [term 1 LEARNER]: Refusing update from remote peer 17eb0a665dc648d9a5ad2c62930861c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:14.956539 16407 consensus_queue.cc:1035] T f357df7d66a646fc817b19b28306a5a0 P 17eb0a665dc648d9a5ad2c62930861c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:14.956763 16205 consensus_queue.cc:1035] T f357df7d66a646fc817b19b28306a5a0 P 17eb0a665dc648d9a5ad2c62930861c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f852460ab584dfb937c35f39f19f394" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34755 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:14.958633 16205 raft_consensus.cc:2932] T f357df7d66a646fc817b19b28306a5a0 P 17eb0a665dc648d9a5ad2c62930861c4 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3f852460ab584dfb937c35f39f19f394 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } } peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } attrs { replace: true } } peers { permanent_uuid: "3f852460ab584dfb937c35f39f19f394" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34755 } attrs { promote: false } } }
I20240503 02:00:14.959184 15184 raft_consensus.cc:2932] T f357df7d66a646fc817b19b28306a5a0 P 5751ea0f088543bcb141b400a6131a1f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3f852460ab584dfb937c35f39f19f394 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } } peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } attrs { replace: true } } peers { permanent_uuid: "3f852460ab584dfb937c35f39f19f394" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34755 } attrs { promote: false } } }
I20240503 02:00:14.960233 16018 raft_consensus.cc:2932] T f357df7d66a646fc817b19b28306a5a0 P 3f852460ab584dfb937c35f39f19f394 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3f852460ab584dfb937c35f39f19f394 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } } peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } attrs { replace: true } } peers { permanent_uuid: "3f852460ab584dfb937c35f39f19f394" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34755 } attrs { promote: false } } }
I20240503 02:00:14.962724 15039 catalog_manager.cc:5484] T f357df7d66a646fc817b19b28306a5a0 P 3f852460ab584dfb937c35f39f19f394 reported cstate change: config changed from index 4 to 5, 3f852460ab584dfb937c35f39f19f394 (127.0.99.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "17eb0a665dc648d9a5ad2c62930861c4" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } } peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } attrs { replace: true } } peers { permanent_uuid: "3f852460ab584dfb937c35f39f19f394" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34755 } attrs { promote: false } } }
I20240503 02:00:14.966858 15580 consensus_queue.cc:237] T f357df7d66a646fc817b19b28306a5a0 P 17eb0a665dc648d9a5ad2c62930861c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } } peers { permanent_uuid: "3f852460ab584dfb937c35f39f19f394" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34755 } attrs { promote: false } }
I20240503 02:00:14.967887 16018 raft_consensus.cc:1261] T f357df7d66a646fc817b19b28306a5a0 P 3f852460ab584dfb937c35f39f19f394 [term 1 FOLLOWER]: Refusing update from remote peer 17eb0a665dc648d9a5ad2c62930861c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:14.968250 16406 consensus_queue.cc:1035] T f357df7d66a646fc817b19b28306a5a0 P 17eb0a665dc648d9a5ad2c62930861c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f852460ab584dfb937c35f39f19f394" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34755 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:14.969749 16407 raft_consensus.cc:2932] T f357df7d66a646fc817b19b28306a5a0 P 17eb0a665dc648d9a5ad2c62930861c4 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 5751ea0f088543bcb141b400a6131a1f (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } } peers { permanent_uuid: "3f852460ab584dfb937c35f39f19f394" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34755 } attrs { promote: false } } }
I20240503 02:00:14.970166 16018 raft_consensus.cc:2932] T f357df7d66a646fc817b19b28306a5a0 P 3f852460ab584dfb937c35f39f19f394 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 5751ea0f088543bcb141b400a6131a1f (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } } peers { permanent_uuid: "3f852460ab584dfb937c35f39f19f394" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34755 } attrs { promote: false } } }
I20240503 02:00:14.971678 15029 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f357df7d66a646fc817b19b28306a5a0 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:00:14.972347 15038 catalog_manager.cc:5484] T f357df7d66a646fc817b19b28306a5a0 P 17eb0a665dc648d9a5ad2c62930861c4 reported cstate change: config changed from index 5 to 6, VOTER 5751ea0f088543bcb141b400a6131a1f (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "17eb0a665dc648d9a5ad2c62930861c4" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3f852460ab584dfb937c35f39f19f394" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34755 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:00:14.974617 15164 tablet_service.cc:1514] Processing DeleteTablet for tablet f357df7d66a646fc817b19b28306a5a0 with delete_type TABLET_DATA_TOMBSTONED (TS 5751ea0f088543bcb141b400a6131a1f not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:48148
I20240503 02:00:14.975106 16477 tablet_replica.cc:330] T f357df7d66a646fc817b19b28306a5a0 P 5751ea0f088543bcb141b400a6131a1f: stopping tablet replica
I20240503 02:00:14.975409 16477 raft_consensus.cc:2229] T f357df7d66a646fc817b19b28306a5a0 P 5751ea0f088543bcb141b400a6131a1f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:14.975631 16477 raft_consensus.cc:2258] T f357df7d66a646fc817b19b28306a5a0 P 5751ea0f088543bcb141b400a6131a1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:14.976612 16477 ts_tablet_manager.cc:1875] T f357df7d66a646fc817b19b28306a5a0 P 5751ea0f088543bcb141b400a6131a1f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:14.978420 16477 ts_tablet_manager.cc:1888] T f357df7d66a646fc817b19b28306a5a0 P 5751ea0f088543bcb141b400a6131a1f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:00:14.978505 16477 log.cc:1196] T f357df7d66a646fc817b19b28306a5a0 P 5751ea0f088543bcb141b400a6131a1f: Deleting WAL directory at /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/f357df7d66a646fc817b19b28306a5a0
I20240503 02:00:14.979107 15026 catalog_manager.cc:4830] TS 5751ea0f088543bcb141b400a6131a1f (127.0.99.129:46041): tablet f357df7d66a646fc817b19b28306a5a0 (table rebalance_test_table_2 [id=1fdb6fbbdfc14fe7bf1064da99b35275]) successfully deleted
I20240503 02:00:15.039393 16438 raft_consensus.cc:1050] T f72aa769aa8f465ba4469dfc34ae35c1 P 5751ea0f088543bcb141b400a6131a1f: attempting to promote NON_VOTER 7932a656946f4546a91738584eb612e2 to VOTER
I20240503 02:00:15.039842 16438 consensus_queue.cc:237] T f72aa769aa8f465ba4469dfc34ae35c1 P 5751ea0f088543bcb141b400a6131a1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } } peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } attrs { replace: true } } peers { permanent_uuid: "7932a656946f4546a91738584eb612e2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46271 } attrs { promote: false } }
I20240503 02:00:15.041226 15580 raft_consensus.cc:1261] T f72aa769aa8f465ba4469dfc34ae35c1 P 17eb0a665dc648d9a5ad2c62930861c4 [term 1 FOLLOWER]: Refusing update from remote peer 5751ea0f088543bcb141b400a6131a1f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:15.041676 15887 raft_consensus.cc:1261] T f72aa769aa8f465ba4469dfc34ae35c1 P 7932a656946f4546a91738584eb612e2 [term 1 LEARNER]: Refusing update from remote peer 5751ea0f088543bcb141b400a6131a1f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:15.042137 16217 consensus_queue.cc:1035] T f72aa769aa8f465ba4469dfc34ae35c1 P 5751ea0f088543bcb141b400a6131a1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7932a656946f4546a91738584eb612e2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46271 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:15.042348 16439 consensus_queue.cc:1035] T f72aa769aa8f465ba4469dfc34ae35c1 P 5751ea0f088543bcb141b400a6131a1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:15.044198 16217 raft_consensus.cc:2932] T f72aa769aa8f465ba4469dfc34ae35c1 P 5751ea0f088543bcb141b400a6131a1f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7932a656946f4546a91738584eb612e2 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } } peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } attrs { replace: true } } peers { permanent_uuid: "7932a656946f4546a91738584eb612e2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46271 } attrs { promote: false } } }
I20240503 02:00:15.046429 15578 raft_consensus.cc:2932] T f72aa769aa8f465ba4469dfc34ae35c1 P 17eb0a665dc648d9a5ad2c62930861c4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7932a656946f4546a91738584eb612e2 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } } peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } attrs { replace: true } } peers { permanent_uuid: "7932a656946f4546a91738584eb612e2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46271 } attrs { promote: false } } }
I20240503 02:00:15.046828 15887 raft_consensus.cc:2932] T f72aa769aa8f465ba4469dfc34ae35c1 P 7932a656946f4546a91738584eb612e2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7932a656946f4546a91738584eb612e2 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } } peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } attrs { replace: true } } peers { permanent_uuid: "7932a656946f4546a91738584eb612e2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46271 } attrs { promote: false } } }
I20240503 02:00:15.049407 15034 catalog_manager.cc:5484] T f72aa769aa8f465ba4469dfc34ae35c1 P 17eb0a665dc648d9a5ad2c62930861c4 reported cstate change: config changed from index 4 to 5, 7932a656946f4546a91738584eb612e2 (127.0.99.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5751ea0f088543bcb141b400a6131a1f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } } peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } attrs { replace: true } } peers { permanent_uuid: "7932a656946f4546a91738584eb612e2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46271 } attrs { promote: false } } }
I20240503 02:00:15.052536 15184 consensus_queue.cc:237] T f72aa769aa8f465ba4469dfc34ae35c1 P 5751ea0f088543bcb141b400a6131a1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } } peers { permanent_uuid: "7932a656946f4546a91738584eb612e2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46271 } attrs { promote: false } }
I20240503 02:00:15.053520 15887 raft_consensus.cc:1261] T f72aa769aa8f465ba4469dfc34ae35c1 P 7932a656946f4546a91738584eb612e2 [term 1 FOLLOWER]: Refusing update from remote peer 5751ea0f088543bcb141b400a6131a1f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:15.053992 16438 consensus_queue.cc:1035] T f72aa769aa8f465ba4469dfc34ae35c1 P 5751ea0f088543bcb141b400a6131a1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7932a656946f4546a91738584eb612e2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46271 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:15.055492 16439 raft_consensus.cc:2932] T f72aa769aa8f465ba4469dfc34ae35c1 P 5751ea0f088543bcb141b400a6131a1f [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 17eb0a665dc648d9a5ad2c62930861c4 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } } peers { permanent_uuid: "7932a656946f4546a91738584eb612e2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46271 } attrs { promote: false } } }
I20240503 02:00:15.055904 15887 raft_consensus.cc:2932] T f72aa769aa8f465ba4469dfc34ae35c1 P 7932a656946f4546a91738584eb612e2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 17eb0a665dc648d9a5ad2c62930861c4 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } } peers { permanent_uuid: "7932a656946f4546a91738584eb612e2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46271 } attrs { promote: false } } }
I20240503 02:00:15.057230 15026 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f72aa769aa8f465ba4469dfc34ae35c1 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:00:15.058120 15038 catalog_manager.cc:5484] T f72aa769aa8f465ba4469dfc34ae35c1 P 5751ea0f088543bcb141b400a6131a1f reported cstate change: config changed from index 5 to 6, VOTER 17eb0a665dc648d9a5ad2c62930861c4 (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "5751ea0f088543bcb141b400a6131a1f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7932a656946f4546a91738584eb612e2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46271 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:00:15.060577 15560 tablet_service.cc:1514] Processing DeleteTablet for tablet f72aa769aa8f465ba4469dfc34ae35c1 with delete_type TABLET_DATA_TOMBSTONED (TS 17eb0a665dc648d9a5ad2c62930861c4 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:38394
I20240503 02:00:15.061307 16506 tablet_replica.cc:330] T f72aa769aa8f465ba4469dfc34ae35c1 P 17eb0a665dc648d9a5ad2c62930861c4: stopping tablet replica
I20240503 02:00:15.061807 16506 raft_consensus.cc:2229] T f72aa769aa8f465ba4469dfc34ae35c1 P 17eb0a665dc648d9a5ad2c62930861c4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:15.062091 16506 raft_consensus.cc:2258] T f72aa769aa8f465ba4469dfc34ae35c1 P 17eb0a665dc648d9a5ad2c62930861c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:15.063097 16506 ts_tablet_manager.cc:1875] T f72aa769aa8f465ba4469dfc34ae35c1 P 17eb0a665dc648d9a5ad2c62930861c4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:15.065016 16506 ts_tablet_manager.cc:1888] T f72aa769aa8f465ba4469dfc34ae35c1 P 17eb0a665dc648d9a5ad2c62930861c4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:00:15.065090 16506 log.cc:1196] T f72aa769aa8f465ba4469dfc34ae35c1 P 17eb0a665dc648d9a5ad2c62930861c4: Deleting WAL directory at /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/f72aa769aa8f465ba4469dfc34ae35c1
I20240503 02:00:15.065577 15029 catalog_manager.cc:4830] TS 17eb0a665dc648d9a5ad2c62930861c4 (127.0.99.132:33559): tablet f72aa769aa8f465ba4469dfc34ae35c1 (table rebalance_test_table_2 [id=1fdb6fbbdfc14fe7bf1064da99b35275]) successfully deleted
I20240503 02:00:15.067039 16438 raft_consensus.cc:1050] T 4438db9442024011bd8d5632a03d34b0 P 5751ea0f088543bcb141b400a6131a1f: attempting to promote NON_VOTER 7932a656946f4546a91738584eb612e2 to VOTER
I20240503 02:00:15.067373 16438 consensus_queue.cc:237] T 4438db9442024011bd8d5632a03d34b0 P 5751ea0f088543bcb141b400a6131a1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } attrs { replace: true } } peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } } peers { permanent_uuid: "7932a656946f4546a91738584eb612e2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46271 } attrs { promote: false } }
I20240503 02:00:15.068567 15887 raft_consensus.cc:1261] T 4438db9442024011bd8d5632a03d34b0 P 7932a656946f4546a91738584eb612e2 [term 1 LEARNER]: Refusing update from remote peer 5751ea0f088543bcb141b400a6131a1f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:15.068598 15578 raft_consensus.cc:1261] T 4438db9442024011bd8d5632a03d34b0 P 17eb0a665dc648d9a5ad2c62930861c4 [term 1 FOLLOWER]: Refusing update from remote peer 5751ea0f088543bcb141b400a6131a1f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:15.068998 16217 consensus_queue.cc:1035] T 4438db9442024011bd8d5632a03d34b0 P 5751ea0f088543bcb141b400a6131a1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7932a656946f4546a91738584eb612e2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46271 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:15.069375 16439 consensus_queue.cc:1035] T 4438db9442024011bd8d5632a03d34b0 P 5751ea0f088543bcb141b400a6131a1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:15.070979 16217 raft_consensus.cc:2932] T 4438db9442024011bd8d5632a03d34b0 P 5751ea0f088543bcb141b400a6131a1f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7932a656946f4546a91738584eb612e2 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } attrs { replace: true } } peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } } peers { permanent_uuid: "7932a656946f4546a91738584eb612e2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46271 } attrs { promote: false } } }
I20240503 02:00:15.071378 15578 raft_consensus.cc:2932] T 4438db9442024011bd8d5632a03d34b0 P 17eb0a665dc648d9a5ad2c62930861c4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7932a656946f4546a91738584eb612e2 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } attrs { replace: true } } peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } } peers { permanent_uuid: "7932a656946f4546a91738584eb612e2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46271 } attrs { promote: false } } }
I20240503 02:00:15.071489 15887 raft_consensus.cc:2932] T 4438db9442024011bd8d5632a03d34b0 P 7932a656946f4546a91738584eb612e2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7932a656946f4546a91738584eb612e2 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } attrs { replace: true } } peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } } peers { permanent_uuid: "7932a656946f4546a91738584eb612e2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46271 } attrs { promote: false } } }
I20240503 02:00:15.073736 15039 catalog_manager.cc:5484] T 4438db9442024011bd8d5632a03d34b0 P 7932a656946f4546a91738584eb612e2 reported cstate change: config changed from index 4 to 5, 7932a656946f4546a91738584eb612e2 (127.0.99.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5751ea0f088543bcb141b400a6131a1f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } attrs { replace: true } } peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } } peers { permanent_uuid: "7932a656946f4546a91738584eb612e2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46271 } attrs { promote: false } } }
I20240503 02:00:15.088342 15184 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4438db9442024011bd8d5632a03d34b0"
dest_uuid: "5751ea0f088543bcb141b400a6131a1f"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:48532
I20240503 02:00:15.088500 15184 raft_consensus.cc:595] T 4438db9442024011bd8d5632a03d34b0 P 5751ea0f088543bcb141b400a6131a1f [term 1 LEADER]: Received request to transfer leadership
I20240503 02:00:15.118163 15184 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4438db9442024011bd8d5632a03d34b0"
dest_uuid: "5751ea0f088543bcb141b400a6131a1f"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:48572
I20240503 02:00:15.118288 15184 raft_consensus.cc:595] T 4438db9442024011bd8d5632a03d34b0 P 5751ea0f088543bcb141b400a6131a1f [term 1 LEADER]: Received request to transfer leadership
I20240503 02:00:15.176856 16205 raft_consensus.cc:1050] T 595c6713bec0492882c1e58225dcce42 P 17eb0a665dc648d9a5ad2c62930861c4: attempting to promote NON_VOTER 3f852460ab584dfb937c35f39f19f394 to VOTER
I20240503 02:00:15.177209 16205 consensus_queue.cc:237] T 595c6713bec0492882c1e58225dcce42 P 17eb0a665dc648d9a5ad2c62930861c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } } peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } attrs { replace: true } } peers { permanent_uuid: "3f852460ab584dfb937c35f39f19f394" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34755 } attrs { promote: false } }
I20240503 02:00:15.178328 15184 raft_consensus.cc:1261] T 595c6713bec0492882c1e58225dcce42 P 5751ea0f088543bcb141b400a6131a1f [term 1 FOLLOWER]: Refusing update from remote peer 17eb0a665dc648d9a5ad2c62930861c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:15.178355 16018 raft_consensus.cc:1261] T 595c6713bec0492882c1e58225dcce42 P 3f852460ab584dfb937c35f39f19f394 [term 1 LEARNER]: Refusing update from remote peer 17eb0a665dc648d9a5ad2c62930861c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:15.178761 16407 consensus_queue.cc:1035] T 595c6713bec0492882c1e58225dcce42 P 17eb0a665dc648d9a5ad2c62930861c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:15.178958 16406 consensus_queue.cc:1035] T 595c6713bec0492882c1e58225dcce42 P 17eb0a665dc648d9a5ad2c62930861c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f852460ab584dfb937c35f39f19f394" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34755 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:15.180721 16205 raft_consensus.cc:2932] T 595c6713bec0492882c1e58225dcce42 P 17eb0a665dc648d9a5ad2c62930861c4 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3f852460ab584dfb937c35f39f19f394 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } } peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } attrs { replace: true } } peers { permanent_uuid: "3f852460ab584dfb937c35f39f19f394" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34755 } attrs { promote: false } } }
I20240503 02:00:15.181182 15184 raft_consensus.cc:2932] T 595c6713bec0492882c1e58225dcce42 P 5751ea0f088543bcb141b400a6131a1f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3f852460ab584dfb937c35f39f19f394 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } } peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } attrs { replace: true } } peers { permanent_uuid: "3f852460ab584dfb937c35f39f19f394" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34755 } attrs { promote: false } } }
I20240503 02:00:15.181267 16018 raft_consensus.cc:2932] T 595c6713bec0492882c1e58225dcce42 P 3f852460ab584dfb937c35f39f19f394 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3f852460ab584dfb937c35f39f19f394 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } } peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } attrs { replace: true } } peers { permanent_uuid: "3f852460ab584dfb937c35f39f19f394" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34755 } attrs { promote: false } } }
I20240503 02:00:15.183115 15038 catalog_manager.cc:5484] T 595c6713bec0492882c1e58225dcce42 P 17eb0a665dc648d9a5ad2c62930861c4 reported cstate change: config changed from index 4 to 5, 3f852460ab584dfb937c35f39f19f394 (127.0.99.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "17eb0a665dc648d9a5ad2c62930861c4" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3f852460ab584dfb937c35f39f19f394" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34755 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:00:15.186194 15578 consensus_queue.cc:237] T 595c6713bec0492882c1e58225dcce42 P 17eb0a665dc648d9a5ad2c62930861c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } } peers { permanent_uuid: "3f852460ab584dfb937c35f39f19f394" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34755 } attrs { promote: false } }
I20240503 02:00:15.187093 16018 raft_consensus.cc:1261] T 595c6713bec0492882c1e58225dcce42 P 3f852460ab584dfb937c35f39f19f394 [term 1 FOLLOWER]: Refusing update from remote peer 17eb0a665dc648d9a5ad2c62930861c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:15.187458 16205 consensus_queue.cc:1035] T 595c6713bec0492882c1e58225dcce42 P 17eb0a665dc648d9a5ad2c62930861c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f852460ab584dfb937c35f39f19f394" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34755 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:15.188927 16406 raft_consensus.cc:2932] T 595c6713bec0492882c1e58225dcce42 P 17eb0a665dc648d9a5ad2c62930861c4 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 5751ea0f088543bcb141b400a6131a1f (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } } peers { permanent_uuid: "3f852460ab584dfb937c35f39f19f394" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34755 } attrs { promote: false } } }
I20240503 02:00:15.189383 16018 raft_consensus.cc:2932] T 595c6713bec0492882c1e58225dcce42 P 3f852460ab584dfb937c35f39f19f394 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 5751ea0f088543bcb141b400a6131a1f (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } } peers { permanent_uuid: "3f852460ab584dfb937c35f39f19f394" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34755 } attrs { promote: false } } }
I20240503 02:00:15.190241 15029 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 595c6713bec0492882c1e58225dcce42 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:00:15.191183 15039 catalog_manager.cc:5484] T 595c6713bec0492882c1e58225dcce42 P 3f852460ab584dfb937c35f39f19f394 reported cstate change: config changed from index 5 to 6, VOTER 5751ea0f088543bcb141b400a6131a1f (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "17eb0a665dc648d9a5ad2c62930861c4" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } } peers { permanent_uuid: "3f852460ab584dfb937c35f39f19f394" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34755 } attrs { promote: false } } }
I20240503 02:00:15.193215 15164 tablet_service.cc:1514] Processing DeleteTablet for tablet 595c6713bec0492882c1e58225dcce42 with delete_type TABLET_DATA_TOMBSTONED (TS 5751ea0f088543bcb141b400a6131a1f not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:48148
I20240503 02:00:15.193430 16477 tablet_replica.cc:330] T 595c6713bec0492882c1e58225dcce42 P 5751ea0f088543bcb141b400a6131a1f: stopping tablet replica
I20240503 02:00:15.193575 16477 raft_consensus.cc:2229] T 595c6713bec0492882c1e58225dcce42 P 5751ea0f088543bcb141b400a6131a1f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:15.193735 16477 raft_consensus.cc:2258] T 595c6713bec0492882c1e58225dcce42 P 5751ea0f088543bcb141b400a6131a1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:15.194321 16477 ts_tablet_manager.cc:1875] T 595c6713bec0492882c1e58225dcce42 P 5751ea0f088543bcb141b400a6131a1f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:15.195894 16477 ts_tablet_manager.cc:1888] T 595c6713bec0492882c1e58225dcce42 P 5751ea0f088543bcb141b400a6131a1f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:00:15.195962 16477 log.cc:1196] T 595c6713bec0492882c1e58225dcce42 P 5751ea0f088543bcb141b400a6131a1f: Deleting WAL directory at /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/595c6713bec0492882c1e58225dcce42
I20240503 02:00:15.196441 15026 catalog_manager.cc:4830] TS 5751ea0f088543bcb141b400a6131a1f (127.0.99.129:46041): tablet 595c6713bec0492882c1e58225dcce42 (table rebalance_test_table_1 [id=174f433588294c61856c430b17973e89]) successfully deleted
I20240503 02:00:15.249573 16439 raft_consensus.cc:1050] T 84425c87d5964b4bb5c5ecfb72360e07 P 5751ea0f088543bcb141b400a6131a1f: attempting to promote NON_VOTER 3f852460ab584dfb937c35f39f19f394 to VOTER
I20240503 02:00:15.249953 16439 consensus_queue.cc:237] T 84425c87d5964b4bb5c5ecfb72360e07 P 5751ea0f088543bcb141b400a6131a1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } attrs { replace: true } } peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } } peers { permanent_uuid: "3f852460ab584dfb937c35f39f19f394" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34755 } attrs { promote: false } }
I20240503 02:00:15.251137 15578 raft_consensus.cc:1261] T 84425c87d5964b4bb5c5ecfb72360e07 P 17eb0a665dc648d9a5ad2c62930861c4 [term 1 FOLLOWER]: Refusing update from remote peer 5751ea0f088543bcb141b400a6131a1f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:15.251483 16018 raft_consensus.cc:1261] T 84425c87d5964b4bb5c5ecfb72360e07 P 3f852460ab584dfb937c35f39f19f394 [term 1 LEARNER]: Refusing update from remote peer 5751ea0f088543bcb141b400a6131a1f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:15.251549 16438 consensus_queue.cc:1035] T 84425c87d5964b4bb5c5ecfb72360e07 P 5751ea0f088543bcb141b400a6131a1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:15.251852 16439 consensus_queue.cc:1035] T 84425c87d5964b4bb5c5ecfb72360e07 P 5751ea0f088543bcb141b400a6131a1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f852460ab584dfb937c35f39f19f394" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34755 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:15.253337 16439 raft_consensus.cc:2932] T 84425c87d5964b4bb5c5ecfb72360e07 P 5751ea0f088543bcb141b400a6131a1f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3f852460ab584dfb937c35f39f19f394 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } attrs { replace: true } } peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } } peers { permanent_uuid: "3f852460ab584dfb937c35f39f19f394" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34755 } attrs { promote: false } } }
I20240503 02:00:15.253481 16438 raft_consensus.cc:1050] T 7244e26e54694bd48373e8634b4daf93 P 5751ea0f088543bcb141b400a6131a1f: attempting to promote NON_VOTER 7932a656946f4546a91738584eb612e2 to VOTER
I20240503 02:00:15.253801 16438 consensus_queue.cc:237] T 7244e26e54694bd48373e8634b4daf93 P 5751ea0f088543bcb141b400a6131a1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } } peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } attrs { replace: true } } peers { permanent_uuid: "7932a656946f4546a91738584eb612e2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46271 } attrs { promote: false } }
I20240503 02:00:15.253870 15578 raft_consensus.cc:2932] T 84425c87d5964b4bb5c5ecfb72360e07 P 17eb0a665dc648d9a5ad2c62930861c4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3f852460ab584dfb937c35f39f19f394 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } attrs { replace: true } } peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } } peers { permanent_uuid: "3f852460ab584dfb937c35f39f19f394" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34755 } attrs { promote: false } } }
I20240503 02:00:15.255024 16018 raft_consensus.cc:2932] T 84425c87d5964b4bb5c5ecfb72360e07 P 3f852460ab584dfb937c35f39f19f394 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3f852460ab584dfb937c35f39f19f394 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } attrs { replace: true } } peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } } peers { permanent_uuid: "3f852460ab584dfb937c35f39f19f394" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34755 } attrs { promote: false } } }
I20240503 02:00:15.257259 15034 catalog_manager.cc:5484] T 84425c87d5964b4bb5c5ecfb72360e07 P 3f852460ab584dfb937c35f39f19f394 reported cstate change: config changed from index 4 to 5, 3f852460ab584dfb937c35f39f19f394 (127.0.99.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5751ea0f088543bcb141b400a6131a1f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } attrs { replace: true } } peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } } peers { permanent_uuid: "3f852460ab584dfb937c35f39f19f394" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34755 } attrs { promote: false } } }
I20240503 02:00:15.257566 15887 raft_consensus.cc:1261] T 7244e26e54694bd48373e8634b4daf93 P 7932a656946f4546a91738584eb612e2 [term 1 LEARNER]: Refusing update from remote peer 5751ea0f088543bcb141b400a6131a1f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:15.258013 16438 consensus_queue.cc:1035] T 7244e26e54694bd48373e8634b4daf93 P 5751ea0f088543bcb141b400a6131a1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7932a656946f4546a91738584eb612e2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46271 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:15.258460 15578 raft_consensus.cc:1261] T 7244e26e54694bd48373e8634b4daf93 P 17eb0a665dc648d9a5ad2c62930861c4 [term 1 FOLLOWER]: Refusing update from remote peer 5751ea0f088543bcb141b400a6131a1f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:15.258767 16438 consensus_queue.cc:1035] T 7244e26e54694bd48373e8634b4daf93 P 5751ea0f088543bcb141b400a6131a1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:15.259948 16438 raft_consensus.cc:2932] T 7244e26e54694bd48373e8634b4daf93 P 5751ea0f088543bcb141b400a6131a1f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7932a656946f4546a91738584eb612e2 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } } peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } attrs { replace: true } } peers { permanent_uuid: "7932a656946f4546a91738584eb612e2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46271 } attrs { promote: false } } }
I20240503 02:00:15.260715 15184 consensus_queue.cc:237] T 84425c87d5964b4bb5c5ecfb72360e07 P 5751ea0f088543bcb141b400a6131a1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } } peers { permanent_uuid: "3f852460ab584dfb937c35f39f19f394" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34755 } attrs { promote: false } }
I20240503 02:00:15.261349 15887 raft_consensus.cc:2932] T 7244e26e54694bd48373e8634b4daf93 P 7932a656946f4546a91738584eb612e2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7932a656946f4546a91738584eb612e2 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } } peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } attrs { replace: true } } peers { permanent_uuid: "7932a656946f4546a91738584eb612e2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46271 } attrs { promote: false } } }
I20240503 02:00:15.262339 15038 catalog_manager.cc:5484] T 7244e26e54694bd48373e8634b4daf93 P 5751ea0f088543bcb141b400a6131a1f reported cstate change: config changed from index 4 to 5, 7932a656946f4546a91738584eb612e2 (127.0.99.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5751ea0f088543bcb141b400a6131a1f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7932a656946f4546a91738584eb612e2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46271 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:00:15.262789 16018 raft_consensus.cc:1261] T 84425c87d5964b4bb5c5ecfb72360e07 P 3f852460ab584dfb937c35f39f19f394 [term 1 FOLLOWER]: Refusing update from remote peer 5751ea0f088543bcb141b400a6131a1f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:15.263200 16439 consensus_queue.cc:1035] T 84425c87d5964b4bb5c5ecfb72360e07 P 5751ea0f088543bcb141b400a6131a1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f852460ab584dfb937c35f39f19f394" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34755 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:15.264727 15184 consensus_queue.cc:237] T 7244e26e54694bd48373e8634b4daf93 P 5751ea0f088543bcb141b400a6131a1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } } peers { permanent_uuid: "7932a656946f4546a91738584eb612e2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46271 } attrs { promote: false } }
I20240503 02:00:15.264967 16579 raft_consensus.cc:2932] T 84425c87d5964b4bb5c5ecfb72360e07 P 5751ea0f088543bcb141b400a6131a1f [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 17eb0a665dc648d9a5ad2c62930861c4 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } } peers { permanent_uuid: "3f852460ab584dfb937c35f39f19f394" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34755 } attrs { promote: false } } }
I20240503 02:00:15.265345 16018 raft_consensus.cc:2932] T 84425c87d5964b4bb5c5ecfb72360e07 P 3f852460ab584dfb937c35f39f19f394 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 17eb0a665dc648d9a5ad2c62930861c4 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } } peers { permanent_uuid: "3f852460ab584dfb937c35f39f19f394" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34755 } attrs { promote: false } } }
I20240503 02:00:15.265866 15887 raft_consensus.cc:1261] T 7244e26e54694bd48373e8634b4daf93 P 7932a656946f4546a91738584eb612e2 [term 1 FOLLOWER]: Refusing update from remote peer 5751ea0f088543bcb141b400a6131a1f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:15.266261 16439 consensus_queue.cc:1035] T 7244e26e54694bd48373e8634b4daf93 P 5751ea0f088543bcb141b400a6131a1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7932a656946f4546a91738584eb612e2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46271 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:15.266903 15026 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 84425c87d5964b4bb5c5ecfb72360e07 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:00:15.267428 15039 catalog_manager.cc:5484] T 84425c87d5964b4bb5c5ecfb72360e07 P 3f852460ab584dfb937c35f39f19f394 reported cstate change: config changed from index 5 to 6, VOTER 17eb0a665dc648d9a5ad2c62930861c4 (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "5751ea0f088543bcb141b400a6131a1f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } } peers { permanent_uuid: "3f852460ab584dfb937c35f39f19f394" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34755 } attrs { promote: false } } }
I20240503 02:00:15.268363 16579 raft_consensus.cc:2932] T 7244e26e54694bd48373e8634b4daf93 P 5751ea0f088543bcb141b400a6131a1f [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 17eb0a665dc648d9a5ad2c62930861c4 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } } peers { permanent_uuid: "7932a656946f4546a91738584eb612e2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46271 } attrs { promote: false } } }
I20240503 02:00:15.268811 15887 raft_consensus.cc:2932] T 7244e26e54694bd48373e8634b4daf93 P 7932a656946f4546a91738584eb612e2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 17eb0a665dc648d9a5ad2c62930861c4 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } } peers { permanent_uuid: "7932a656946f4546a91738584eb612e2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46271 } attrs { promote: false } } }
I20240503 02:00:15.269732 15560 tablet_service.cc:1514] Processing DeleteTablet for tablet 84425c87d5964b4bb5c5ecfb72360e07 with delete_type TABLET_DATA_TOMBSTONED (TS 17eb0a665dc648d9a5ad2c62930861c4 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:38394
I20240503 02:00:15.269953 16506 tablet_replica.cc:330] T 84425c87d5964b4bb5c5ecfb72360e07 P 17eb0a665dc648d9a5ad2c62930861c4: stopping tablet replica
I20240503 02:00:15.270077 16506 raft_consensus.cc:2229] T 84425c87d5964b4bb5c5ecfb72360e07 P 17eb0a665dc648d9a5ad2c62930861c4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:15.270211 16506 raft_consensus.cc:2258] T 84425c87d5964b4bb5c5ecfb72360e07 P 17eb0a665dc648d9a5ad2c62930861c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:15.270814 15038 catalog_manager.cc:5484] T 7244e26e54694bd48373e8634b4daf93 P 7932a656946f4546a91738584eb612e2 reported cstate change: config changed from index 5 to 6, VOTER 17eb0a665dc648d9a5ad2c62930861c4 (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "5751ea0f088543bcb141b400a6131a1f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } } peers { permanent_uuid: "7932a656946f4546a91738584eb612e2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46271 } attrs { promote: false } } }
I20240503 02:00:15.270987 16506 ts_tablet_manager.cc:1875] T 84425c87d5964b4bb5c5ecfb72360e07 P 17eb0a665dc648d9a5ad2c62930861c4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:15.271181 15026 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7244e26e54694bd48373e8634b4daf93 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:00:15.272910 15560 tablet_service.cc:1514] Processing DeleteTablet for tablet 7244e26e54694bd48373e8634b4daf93 with delete_type TABLET_DATA_TOMBSTONED (TS 17eb0a665dc648d9a5ad2c62930861c4 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:38394
I20240503 02:00:15.273105 16506 ts_tablet_manager.cc:1888] T 84425c87d5964b4bb5c5ecfb72360e07 P 17eb0a665dc648d9a5ad2c62930861c4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:00:15.273175 16506 log.cc:1196] T 84425c87d5964b4bb5c5ecfb72360e07 P 17eb0a665dc648d9a5ad2c62930861c4: Deleting WAL directory at /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/84425c87d5964b4bb5c5ecfb72360e07
I20240503 02:00:15.273582 16506 tablet_replica.cc:330] T 7244e26e54694bd48373e8634b4daf93 P 17eb0a665dc648d9a5ad2c62930861c4: stopping tablet replica
I20240503 02:00:15.273645 15029 catalog_manager.cc:4830] TS 17eb0a665dc648d9a5ad2c62930861c4 (127.0.99.132:33559): tablet 84425c87d5964b4bb5c5ecfb72360e07 (table rebalance_test_table_0 [id=b9f4cdc190fd486bb9d37c53db387db4]) successfully deleted
I20240503 02:00:15.273687 16506 raft_consensus.cc:2229] T 7244e26e54694bd48373e8634b4daf93 P 17eb0a665dc648d9a5ad2c62930861c4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:15.273766 16506 pending_rounds.cc:62] T 7244e26e54694bd48373e8634b4daf93 P 17eb0a665dc648d9a5ad2c62930861c4: Trying to abort 1 pending ops.
I20240503 02:00:15.273819 16506 pending_rounds.cc:68] T 7244e26e54694bd48373e8634b4daf93 P 17eb0a665dc648d9a5ad2c62930861c4: Aborting op as it isn't in flight: id { term: 1 index: 5 } timestamp: 7023417816078594048 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "7244e26e54694bd48373e8634b4daf93" old_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } } peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } attrs { replace: true } } peers { permanent_uuid: "7932a656946f4546a91738584eb612e2" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46271 } attrs { promote: true } } } new_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } } peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } attrs { replace: true } } peers { permanent_uuid: "7932a656946f4546a91738584eb612e2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46271 } attrs { promote: false } } } }
I20240503 02:00:15.273910 16506 raft_consensus.cc:2894] T 7244e26e54694bd48373e8634b4daf93 P 17eb0a665dc648d9a5ad2c62930861c4 [term 1 FOLLOWER]: Aborting config change with OpId 1.5: Aborted: Op aborted
I20240503 02:00:15.273957 16506 raft_consensus.cc:2866] T 7244e26e54694bd48373e8634b4daf93 P 17eb0a665dc648d9a5ad2c62930861c4 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240503 02:00:15.273991 16506 raft_consensus.cc:2258] T 7244e26e54694bd48373e8634b4daf93 P 17eb0a665dc648d9a5ad2c62930861c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:15.274487 16506 ts_tablet_manager.cc:1875] T 7244e26e54694bd48373e8634b4daf93 P 17eb0a665dc648d9a5ad2c62930861c4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:15.275755 16506 ts_tablet_manager.cc:1888] T 7244e26e54694bd48373e8634b4daf93 P 17eb0a665dc648d9a5ad2c62930861c4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:00:15.275816 16506 log.cc:1196] T 7244e26e54694bd48373e8634b4daf93 P 17eb0a665dc648d9a5ad2c62930861c4: Deleting WAL directory at /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/7244e26e54694bd48373e8634b4daf93
I20240503 02:00:15.276279 15029 catalog_manager.cc:4830] TS 17eb0a665dc648d9a5ad2c62930861c4 (127.0.99.132:33559): tablet 7244e26e54694bd48373e8634b4daf93 (table rebalance_test_table_1 [id=174f433588294c61856c430b17973e89]) successfully deleted
I20240503 02:00:15.346338 15184 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4438db9442024011bd8d5632a03d34b0"
dest_uuid: "5751ea0f088543bcb141b400a6131a1f"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:48604
I20240503 02:00:15.346535 15184 raft_consensus.cc:595] T 4438db9442024011bd8d5632a03d34b0 P 5751ea0f088543bcb141b400a6131a1f [term 1 LEADER]: Received request to transfer leadership
I20240503 02:00:15.375042 15184 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4438db9442024011bd8d5632a03d34b0"
dest_uuid: "5751ea0f088543bcb141b400a6131a1f"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:48648
I20240503 02:00:15.375208 15184 raft_consensus.cc:595] T 4438db9442024011bd8d5632a03d34b0 P 5751ea0f088543bcb141b400a6131a1f [term 1 LEADER]: Received request to transfer leadership
I20240503 02:00:15.586892 15184 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4438db9442024011bd8d5632a03d34b0"
dest_uuid: "5751ea0f088543bcb141b400a6131a1f"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:48662
I20240503 02:00:15.587071 15184 raft_consensus.cc:595] T 4438db9442024011bd8d5632a03d34b0 P 5751ea0f088543bcb141b400a6131a1f [term 1 LEADER]: Received request to transfer leadership
I20240503 02:00:15.610213 16439 raft_consensus.cc:979] T 4438db9442024011bd8d5632a03d34b0 P 5751ea0f088543bcb141b400a6131a1f: : Instructing follower 7932a656946f4546a91738584eb612e2 to start an election
I20240503 02:00:15.610369 16439 raft_consensus.cc:1067] T 4438db9442024011bd8d5632a03d34b0 P 5751ea0f088543bcb141b400a6131a1f [term 1 LEADER]: Signalling peer 7932a656946f4546a91738584eb612e2 to start an election
I20240503 02:00:15.611019 15887 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4438db9442024011bd8d5632a03d34b0"
dest_uuid: "7932a656946f4546a91738584eb612e2"
from {username='slave'} at 127.0.99.129:56841
I20240503 02:00:15.611243 15887 raft_consensus.cc:482] T 4438db9442024011bd8d5632a03d34b0 P 7932a656946f4546a91738584eb612e2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:15.611347 15887 raft_consensus.cc:3037] T 4438db9442024011bd8d5632a03d34b0 P 7932a656946f4546a91738584eb612e2 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:15.612252 15887 raft_consensus.cc:504] T 4438db9442024011bd8d5632a03d34b0 P 7932a656946f4546a91738584eb612e2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } attrs { replace: true } } peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } } peers { permanent_uuid: "7932a656946f4546a91738584eb612e2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46271 } attrs { promote: false } }
I20240503 02:00:15.613318 15887 leader_election.cc:288] T 4438db9442024011bd8d5632a03d34b0 P 7932a656946f4546a91738584eb612e2 [CANDIDATE]: Term 2 election: Requested vote from peers 5751ea0f088543bcb141b400a6131a1f (127.0.99.129:46041), 17eb0a665dc648d9a5ad2c62930861c4 (127.0.99.132:33559)
I20240503 02:00:15.613657 15184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4438db9442024011bd8d5632a03d34b0" candidate_uuid: "7932a656946f4546a91738584eb612e2" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "5751ea0f088543bcb141b400a6131a1f"
I20240503 02:00:15.613807 15184 raft_consensus.cc:3032] T 4438db9442024011bd8d5632a03d34b0 P 5751ea0f088543bcb141b400a6131a1f [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:00:15.613852 15184 raft_consensus.cc:729] T 4438db9442024011bd8d5632a03d34b0 P 5751ea0f088543bcb141b400a6131a1f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5751ea0f088543bcb141b400a6131a1f, State: Running, Role: LEADER
I20240503 02:00:15.614032 15184 consensus_queue.cc:260] T 4438db9442024011bd8d5632a03d34b0 P 5751ea0f088543bcb141b400a6131a1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } attrs { replace: true } } peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } } peers { permanent_uuid: "7932a656946f4546a91738584eb612e2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46271 } attrs { promote: false } }
I20240503 02:00:15.614245 15184 raft_consensus.cc:3037] T 4438db9442024011bd8d5632a03d34b0 P 5751ea0f088543bcb141b400a6131a1f [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:15.615329 15184 raft_consensus.cc:2454] T 4438db9442024011bd8d5632a03d34b0 P 5751ea0f088543bcb141b400a6131a1f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7932a656946f4546a91738584eb612e2 in term 2.
I20240503 02:00:15.615716 15820 leader_election.cc:302] T 4438db9442024011bd8d5632a03d34b0 P 7932a656946f4546a91738584eb612e2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5751ea0f088543bcb141b400a6131a1f, 7932a656946f4546a91738584eb612e2; no voters:
I20240503 02:00:15.615995 16350 raft_consensus.cc:2781] T 4438db9442024011bd8d5632a03d34b0 P 7932a656946f4546a91738584eb612e2 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:00:15.616253 16350 raft_consensus.cc:686] T 4438db9442024011bd8d5632a03d34b0 P 7932a656946f4546a91738584eb612e2 [term 2 LEADER]: Becoming Leader. State: Replica: 7932a656946f4546a91738584eb612e2, State: Running, Role: LEADER
I20240503 02:00:15.616611 16350 consensus_queue.cc:237] T 4438db9442024011bd8d5632a03d34b0 P 7932a656946f4546a91738584eb612e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } attrs { replace: true } } peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } } peers { permanent_uuid: "7932a656946f4546a91738584eb612e2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46271 } attrs { promote: false } }
I20240503 02:00:15.617830 15578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4438db9442024011bd8d5632a03d34b0" candidate_uuid: "7932a656946f4546a91738584eb612e2" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "17eb0a665dc648d9a5ad2c62930861c4"
I20240503 02:00:15.617959 15578 raft_consensus.cc:3037] T 4438db9442024011bd8d5632a03d34b0 P 17eb0a665dc648d9a5ad2c62930861c4 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:15.618995 15578 raft_consensus.cc:2454] T 4438db9442024011bd8d5632a03d34b0 P 17eb0a665dc648d9a5ad2c62930861c4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7932a656946f4546a91738584eb612e2 in term 2.
I20240503 02:00:15.619426 15039 catalog_manager.cc:5484] T 4438db9442024011bd8d5632a03d34b0 P 7932a656946f4546a91738584eb612e2 reported cstate change: term changed from 1 to 2, leader changed from 5751ea0f088543bcb141b400a6131a1f (127.0.99.129) to 7932a656946f4546a91738584eb612e2 (127.0.99.130). New cstate: current_term: 2 leader_uuid: "7932a656946f4546a91738584eb612e2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7932a656946f4546a91738584eb612e2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46271 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:00:16.126250 15184 raft_consensus.cc:1261] T 4438db9442024011bd8d5632a03d34b0 P 5751ea0f088543bcb141b400a6131a1f [term 2 FOLLOWER]: Refusing update from remote peer 7932a656946f4546a91738584eb612e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 02:00:16.126844 16640 consensus_queue.cc:1035] T 4438db9442024011bd8d5632a03d34b0 P 7932a656946f4546a91738584eb612e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5751ea0f088543bcb141b400a6131a1f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46041 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:16.130354 15578 raft_consensus.cc:1261] T 4438db9442024011bd8d5632a03d34b0 P 17eb0a665dc648d9a5ad2c62930861c4 [term 2 FOLLOWER]: Refusing update from remote peer 7932a656946f4546a91738584eb612e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 02:00:16.130816 16642 consensus_queue.cc:1035] T 4438db9442024011bd8d5632a03d34b0 P 7932a656946f4546a91738584eb612e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:16.703740 15887 consensus_queue.cc:237] T 4438db9442024011bd8d5632a03d34b0 P 7932a656946f4546a91738584eb612e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } } peers { permanent_uuid: "7932a656946f4546a91738584eb612e2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46271 } attrs { promote: false } }
I20240503 02:00:16.704782 15578 raft_consensus.cc:1261] T 4438db9442024011bd8d5632a03d34b0 P 17eb0a665dc648d9a5ad2c62930861c4 [term 2 FOLLOWER]: Refusing update from remote peer 7932a656946f4546a91738584eb612e2: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 02:00:16.705170 16666 consensus_queue.cc:1035] T 4438db9442024011bd8d5632a03d34b0 P 7932a656946f4546a91738584eb612e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:16.706626 16665 raft_consensus.cc:2932] T 4438db9442024011bd8d5632a03d34b0 P 7932a656946f4546a91738584eb612e2 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 5751ea0f088543bcb141b400a6131a1f (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } } peers { permanent_uuid: "7932a656946f4546a91738584eb612e2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46271 } attrs { promote: false } } }
I20240503 02:00:16.707158 15578 raft_consensus.cc:2932] T 4438db9442024011bd8d5632a03d34b0 P 17eb0a665dc648d9a5ad2c62930861c4 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 5751ea0f088543bcb141b400a6131a1f (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } } peers { permanent_uuid: "7932a656946f4546a91738584eb612e2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46271 } attrs { promote: false } } }
I20240503 02:00:16.707955 15026 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4438db9442024011bd8d5632a03d34b0 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:00:16.708936 15038 catalog_manager.cc:5484] T 4438db9442024011bd8d5632a03d34b0 P 7932a656946f4546a91738584eb612e2 reported cstate change: config changed from index 5 to 7, VOTER 5751ea0f088543bcb141b400a6131a1f (127.0.99.129) evicted. New cstate: current_term: 2 leader_uuid: "7932a656946f4546a91738584eb612e2" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "17eb0a665dc648d9a5ad2c62930861c4" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7932a656946f4546a91738584eb612e2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46271 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:00:16.711707 15164 tablet_service.cc:1514] Processing DeleteTablet for tablet 4438db9442024011bd8d5632a03d34b0 with delete_type TABLET_DATA_TOMBSTONED (TS 5751ea0f088543bcb141b400a6131a1f not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:48148
I20240503 02:00:16.712050 16672 tablet_replica.cc:330] T 4438db9442024011bd8d5632a03d34b0 P 5751ea0f088543bcb141b400a6131a1f: stopping tablet replica
I20240503 02:00:16.712193 16672 raft_consensus.cc:2229] T 4438db9442024011bd8d5632a03d34b0 P 5751ea0f088543bcb141b400a6131a1f [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:16.712314 16672 raft_consensus.cc:2258] T 4438db9442024011bd8d5632a03d34b0 P 5751ea0f088543bcb141b400a6131a1f [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:16.713001 16672 ts_tablet_manager.cc:1875] T 4438db9442024011bd8d5632a03d34b0 P 5751ea0f088543bcb141b400a6131a1f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:16.714835 16672 ts_tablet_manager.cc:1888] T 4438db9442024011bd8d5632a03d34b0 P 5751ea0f088543bcb141b400a6131a1f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 02:00:16.714901 16672 log.cc:1196] T 4438db9442024011bd8d5632a03d34b0 P 5751ea0f088543bcb141b400a6131a1f: Deleting WAL directory at /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/4438db9442024011bd8d5632a03d34b0
I20240503 02:00:16.715356 15026 catalog_manager.cc:4830] TS 5751ea0f088543bcb141b400a6131a1f (127.0.99.129:46041): tablet 4438db9442024011bd8d5632a03d34b0 (table rebalance_test_table_0 [id=b9f4cdc190fd486bb9d37c53db387db4]) successfully deleted
I20240503 02:00:16.878557 15867 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:16.878974 15560 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:00:16.878989 15998 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:16.879053 15164 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:00:16.961560 15867 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:16.961947 15998 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:16.962507 15164 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:00:16.963080 15560 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:00:17.038576 15164 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:00:17.039234 15560 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:00:17.039443 15998 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:17.039888 15867 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:17.083492 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task42rE7p/build/debug/bin/kudu with pid 15102
I20240503 02:00:17.090413 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task42rE7p/build/debug/bin/kudu with pid 15805
I20240503 02:00:17.096216 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task42rE7p/build/debug/bin/kudu with pid 15936
I20240503 02:00:17.102159 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task42rE7p/build/debug/bin/kudu with pid 15498
I20240503 02:00:17.108196 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task42rE7p/build/debug/bin/kudu with pid 15011
2024-05-03T02:00:17Z chronyd exiting
[ OK ] LocationAwareRebalancingParamTest.Rebalance/idx7_rf2_t3_l_1_2_1 (11088 ms)
[ RUN ] LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer/idx6_rf7_t7_l_3_4_3
I20240503 02:00:17.133973 398 test_util.cc:268] Using random seed: 348715407
I20240503 02:00:17.134816 398 ts_itest-base.cc:112] Starting cluster with:
I20240503 02:00:17.134858 398 ts_itest-base.cc:113] --------------
I20240503 02:00:17.134886 398 ts_itest-base.cc:114] 10 tablet servers
I20240503 02:00:17.134903 398 ts_itest-base.cc:115] 7 replicas per TS
I20240503 02:00:17.134928 398 ts_itest-base.cc:116] --------------
2024-05-03T02:00:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T02:00:17Z Disabled control of system clock
I20240503 02:00:17.143764 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task42rE7p/build/debug/bin/kudu
/tmp/dist-test-task42rE7p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:41395
--webserver_interface=127.0.99.190
--webserver_port=0
--builtin_ntp_servers=127.0.99.148:44551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.99.190:41395
--location_mapping_cmd=/tmp/dist-test-task42rE7p/build/debug/bin/testdata/assign-location.py --state_store=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/location-assignment.state --map /A:3 --map /B:4 --map /C:3
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false
--master_client_location_assignment_enabled=false
--allow_unsafe_replication_factor with env {}
W20240503 02:00:17.245872 16787 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240503 02:00:17.246093 16787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:00:17.246122 16787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:00:17.246189 16787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:00:17.249193 16787 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:00:17.249256 16787 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 02:00:17.249287 16787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:00:17.249325 16787 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 02:00:17.249365 16787 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 02:00:17.252779 16787 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:44551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--location_mapping_cmd=/tmp/dist-test-task42rE7p/build/debug/bin/testdata/assign-location.py --state_store=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/location-assignment.state --map /A:3 --map /B:4 --map /C:3
--ipki_ca_key_size=768
--master_addresses=127.0.99.190:41395
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:41395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.99.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.16787
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 02:00:17.253582 16787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:00:17.254765 16787 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:00:17.262879 16793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:17.263661 16795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:17.264933 16792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:00:17.265990 16787 server_base.cc:1034] running on GCE node
I20240503 02:00:17.266467 16787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:00:17.267346 16787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:00:17.268497 16787 hybrid_clock.cc:648] HybridClock initialized: now 1714701617268471 us; error 38 us; skew 500 ppm
I20240503 02:00:17.270377 16787 webserver.cc:460] Webserver started at http://127.0.99.190:44623/ using document root <none> and password file <none>
I20240503 02:00:17.270910 16787 fs_manager.cc:359] Metadata directory not provided
I20240503 02:00:17.270970 16787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:00:17.271134 16787 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:00:17.272778 16787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "f044076ee81e4df781b895b122f8853c"
format_stamp: "Formatted at 2024-05-03 02:00:17 on dist-test-slave-dtz3"
I20240503 02:00:17.273244 16787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "f044076ee81e4df781b895b122f8853c"
format_stamp: "Formatted at 2024-05-03 02:00:17 on dist-test-slave-dtz3"
I20240503 02:00:17.276507 16787 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240503 02:00:17.278676 16801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:17.279580 16787 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:17.279716 16787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/wal
uuid: "f044076ee81e4df781b895b122f8853c"
format_stamp: "Formatted at 2024-05-03 02:00:17 on dist-test-slave-dtz3"
I20240503 02:00:17.279796 16787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:00:17.293576 16787 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:00:17.294165 16787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:00:17.294320 16787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:00:17.300732 16787 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.190:41395
I20240503 02:00:17.300751 16853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.190:41395 every 8 connection(s)
I20240503 02:00:17.301653 16787 server_base.cc:1166] Dumped server information to /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 02:00:17.303998 16854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:17.308547 16854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f044076ee81e4df781b895b122f8853c: Bootstrap starting.
I20240503 02:00:17.308858 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task42rE7p/build/debug/bin/kudu as pid 16787
I20240503 02:00:17.308969 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 02:00:17.310734 16854 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f044076ee81e4df781b895b122f8853c: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:17.311353 16854 log.cc:824] T 00000000000000000000000000000000 P f044076ee81e4df781b895b122f8853c: Log is configured to *not* fsync() on all Append() calls
I20240503 02:00:17.313118 16854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f044076ee81e4df781b895b122f8853c: No bootstrap required, opened a new log
I20240503 02:00:17.316004 16854 raft_consensus.cc:348] T 00000000000000000000000000000000 P f044076ee81e4df781b895b122f8853c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f044076ee81e4df781b895b122f8853c" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 41395 } }
I20240503 02:00:17.316205 16854 raft_consensus.cc:374] T 00000000000000000000000000000000 P f044076ee81e4df781b895b122f8853c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:17.316270 16854 raft_consensus.cc:729] T 00000000000000000000000000000000 P f044076ee81e4df781b895b122f8853c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f044076ee81e4df781b895b122f8853c, State: Initialized, Role: FOLLOWER
I20240503 02:00:17.316745 16854 consensus_queue.cc:260] T 00000000000000000000000000000000 P f044076ee81e4df781b895b122f8853c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f044076ee81e4df781b895b122f8853c" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 41395 } }
I20240503 02:00:17.316874 16854 raft_consensus.cc:388] T 00000000000000000000000000000000 P f044076ee81e4df781b895b122f8853c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:00:17.316936 16854 raft_consensus.cc:482] T 00000000000000000000000000000000 P f044076ee81e4df781b895b122f8853c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:00:17.317040 16854 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f044076ee81e4df781b895b122f8853c [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:17.318379 16854 raft_consensus.cc:504] T 00000000000000000000000000000000 P f044076ee81e4df781b895b122f8853c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f044076ee81e4df781b895b122f8853c" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 41395 } }
I20240503 02:00:17.318662 16854 leader_election.cc:302] T 00000000000000000000000000000000 P f044076ee81e4df781b895b122f8853c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f044076ee81e4df781b895b122f8853c; no voters:
I20240503 02:00:17.318876 16854 leader_election.cc:288] T 00000000000000000000000000000000 P f044076ee81e4df781b895b122f8853c [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:00:17.318993 16859 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f044076ee81e4df781b895b122f8853c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:17.319218 16859 raft_consensus.cc:686] T 00000000000000000000000000000000 P f044076ee81e4df781b895b122f8853c [term 1 LEADER]: Becoming Leader. State: Replica: f044076ee81e4df781b895b122f8853c, State: Running, Role: LEADER
I20240503 02:00:17.319555 16859 consensus_queue.cc:237] T 00000000000000000000000000000000 P f044076ee81e4df781b895b122f8853c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f044076ee81e4df781b895b122f8853c" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 41395 } }
I20240503 02:00:17.319859 16854 sys_catalog.cc:564] T 00000000000000000000000000000000 P f044076ee81e4df781b895b122f8853c [sys.catalog]: configured and running, proceeding with master startup.
I20240503 02:00:17.320945 16860 sys_catalog.cc:455] T 00000000000000000000000000000000 P f044076ee81e4df781b895b122f8853c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f044076ee81e4df781b895b122f8853c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f044076ee81e4df781b895b122f8853c" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 41395 } } }
I20240503 02:00:17.321061 16860 sys_catalog.cc:458] T 00000000000000000000000000000000 P f044076ee81e4df781b895b122f8853c [sys.catalog]: This master's current role is: LEADER
I20240503 02:00:17.321368 16861 sys_catalog.cc:455] T 00000000000000000000000000000000 P f044076ee81e4df781b895b122f8853c [sys.catalog]: SysCatalogTable state changed. Reason: New leader f044076ee81e4df781b895b122f8853c. Latest consensus state: current_term: 1 leader_uuid: "f044076ee81e4df781b895b122f8853c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f044076ee81e4df781b895b122f8853c" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 41395 } } }
I20240503 02:00:17.321452 16868 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 02:00:17.321525 16861 sys_catalog.cc:458] T 00000000000000000000000000000000 P f044076ee81e4df781b895b122f8853c [sys.catalog]: This master's current role is: LEADER
I20240503 02:00:17.324231 16868 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 02:00:17.328919 16868 catalog_manager.cc:1347] Generated new cluster ID: 0bee722269fe496ea14c73b75081e66f
I20240503 02:00:17.328986 16868 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 02:00:17.343739 16868 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 02:00:17.344911 16868 catalog_manager.cc:1504] Loading token signing keys...
I20240503 02:00:17.354038 16868 catalog_manager.cc:5857] T 00000000000000000000000000000000 P f044076ee81e4df781b895b122f8853c: Generated new TSK 0
I20240503 02:00:17.354954 16868 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 02:00:17.359315 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task42rE7p/build/debug/bin/kudu
/tmp/dist-test-task42rE7p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.129:0
--local_ip_for_outbound_sockets=127.0.99.129
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:41395
--builtin_ntp_servers=127.0.99.148:44551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 02:00:17.465860 16878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:00:17.466063 16878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:00:17.466130 16878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:00:17.468971 16878 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:00:17.469031 16878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:00:17.469131 16878 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.129
I20240503 02:00:17.472518 16878 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:44551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:41395
--never_fsync=true
--heap_profile_path=/tmp/kudu.16878
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.129
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 02:00:17.473285 16878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:00:17.474301 16878 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:00:17.482597 16883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:17.483803 16884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:17.485095 16886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:00:17.486209 16878 server_base.cc:1034] running on GCE node
I20240503 02:00:17.486608 16878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:00:17.487213 16878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:00:17.488389 16878 hybrid_clock.cc:648] HybridClock initialized: now 1714701617488368 us; error 38 us; skew 500 ppm
I20240503 02:00:17.490481 16878 webserver.cc:460] Webserver started at http://127.0.99.129:44389/ using document root <none> and password file <none>
I20240503 02:00:17.491027 16878 fs_manager.cc:359] Metadata directory not provided
I20240503 02:00:17.491086 16878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:00:17.491255 16878 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:00:17.492934 16878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "6398c8993c3c45dcaab25b42142d5671"
format_stamp: "Formatted at 2024-05-03 02:00:17 on dist-test-slave-dtz3"
I20240503 02:00:17.493376 16878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "6398c8993c3c45dcaab25b42142d5671"
format_stamp: "Formatted at 2024-05-03 02:00:17 on dist-test-slave-dtz3"
I20240503 02:00:17.496542 16878 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240503 02:00:17.498652 16892 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:17.499630 16878 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:17.499759 16878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "6398c8993c3c45dcaab25b42142d5671"
format_stamp: "Formatted at 2024-05-03 02:00:17 on dist-test-slave-dtz3"
I20240503 02:00:17.499846 16878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:00:17.524931 16878 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:00:17.525559 16878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:00:17.525727 16878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:00:17.526249 16878 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:00:17.527128 16878 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:00:17.527176 16878 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:17.527235 16878 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:00:17.527262 16878 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:17.535305 16878 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.129:36983
I20240503 02:00:17.535352 17005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.129:36983 every 8 connection(s)
I20240503 02:00:17.536242 16878 server_base.cc:1166] Dumped server information to /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 02:00:17.544586 17006 heartbeater.cc:347] Connected to a master server at 127.0.99.190:41395
I20240503 02:00:17.544837 17006 heartbeater.cc:464] Registering TS with master...
I20240503 02:00:17.545341 17006 heartbeater.cc:511] Master 127.0.99.190:41395 requested a full tablet report, sending...
I20240503 02:00:17.545666 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task42rE7p/build/debug/bin/kudu as pid 16878
I20240503 02:00:17.545799 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 02:00:17.550148 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task42rE7p/build/debug/bin/kudu
/tmp/dist-test-task42rE7p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.130:0
--local_ip_for_outbound_sockets=127.0.99.130
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:41395
--builtin_ntp_servers=127.0.99.148:44551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:00:17.584985 16818 ts_manager.cc:194] Registered new tserver with Master: 6398c8993c3c45dcaab25b42142d5671 (127.0.99.129:36983)
I20240503 02:00:17.586378 16818 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.129:48595
W20240503 02:00:17.661872 17010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:00:17.662086 17010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:00:17.662153 17010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:00:17.665122 17010 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:00:17.665185 17010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:00:17.665285 17010 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.130
I20240503 02:00:17.668808 17010 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:44551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:41395
--never_fsync=true
--heap_profile_path=/tmp/kudu.17010
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.130
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 02:00:17.669673 17010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:00:17.670753 17010 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:00:17.679867 17016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:17.681079 17018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:17.682080 17015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:00:17.683164 17010 server_base.cc:1034] running on GCE node
I20240503 02:00:17.683539 17010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:00:17.684082 17010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:00:17.685243 17010 hybrid_clock.cc:648] HybridClock initialized: now 1714701617685228 us; error 27 us; skew 500 ppm
I20240503 02:00:17.687157 17010 webserver.cc:460] Webserver started at http://127.0.99.130:37643/ using document root <none> and password file <none>
I20240503 02:00:17.687686 17010 fs_manager.cc:359] Metadata directory not provided
I20240503 02:00:17.687744 17010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:00:17.687912 17010 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:00:17.689580 17010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "89dd09f8aaee4b249ae3ce924a422a67"
format_stamp: "Formatted at 2024-05-03 02:00:17 on dist-test-slave-dtz3"
I20240503 02:00:17.690055 17010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "89dd09f8aaee4b249ae3ce924a422a67"
format_stamp: "Formatted at 2024-05-03 02:00:17 on dist-test-slave-dtz3"
I20240503 02:00:17.693344 17010 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240503 02:00:17.695449 17024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:17.696502 17010 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:17.696606 17010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "89dd09f8aaee4b249ae3ce924a422a67"
format_stamp: "Formatted at 2024-05-03 02:00:17 on dist-test-slave-dtz3"
I20240503 02:00:17.696731 17010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:00:17.723490 17010 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:00:17.724150 17010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:00:17.724319 17010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:00:17.724887 17010 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:00:17.725802 17010 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:00:17.725860 17010 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:17.725924 17010 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:00:17.725950 17010 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:17.734135 17010 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.130:38889
I20240503 02:00:17.734175 17137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.130:38889 every 8 connection(s)
I20240503 02:00:17.735042 17010 server_base.cc:1166] Dumped server information to /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 02:00:17.736510 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task42rE7p/build/debug/bin/kudu as pid 17010
I20240503 02:00:17.736662 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 02:00:17.738934 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task42rE7p/build/debug/bin/kudu
/tmp/dist-test-task42rE7p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.131:0
--local_ip_for_outbound_sockets=127.0.99.131
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:41395
--builtin_ntp_servers=127.0.99.148:44551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:00:17.745016 17138 heartbeater.cc:347] Connected to a master server at 127.0.99.190:41395
I20240503 02:00:17.745222 17138 heartbeater.cc:464] Registering TS with master...
I20240503 02:00:17.745707 17138 heartbeater.cc:511] Master 127.0.99.190:41395 requested a full tablet report, sending...
I20240503 02:00:17.791100 16818 ts_manager.cc:194] Registered new tserver with Master: 89dd09f8aaee4b249ae3ce924a422a67 (127.0.99.130:38889)
I20240503 02:00:17.792107 16818 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.130:43889
W20240503 02:00:17.859712 17141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:00:17.859912 17141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:00:17.859962 17141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:00:17.863296 17141 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:00:17.863360 17141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:00:17.863427 17141 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.131
I20240503 02:00:17.866966 17141 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:44551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:41395
--never_fsync=true
--heap_profile_path=/tmp/kudu.17141
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.131
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 02:00:17.867771 17141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:00:17.868865 17141 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:00:17.877666 17148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:17.878844 17147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:17.879773 17150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:00:17.880895 17141 server_base.cc:1034] running on GCE node
I20240503 02:00:17.881297 17141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:00:17.881848 17141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:00:17.883075 17141 hybrid_clock.cc:648] HybridClock initialized: now 1714701617883052 us; error 39 us; skew 500 ppm
I20240503 02:00:17.885068 17141 webserver.cc:460] Webserver started at http://127.0.99.131:42921/ using document root <none> and password file <none>
I20240503 02:00:17.885596 17141 fs_manager.cc:359] Metadata directory not provided
I20240503 02:00:17.885658 17141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:00:17.885854 17141 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:00:17.887492 17141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "5b8291b26a7f448b8636187d1caf86d0"
format_stamp: "Formatted at 2024-05-03 02:00:17 on dist-test-slave-dtz3"
I20240503 02:00:17.887938 17141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "5b8291b26a7f448b8636187d1caf86d0"
format_stamp: "Formatted at 2024-05-03 02:00:17 on dist-test-slave-dtz3"
I20240503 02:00:17.891146 17141 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 02:00:17.893193 17156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:17.894151 17141 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:17.894261 17141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "5b8291b26a7f448b8636187d1caf86d0"
format_stamp: "Formatted at 2024-05-03 02:00:17 on dist-test-slave-dtz3"
I20240503 02:00:17.894341 17141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:00:17.918160 17141 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:00:17.918730 17141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:00:17.918892 17141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:00:17.919405 17141 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:00:17.920351 17141 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:00:17.920408 17141 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:17.920465 17141 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:00:17.920492 17141 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:17.928786 17141 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.131:44479
I20240503 02:00:17.928824 17269 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.131:44479 every 8 connection(s)
I20240503 02:00:17.929651 17141 server_base.cc:1166] Dumped server information to /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 02:00:17.935787 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task42rE7p/build/debug/bin/kudu as pid 17141
I20240503 02:00:17.935925 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 02:00:17.937985 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task42rE7p/build/debug/bin/kudu
/tmp/dist-test-task42rE7p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.132:0
--local_ip_for_outbound_sockets=127.0.99.132
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:41395
--builtin_ntp_servers=127.0.99.148:44551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:00:17.938068 17270 heartbeater.cc:347] Connected to a master server at 127.0.99.190:41395
I20240503 02:00:17.938263 17270 heartbeater.cc:464] Registering TS with master...
I20240503 02:00:17.938839 17270 heartbeater.cc:511] Master 127.0.99.190:41395 requested a full tablet report, sending...
I20240503 02:00:17.971702 16818 ts_manager.cc:194] Registered new tserver with Master: 5b8291b26a7f448b8636187d1caf86d0 (127.0.99.131:44479)
I20240503 02:00:17.972455 16818 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.131:54957
W20240503 02:00:18.044835 17273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:00:18.045042 17273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:00:18.045091 17273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:00:18.048027 17273 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:00:18.048084 17273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:00:18.048167 17273 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.132
I20240503 02:00:18.051685 17273 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:44551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:41395
--never_fsync=true
--heap_profile_path=/tmp/kudu.17273
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.132
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 02:00:18.052501 17273 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:00:18.053593 17273 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:00:18.062201 17279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:18.063073 17280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:18.064257 17282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:00:18.065443 17273 server_base.cc:1034] running on GCE node
I20240503 02:00:18.065797 17273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:00:18.066303 17273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:00:18.067473 17273 hybrid_clock.cc:648] HybridClock initialized: now 1714701618067461 us; error 25 us; skew 500 ppm
I20240503 02:00:18.069416 17273 webserver.cc:460] Webserver started at http://127.0.99.132:39007/ using document root <none> and password file <none>
I20240503 02:00:18.069923 17273 fs_manager.cc:359] Metadata directory not provided
I20240503 02:00:18.069983 17273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:00:18.070137 17273 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:00:18.071672 17273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "d43ed6b8663947d99a72cd922cc323aa"
format_stamp: "Formatted at 2024-05-03 02:00:18 on dist-test-slave-dtz3"
I20240503 02:00:18.072093 17273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "d43ed6b8663947d99a72cd922cc323aa"
format_stamp: "Formatted at 2024-05-03 02:00:18 on dist-test-slave-dtz3"
I20240503 02:00:18.075241 17273 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 02:00:18.077304 17288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:18.078274 17273 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:18.078373 17273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "d43ed6b8663947d99a72cd922cc323aa"
format_stamp: "Formatted at 2024-05-03 02:00:18 on dist-test-slave-dtz3"
I20240503 02:00:18.078470 17273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:00:18.093945 17273 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:00:18.094524 17273 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:00:18.094681 17273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:00:18.095202 17273 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:00:18.096069 17273 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:00:18.096117 17273 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:18.096155 17273 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:00:18.096186 17273 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:18.104458 17273 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.132:40023
I20240503 02:00:18.104485 17401 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.132:40023 every 8 connection(s)
I20240503 02:00:18.105394 17273 server_base.cc:1166] Dumped server information to /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 02:00:18.113337 17402 heartbeater.cc:347] Connected to a master server at 127.0.99.190:41395
I20240503 02:00:18.113551 17402 heartbeater.cc:464] Registering TS with master...
I20240503 02:00:18.114025 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task42rE7p/build/debug/bin/kudu as pid 17273
I20240503 02:00:18.114151 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 02:00:18.114149 17402 heartbeater.cc:511] Master 127.0.99.190:41395 requested a full tablet report, sending...
I20240503 02:00:18.118353 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task42rE7p/build/debug/bin/kudu
/tmp/dist-test-task42rE7p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:0
--local_ip_for_outbound_sockets=127.0.99.133
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:41395
--builtin_ntp_servers=127.0.99.148:44551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:00:18.147857 16818 ts_manager.cc:194] Registered new tserver with Master: d43ed6b8663947d99a72cd922cc323aa (127.0.99.132:40023)
I20240503 02:00:18.148571 16818 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.132:50897
W20240503 02:00:18.223734 17406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:00:18.223963 17406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:00:18.224048 17406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:00:18.227025 17406 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:00:18.227097 17406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:00:18.227202 17406 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240503 02:00:18.231377 17406 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:44551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:41395
--never_fsync=true
--heap_profile_path=/tmp/kudu.17406
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 02:00:18.232287 17406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:00:18.233453 17406 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:00:18.242306 17414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:18.243507 17412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:18.244612 17411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:00:18.245594 17406 server_base.cc:1034] running on GCE node
I20240503 02:00:18.245959 17406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:00:18.246471 17406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:00:18.247627 17406 hybrid_clock.cc:648] HybridClock initialized: now 1714701618247610 us; error 27 us; skew 500 ppm
I20240503 02:00:18.249516 17406 webserver.cc:460] Webserver started at http://127.0.99.133:33299/ using document root <none> and password file <none>
I20240503 02:00:18.250044 17406 fs_manager.cc:359] Metadata directory not provided
I20240503 02:00:18.250103 17406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:00:18.250265 17406 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:00:18.251804 17406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "16d437fe629643948c95d628926d5ffa"
format_stamp: "Formatted at 2024-05-03 02:00:18 on dist-test-slave-dtz3"
I20240503 02:00:18.252228 17406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "16d437fe629643948c95d628926d5ffa"
format_stamp: "Formatted at 2024-05-03 02:00:18 on dist-test-slave-dtz3"
I20240503 02:00:18.255308 17406 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 02:00:18.257345 17420 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:18.258244 17406 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:18.258342 17406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "16d437fe629643948c95d628926d5ffa"
format_stamp: "Formatted at 2024-05-03 02:00:18 on dist-test-slave-dtz3"
I20240503 02:00:18.258419 17406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:00:18.271874 17406 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:00:18.272358 17406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:00:18.272495 17406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:00:18.273000 17406 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:00:18.273826 17406 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:00:18.273872 17406 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:18.273927 17406 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:00:18.273953 17406 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:18.282052 17406 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:46581
I20240503 02:00:18.282109 17533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:46581 every 8 connection(s)
I20240503 02:00:18.282968 17406 server_base.cc:1166] Dumped server information to /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 02:00:18.284183 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task42rE7p/build/debug/bin/kudu as pid 17406
I20240503 02:00:18.284307 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 02:00:18.286525 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task42rE7p/build/debug/bin/kudu
/tmp/dist-test-task42rE7p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.134:0
--local_ip_for_outbound_sockets=127.0.99.134
--webserver_interface=127.0.99.134
--webserver_port=0
--tserver_master_addrs=127.0.99.190:41395
--builtin_ntp_servers=127.0.99.148:44551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:00:18.292384 17534 heartbeater.cc:347] Connected to a master server at 127.0.99.190:41395
I20240503 02:00:18.292573 17534 heartbeater.cc:464] Registering TS with master...
I20240503 02:00:18.293066 17534 heartbeater.cc:511] Master 127.0.99.190:41395 requested a full tablet report, sending...
I20240503 02:00:18.332075 16818 ts_manager.cc:194] Registered new tserver with Master: 16d437fe629643948c95d628926d5ffa (127.0.99.133:46581)
I20240503 02:00:18.332993 16818 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:50505
W20240503 02:00:18.393576 17537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:00:18.393766 17537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:00:18.393808 17537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:00:18.396633 17537 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:00:18.396714 17537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:00:18.396790 17537 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.134
I20240503 02:00:18.400231 17537 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:44551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.134:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.0.99.134
--webserver_port=0
--tserver_master_addrs=127.0.99.190:41395
--never_fsync=true
--heap_profile_path=/tmp/kudu.17537
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.134
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 02:00:18.401034 17537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:00:18.402165 17537 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:00:18.412194 17544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:18.413290 17546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:18.414343 17543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:00:18.415350 17537 server_base.cc:1034] running on GCE node
I20240503 02:00:18.415758 17537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:00:18.416312 17537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:00:18.417481 17537 hybrid_clock.cc:648] HybridClock initialized: now 1714701618417456 us; error 37 us; skew 500 ppm
I20240503 02:00:18.419556 17537 webserver.cc:460] Webserver started at http://127.0.99.134:35195/ using document root <none> and password file <none>
I20240503 02:00:18.420109 17537 fs_manager.cc:359] Metadata directory not provided
I20240503 02:00:18.420178 17537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:00:18.420357 17537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:00:18.422083 17537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "c843193b801b4f1a88f630ed3ed0f375"
format_stamp: "Formatted at 2024-05-03 02:00:18 on dist-test-slave-dtz3"
I20240503 02:00:18.422516 17537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "c843193b801b4f1a88f630ed3ed0f375"
format_stamp: "Formatted at 2024-05-03 02:00:18 on dist-test-slave-dtz3"
I20240503 02:00:18.425841 17537 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 02:00:18.428014 17552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:18.429203 17537 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 02:00:18.429303 17537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "c843193b801b4f1a88f630ed3ed0f375"
format_stamp: "Formatted at 2024-05-03 02:00:18 on dist-test-slave-dtz3"
I20240503 02:00:18.429378 17537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:00:18.459617 17537 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:00:18.460184 17537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:00:18.460322 17537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:00:18.460844 17537 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:00:18.461977 17537 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:00:18.462038 17537 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:18.462096 17537 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:00:18.462128 17537 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:18.471714 17537 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.134:44343
I20240503 02:00:18.471825 17665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.134:44343 every 8 connection(s)
I20240503 02:00:18.472714 17537 server_base.cc:1166] Dumped server information to /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 02:00:18.475643 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task42rE7p/build/debug/bin/kudu as pid 17537
I20240503 02:00:18.475790 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240503 02:00:18.478827 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task42rE7p/build/debug/bin/kudu
/tmp/dist-test-task42rE7p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.135:0
--local_ip_for_outbound_sockets=127.0.99.135
--webserver_interface=127.0.99.135
--webserver_port=0
--tserver_master_addrs=127.0.99.190:41395
--builtin_ntp_servers=127.0.99.148:44551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:00:18.483878 17666 heartbeater.cc:347] Connected to a master server at 127.0.99.190:41395
I20240503 02:00:18.484213 17666 heartbeater.cc:464] Registering TS with master...
I20240503 02:00:18.485957 17666 heartbeater.cc:511] Master 127.0.99.190:41395 requested a full tablet report, sending...
I20240503 02:00:18.522125 16818 ts_manager.cc:194] Registered new tserver with Master: c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134:44343)
I20240503 02:00:18.523286 16818 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.134:51809
I20240503 02:00:18.588918 17006 heartbeater.cc:503] Master 127.0.99.190:41395 was elected leader, sending a full tablet report...
W20240503 02:00:18.607929 17669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:00:18.608165 17669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:00:18.608211 17669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:00:18.611573 17669 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:00:18.611639 17669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:00:18.611721 17669 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.135
I20240503 02:00:18.615442 17669 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:44551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.135:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.0.99.135
--webserver_port=0
--tserver_master_addrs=127.0.99.190:41395
--never_fsync=true
--heap_profile_path=/tmp/kudu.17669
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.135
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 02:00:18.616499 17669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:00:18.617830 17669 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:00:18.628464 17675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:18.629603 17678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:18.630493 17676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:00:18.631444 17669 server_base.cc:1034] running on GCE node
I20240503 02:00:18.631925 17669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:00:18.632678 17669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:00:18.633850 17669 hybrid_clock.cc:648] HybridClock initialized: now 1714701618633826 us; error 46 us; skew 500 ppm
I20240503 02:00:18.636157 17669 webserver.cc:460] Webserver started at http://127.0.99.135:41047/ using document root <none> and password file <none>
I20240503 02:00:18.636837 17669 fs_manager.cc:359] Metadata directory not provided
I20240503 02:00:18.636907 17669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:00:18.637101 17669 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:00:18.638823 17669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "002cc3411acb4fdfa0b67c6f57f687a6"
format_stamp: "Formatted at 2024-05-03 02:00:18 on dist-test-slave-dtz3"
I20240503 02:00:18.639304 17669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "002cc3411acb4fdfa0b67c6f57f687a6"
format_stamp: "Formatted at 2024-05-03 02:00:18 on dist-test-slave-dtz3"
I20240503 02:00:18.642910 17669 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.005s
I20240503 02:00:18.645316 17684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:18.646517 17669 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:18.646618 17669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "002cc3411acb4fdfa0b67c6f57f687a6"
format_stamp: "Formatted at 2024-05-03 02:00:18 on dist-test-slave-dtz3"
I20240503 02:00:18.646715 17669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:00:18.664279 17669 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:00:18.664937 17669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:00:18.665119 17669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:00:18.665658 17669 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:00:18.666560 17669 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:00:18.666615 17669 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:18.666669 17669 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:00:18.666704 17669 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:18.675521 17669 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.135:38989
I20240503 02:00:18.675587 17797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.135:38989 every 8 connection(s)
I20240503 02:00:18.676501 17669 server_base.cc:1166] Dumped server information to /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 02:00:18.686211 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task42rE7p/build/debug/bin/kudu as pid 17669
I20240503 02:00:18.686368 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240503 02:00:18.686547 17798 heartbeater.cc:347] Connected to a master server at 127.0.99.190:41395
I20240503 02:00:18.686779 17798 heartbeater.cc:464] Registering TS with master...
I20240503 02:00:18.687426 17798 heartbeater.cc:511] Master 127.0.99.190:41395 requested a full tablet report, sending...
I20240503 02:00:18.688917 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task42rE7p/build/debug/bin/kudu
/tmp/dist-test-task42rE7p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.136:0
--local_ip_for_outbound_sockets=127.0.99.136
--webserver_interface=127.0.99.136
--webserver_port=0
--tserver_master_addrs=127.0.99.190:41395
--builtin_ntp_servers=127.0.99.148:44551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:00:18.754740 16818 ts_manager.cc:194] Registered new tserver with Master: 002cc3411acb4fdfa0b67c6f57f687a6 (127.0.99.135:38989)
I20240503 02:00:18.755630 16818 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.135:55587
I20240503 02:00:18.794385 17138 heartbeater.cc:503] Master 127.0.99.190:41395 was elected leader, sending a full tablet report...
W20240503 02:00:18.805989 17801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:00:18.806212 17801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:00:18.806281 17801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:00:18.809293 17801 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:00:18.809371 17801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:00:18.809465 17801 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.136
I20240503 02:00:18.813076 17801 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:44551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.136:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.0.99.136
--webserver_port=0
--tserver_master_addrs=127.0.99.190:41395
--never_fsync=true
--heap_profile_path=/tmp/kudu.17801
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.136
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 02:00:18.814030 17801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:00:18.815245 17801 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:00:18.825075 17807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:18.825932 17808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:18.826890 17810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:00:18.828238 17801 server_base.cc:1034] running on GCE node
I20240503 02:00:18.828698 17801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:00:18.829289 17801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:00:18.830463 17801 hybrid_clock.cc:648] HybridClock initialized: now 1714701618830441 us; error 51 us; skew 500 ppm
I20240503 02:00:18.832464 17801 webserver.cc:460] Webserver started at http://127.0.99.136:45657/ using document root <none> and password file <none>
I20240503 02:00:18.833057 17801 fs_manager.cc:359] Metadata directory not provided
I20240503 02:00:18.833145 17801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:00:18.833338 17801 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:00:18.835096 17801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "375d0f2ec11d4e25bd9d093bb6a5572d"
format_stamp: "Formatted at 2024-05-03 02:00:18 on dist-test-slave-dtz3"
I20240503 02:00:18.835597 17801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "375d0f2ec11d4e25bd9d093bb6a5572d"
format_stamp: "Formatted at 2024-05-03 02:00:18 on dist-test-slave-dtz3"
I20240503 02:00:18.839056 17801 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 02:00:18.841317 17816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:18.842514 17801 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:18.842615 17801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "375d0f2ec11d4e25bd9d093bb6a5572d"
format_stamp: "Formatted at 2024-05-03 02:00:18 on dist-test-slave-dtz3"
I20240503 02:00:18.842710 17801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:00:18.858204 17801 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:00:18.858811 17801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:00:18.858974 17801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:00:18.859515 17801 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:00:18.860424 17801 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:00:18.860477 17801 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:18.860534 17801 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:00:18.860569 17801 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:18.870214 17801 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.136:40167
I20240503 02:00:18.870321 17929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.136:40167 every 8 connection(s)
I20240503 02:00:18.871217 17801 server_base.cc:1166] Dumped server information to /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 02:00:18.875273 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task42rE7p/build/debug/bin/kudu as pid 17801
I20240503 02:00:18.875416 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240503 02:00:18.877954 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task42rE7p/build/debug/bin/kudu
/tmp/dist-test-task42rE7p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-8/wal
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-8/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-8/logs
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-8/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.137:0
--local_ip_for_outbound_sockets=127.0.99.137
--webserver_interface=127.0.99.137
--webserver_port=0
--tserver_master_addrs=127.0.99.190:41395
--builtin_ntp_servers=127.0.99.148:44551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:00:18.882122 17930 heartbeater.cc:347] Connected to a master server at 127.0.99.190:41395
I20240503 02:00:18.882406 17930 heartbeater.cc:464] Registering TS with master...
I20240503 02:00:18.882934 17930 heartbeater.cc:511] Master 127.0.99.190:41395 requested a full tablet report, sending...
I20240503 02:00:18.924309 16818 ts_manager.cc:194] Registered new tserver with Master: 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167)
I20240503 02:00:18.925472 16818 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.136:39059
I20240503 02:00:18.974726 17270 heartbeater.cc:503] Master 127.0.99.190:41395 was elected leader, sending a full tablet report...
W20240503 02:00:19.001993 17933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:00:19.002240 17933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:00:19.002286 17933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:00:19.005437 17933 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:00:19.005522 17933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:00:19.005607 17933 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.137
I20240503 02:00:19.009279 17933 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:44551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-8/data
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-8/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.137:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-8/data/info.pb
--webserver_interface=127.0.99.137
--webserver_port=0
--tserver_master_addrs=127.0.99.190:41395
--never_fsync=true
--heap_profile_path=/tmp/kudu.17933
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.137
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-8/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 02:00:19.010337 17933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:00:19.011701 17933 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:00:19.022176 17940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:19.023138 17942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:19.024405 17939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:00:19.025545 17933 server_base.cc:1034] running on GCE node
I20240503 02:00:19.026044 17933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:00:19.026706 17933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:00:19.028020 17933 hybrid_clock.cc:648] HybridClock initialized: now 1714701619027992 us; error 41 us; skew 500 ppm
I20240503 02:00:19.030448 17933 webserver.cc:460] Webserver started at http://127.0.99.137:40699/ using document root <none> and password file <none>
I20240503 02:00:19.031106 17933 fs_manager.cc:359] Metadata directory not provided
I20240503 02:00:19.031167 17933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:00:19.031352 17933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:00:19.033288 17933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-8/data/instance:
uuid: "838ef968bb45484bb27f5af6f3989bae"
format_stamp: "Formatted at 2024-05-03 02:00:19 on dist-test-slave-dtz3"
I20240503 02:00:19.033862 17933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-8/wal/instance:
uuid: "838ef968bb45484bb27f5af6f3989bae"
format_stamp: "Formatted at 2024-05-03 02:00:19 on dist-test-slave-dtz3"
I20240503 02:00:19.038395 17933 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240503 02:00:19.041713 17948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:19.043125 17933 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240503 02:00:19.043262 17933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-8/data,/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-8/wal
uuid: "838ef968bb45484bb27f5af6f3989bae"
format_stamp: "Formatted at 2024-05-03 02:00:19 on dist-test-slave-dtz3"
I20240503 02:00:19.043378 17933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-8/wal
metadata directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-8/wal
1 data directories: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-8/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:00:19.057847 17933 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:00:19.058382 17933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:00:19.058518 17933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:00:19.059008 17933 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:00:19.059890 17933 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:00:19.059953 17933 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:19.059993 17933 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:00:19.060030 17933 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:19.069443 17933 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.137:41887
I20240503 02:00:19.069514 18061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.137:41887 every 8 connection(s)
I20240503 02:00:19.070407 17933 server_base.cc:1166] Dumped server information to /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-8/data/info.pb
I20240503 02:00:19.075670 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task42rE7p/build/debug/bin/kudu as pid 17933
I20240503 02:00:19.075845 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-8/wal/instance
I20240503 02:00:19.078840 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task42rE7p/build/debug/bin/kudu
/tmp/dist-test-task42rE7p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-9/wal
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-9/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-9/logs
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-9/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.138:0
--local_ip_for_outbound_sockets=127.0.99.138
--webserver_interface=127.0.99.138
--webserver_port=0
--tserver_master_addrs=127.0.99.190:41395
--builtin_ntp_servers=127.0.99.148:44551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:00:19.081140 18062 heartbeater.cc:347] Connected to a master server at 127.0.99.190:41395
I20240503 02:00:19.081430 18062 heartbeater.cc:464] Registering TS with master...
I20240503 02:00:19.082094 18062 heartbeater.cc:511] Master 127.0.99.190:41395 requested a full tablet report, sending...
I20240503 02:00:19.131479 16818 ts_manager.cc:194] Registered new tserver with Master: 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887)
I20240503 02:00:19.132375 16818 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.137:34749
I20240503 02:00:19.150859 17402 heartbeater.cc:503] Master 127.0.99.190:41395 was elected leader, sending a full tablet report...
W20240503 02:00:19.205607 18065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:00:19.205847 18065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:00:19.205925 18065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:00:19.208855 18065 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:00:19.208928 18065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:00:19.209025 18065 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.138
I20240503 02:00:19.212484 18065 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:44551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-9/data
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-9/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.138:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-9/data/info.pb
--webserver_interface=127.0.99.138
--webserver_port=0
--tserver_master_addrs=127.0.99.190:41395
--never_fsync=true
--heap_profile_path=/tmp/kudu.18065
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.138
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-9/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 02:00:19.213405 18065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:00:19.214521 18065 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:00:19.224529 18074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:19.225363 18072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:19.226675 18071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:00:19.227690 18065 server_base.cc:1034] running on GCE node
I20240503 02:00:19.228183 18065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:00:19.228942 18065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:00:19.230119 18065 hybrid_clock.cc:648] HybridClock initialized: now 1714701619230085 us; error 50 us; skew 500 ppm
I20240503 02:00:19.232506 18065 webserver.cc:460] Webserver started at http://127.0.99.138:35613/ using document root <none> and password file <none>
I20240503 02:00:19.233232 18065 fs_manager.cc:359] Metadata directory not provided
I20240503 02:00:19.233294 18065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:00:19.233507 18065 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:00:19.235688 18065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-9/data/instance:
uuid: "9564311af29a4493831a8db658a085c4"
format_stamp: "Formatted at 2024-05-03 02:00:19 on dist-test-slave-dtz3"
I20240503 02:00:19.236342 18065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-9/wal/instance:
uuid: "9564311af29a4493831a8db658a085c4"
format_stamp: "Formatted at 2024-05-03 02:00:19 on dist-test-slave-dtz3"
I20240503 02:00:19.240164 18065 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20240503 02:00:19.242785 18080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:19.243963 18065 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:19.244068 18065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-9/data,/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-9/wal
uuid: "9564311af29a4493831a8db658a085c4"
format_stamp: "Formatted at 2024-05-03 02:00:19 on dist-test-slave-dtz3"
I20240503 02:00:19.244153 18065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-9/wal
metadata directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-9/wal
1 data directories: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-9/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:00:19.264030 18065 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:00:19.264750 18065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:00:19.264890 18065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:00:19.265499 18065 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:00:19.266515 18065 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:00:19.266566 18065 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:19.266607 18065 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:00:19.266631 18065 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:19.277390 18065 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.138:33181
I20240503 02:00:19.277426 18193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.138:33181 every 8 connection(s)
I20240503 02:00:19.278471 18065 server_base.cc:1166] Dumped server information to /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-9/data/info.pb
I20240503 02:00:19.285799 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task42rE7p/build/debug/bin/kudu as pid 18065
I20240503 02:00:19.285948 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-9/wal/instance
I20240503 02:00:19.289136 18194 heartbeater.cc:347] Connected to a master server at 127.0.99.190:41395
I20240503 02:00:19.289367 18194 heartbeater.cc:464] Registering TS with master...
I20240503 02:00:19.289922 18194 heartbeater.cc:511] Master 127.0.99.190:41395 requested a full tablet report, sending...
I20240503 02:00:19.329152 16817 ts_manager.cc:194] Registered new tserver with Master: 9564311af29a4493831a8db658a085c4 (127.0.99.138:33181)
I20240503 02:00:19.329993 16817 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.138:39377
I20240503 02:00:19.335274 17534 heartbeater.cc:503] Master 127.0.99.190:41395 was elected leader, sending a full tablet report...
I20240503 02:00:19.338238 398 external_mini_cluster.cc:930] 10 TS(s) registered with all masters
I20240503 02:00:19.348624 398 external_mini_cluster.cc:930] 10 TS(s) registered with all masters
I20240503 02:00:19.348750 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task42rE7p/build/debug/bin/kudu with pid 17933
I20240503 02:00:19.355975 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task42rE7p/build/debug/bin/kudu with pid 18065
I20240503 02:00:19.362708 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task42rE7p/build/debug/bin/kudu with pid 17801
I20240503 02:00:19.526312 17666 heartbeater.cc:503] Master 127.0.99.190:41395 was elected leader, sending a full tablet report...
I20240503 02:00:19.758626 17798 heartbeater.cc:503] Master 127.0.99.190:41395 was elected leader, sending a full tablet report...
I20240503 02:00:23.122671 16817 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39916:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 7
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240503 02:00:23.126487 16817 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_0 in case of a server failure: 8 tablet servers would be needed, 7 are available. Consider bringing up more tablet servers.
I20240503 02:00:23.149000 16938 tablet_service.cc:1467] Processing CreateTablet for tablet acfd773c1f384c219d776878407d1144 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ee1e0b20af0b4ecea04ae147701a0482]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:23.149870 16938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acfd773c1f384c219d776878407d1144. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:23.152140 17600 tablet_service.cc:1467] Processing CreateTablet for tablet fbec07350ad04c49afd42e5d88dda6ad (DEFAULT_TABLE table=rebalance_test_table_0 [id=ee1e0b20af0b4ecea04ae147701a0482]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:23.153160 17600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbec07350ad04c49afd42e5d88dda6ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:23.153220 17731 tablet_service.cc:1467] Processing CreateTablet for tablet 81b54f3bcf34485eb11b1bc954a2bd8f (DEFAULT_TABLE table=rebalance_test_table_0 [id=ee1e0b20af0b4ecea04ae147701a0482]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:23.153949 17730 tablet_service.cc:1467] Processing CreateTablet for tablet acfd773c1f384c219d776878407d1144 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ee1e0b20af0b4ecea04ae147701a0482]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:23.154019 17731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81b54f3bcf34485eb11b1bc954a2bd8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:23.154532 17468 tablet_service.cc:1467] Processing CreateTablet for tablet fbec07350ad04c49afd42e5d88dda6ad (DEFAULT_TABLE table=rebalance_test_table_0 [id=ee1e0b20af0b4ecea04ae147701a0482]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:23.155282 17468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbec07350ad04c49afd42e5d88dda6ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:23.155906 17202 tablet_service.cc:1467] Processing CreateTablet for tablet acfd773c1f384c219d776878407d1144 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ee1e0b20af0b4ecea04ae147701a0482]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:23.156981 17732 tablet_service.cc:1467] Processing CreateTablet for tablet fbec07350ad04c49afd42e5d88dda6ad (DEFAULT_TABLE table=rebalance_test_table_0 [id=ee1e0b20af0b4ecea04ae147701a0482]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:23.156976 17466 tablet_service.cc:1467] Processing CreateTablet for tablet acfd773c1f384c219d776878407d1144 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ee1e0b20af0b4ecea04ae147701a0482]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:23.157213 17732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbec07350ad04c49afd42e5d88dda6ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:23.157230 17466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acfd773c1f384c219d776878407d1144. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:23.157688 17467 tablet_service.cc:1467] Processing CreateTablet for tablet 81b54f3bcf34485eb11b1bc954a2bd8f (DEFAULT_TABLE table=rebalance_test_table_0 [id=ee1e0b20af0b4ecea04ae147701a0482]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:23.157902 17467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81b54f3bcf34485eb11b1bc954a2bd8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:23.160418 17203 tablet_service.cc:1467] Processing CreateTablet for tablet 81b54f3bcf34485eb11b1bc954a2bd8f (DEFAULT_TABLE table=rebalance_test_table_0 [id=ee1e0b20af0b4ecea04ae147701a0482]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:23.160840 17204 tablet_service.cc:1467] Processing CreateTablet for tablet fbec07350ad04c49afd42e5d88dda6ad (DEFAULT_TABLE table=rebalance_test_table_0 [id=ee1e0b20af0b4ecea04ae147701a0482]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:23.161607 17730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acfd773c1f384c219d776878407d1144. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:23.163844 17599 tablet_service.cc:1467] Processing CreateTablet for tablet 81b54f3bcf34485eb11b1bc954a2bd8f (DEFAULT_TABLE table=rebalance_test_table_0 [id=ee1e0b20af0b4ecea04ae147701a0482]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:23.164131 17599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81b54f3bcf34485eb11b1bc954a2bd8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:23.164927 17072 tablet_service.cc:1467] Processing CreateTablet for tablet fbec07350ad04c49afd42e5d88dda6ad (DEFAULT_TABLE table=rebalance_test_table_0 [id=ee1e0b20af0b4ecea04ae147701a0482]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:23.165908 17072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbec07350ad04c49afd42e5d88dda6ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:23.167174 16940 tablet_service.cc:1467] Processing CreateTablet for tablet fbec07350ad04c49afd42e5d88dda6ad (DEFAULT_TABLE table=rebalance_test_table_0 [id=ee1e0b20af0b4ecea04ae147701a0482]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:23.167520 16940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbec07350ad04c49afd42e5d88dda6ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:23.169206 18224 tablet_bootstrap.cc:492] T acfd773c1f384c219d776878407d1144 P 6398c8993c3c45dcaab25b42142d5671: Bootstrap starting.
I20240503 02:00:23.170349 17598 tablet_service.cc:1467] Processing CreateTablet for tablet acfd773c1f384c219d776878407d1144 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ee1e0b20af0b4ecea04ae147701a0482]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:23.170684 17598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acfd773c1f384c219d776878407d1144. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:23.172637 18224 tablet_bootstrap.cc:654] T acfd773c1f384c219d776878407d1144 P 6398c8993c3c45dcaab25b42142d5671: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:23.146488 16939 tablet_service.cc:1467] Processing CreateTablet for tablet 81b54f3bcf34485eb11b1bc954a2bd8f (DEFAULT_TABLE table=rebalance_test_table_0 [id=ee1e0b20af0b4ecea04ae147701a0482]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:23.172933 17202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acfd773c1f384c219d776878407d1144. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:23.172987 17071 tablet_service.cc:1467] Processing CreateTablet for tablet acfd773c1f384c219d776878407d1144 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ee1e0b20af0b4ecea04ae147701a0482]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:23.173161 16939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81b54f3bcf34485eb11b1bc954a2bd8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:23.173246 17071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acfd773c1f384c219d776878407d1144. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:23.173666 18224 log.cc:824] T acfd773c1f384c219d776878407d1144 P 6398c8993c3c45dcaab25b42142d5671: Log is configured to *not* fsync() on all Append() calls
I20240503 02:00:23.175179 17203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81b54f3bcf34485eb11b1bc954a2bd8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:23.176760 17204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbec07350ad04c49afd42e5d88dda6ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:23.178488 18221 tablet_bootstrap.cc:492] T fbec07350ad04c49afd42e5d88dda6ad P 002cc3411acb4fdfa0b67c6f57f687a6: Bootstrap starting.
I20240503 02:00:23.180142 17072 tablet_service.cc:1467] Processing CreateTablet for tablet 81b54f3bcf34485eb11b1bc954a2bd8f (DEFAULT_TABLE table=rebalance_test_table_0 [id=ee1e0b20af0b4ecea04ae147701a0482]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:23.180447 18227 tablet_bootstrap.cc:492] T acfd773c1f384c219d776878407d1144 P 5b8291b26a7f448b8636187d1caf86d0: Bootstrap starting.
I20240503 02:00:23.180490 17072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81b54f3bcf34485eb11b1bc954a2bd8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:23.180996 18221 tablet_bootstrap.cc:654] T fbec07350ad04c49afd42e5d88dda6ad P 002cc3411acb4fdfa0b67c6f57f687a6: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:23.182015 18221 log.cc:824] T fbec07350ad04c49afd42e5d88dda6ad P 002cc3411acb4fdfa0b67c6f57f687a6: Log is configured to *not* fsync() on all Append() calls
I20240503 02:00:23.182498 18227 tablet_bootstrap.cc:654] T acfd773c1f384c219d776878407d1144 P 5b8291b26a7f448b8636187d1caf86d0: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:23.183316 18227 log.cc:824] T acfd773c1f384c219d776878407d1144 P 5b8291b26a7f448b8636187d1caf86d0: Log is configured to *not* fsync() on all Append() calls
I20240503 02:00:23.184032 18221 tablet_bootstrap.cc:492] T fbec07350ad04c49afd42e5d88dda6ad P 002cc3411acb4fdfa0b67c6f57f687a6: No bootstrap required, opened a new log
I20240503 02:00:23.184213 18221 ts_tablet_manager.cc:1367] T fbec07350ad04c49afd42e5d88dda6ad P 002cc3411acb4fdfa0b67c6f57f687a6: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240503 02:00:23.185019 18227 tablet_bootstrap.cc:492] T acfd773c1f384c219d776878407d1144 P 5b8291b26a7f448b8636187d1caf86d0: No bootstrap required, opened a new log
I20240503 02:00:23.186110 18223 tablet_bootstrap.cc:492] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa: Bootstrap starting.
I20240503 02:00:23.187301 18221 raft_consensus.cc:348] T fbec07350ad04c49afd42e5d88dda6ad P 002cc3411acb4fdfa0b67c6f57f687a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } }
I20240503 02:00:23.187510 18221 raft_consensus.cc:374] T fbec07350ad04c49afd42e5d88dda6ad P 002cc3411acb4fdfa0b67c6f57f687a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:23.187584 18221 raft_consensus.cc:729] T fbec07350ad04c49afd42e5d88dda6ad P 002cc3411acb4fdfa0b67c6f57f687a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 002cc3411acb4fdfa0b67c6f57f687a6, State: Initialized, Role: FOLLOWER
I20240503 02:00:23.187372 17334 tablet_service.cc:1467] Processing CreateTablet for tablet acfd773c1f384c219d776878407d1144 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ee1e0b20af0b4ecea04ae147701a0482]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:23.188066 18221 consensus_queue.cc:260] T fbec07350ad04c49afd42e5d88dda6ad P 002cc3411acb4fdfa0b67c6f57f687a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } }
I20240503 02:00:23.188143 18223 tablet_bootstrap.cc:654] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:23.188445 17334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acfd773c1f384c219d776878407d1144. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:23.188905 18223 log.cc:824] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa: Log is configured to *not* fsync() on all Append() calls
I20240503 02:00:23.189596 18225 tablet_bootstrap.cc:492] T fbec07350ad04c49afd42e5d88dda6ad P 89dd09f8aaee4b249ae3ce924a422a67: Bootstrap starting.
I20240503 02:00:23.190097 18221 ts_tablet_manager.cc:1398] T fbec07350ad04c49afd42e5d88dda6ad P 002cc3411acb4fdfa0b67c6f57f687a6: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240503 02:00:23.190326 18221 tablet_bootstrap.cc:492] T acfd773c1f384c219d776878407d1144 P 002cc3411acb4fdfa0b67c6f57f687a6: Bootstrap starting.
I20240503 02:00:23.191078 18221 tablet_bootstrap.cc:654] T acfd773c1f384c219d776878407d1144 P 002cc3411acb4fdfa0b67c6f57f687a6: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:23.188704 17335 tablet_service.cc:1467] Processing CreateTablet for tablet 81b54f3bcf34485eb11b1bc954a2bd8f (DEFAULT_TABLE table=rebalance_test_table_0 [id=ee1e0b20af0b4ecea04ae147701a0482]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:23.192855 18221 tablet_bootstrap.cc:492] T acfd773c1f384c219d776878407d1144 P 002cc3411acb4fdfa0b67c6f57f687a6: No bootstrap required, opened a new log
I20240503 02:00:23.192878 17335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81b54f3bcf34485eb11b1bc954a2bd8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:23.192943 18221 ts_tablet_manager.cc:1367] T acfd773c1f384c219d776878407d1144 P 002cc3411acb4fdfa0b67c6f57f687a6: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 02:00:23.193671 18221 raft_consensus.cc:348] T acfd773c1f384c219d776878407d1144 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:23.193754 18221 raft_consensus.cc:374] T acfd773c1f384c219d776878407d1144 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:23.193774 18221 raft_consensus.cc:729] T acfd773c1f384c219d776878407d1144 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 002cc3411acb4fdfa0b67c6f57f687a6, State: Initialized, Role: FOLLOWER
I20240503 02:00:23.193881 18221 consensus_queue.cc:260] T acfd773c1f384c219d776878407d1144 P 002cc3411acb4fdfa0b67c6f57f687a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:23.194038 18225 tablet_bootstrap.cc:654] T fbec07350ad04c49afd42e5d88dda6ad P 89dd09f8aaee4b249ae3ce924a422a67: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:23.194464 18221 ts_tablet_manager.cc:1398] T acfd773c1f384c219d776878407d1144 P 002cc3411acb4fdfa0b67c6f57f687a6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:23.194625 18221 tablet_bootstrap.cc:492] T 81b54f3bcf34485eb11b1bc954a2bd8f P 002cc3411acb4fdfa0b67c6f57f687a6: Bootstrap starting.
I20240503 02:00:23.194730 18225 log.cc:824] T fbec07350ad04c49afd42e5d88dda6ad P 89dd09f8aaee4b249ae3ce924a422a67: Log is configured to *not* fsync() on all Append() calls
I20240503 02:00:23.196012 18221 tablet_bootstrap.cc:654] T 81b54f3bcf34485eb11b1bc954a2bd8f P 002cc3411acb4fdfa0b67c6f57f687a6: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:23.196277 18223 tablet_bootstrap.cc:492] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa: No bootstrap required, opened a new log
I20240503 02:00:23.196462 18223 ts_tablet_manager.cc:1367] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20240503 02:00:23.196506 18234 tablet_bootstrap.cc:492] T 81b54f3bcf34485eb11b1bc954a2bd8f P d43ed6b8663947d99a72cd922cc323aa: Bootstrap starting.
I20240503 02:00:23.188921 17336 tablet_service.cc:1467] Processing CreateTablet for tablet fbec07350ad04c49afd42e5d88dda6ad (DEFAULT_TABLE table=rebalance_test_table_0 [id=ee1e0b20af0b4ecea04ae147701a0482]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:23.197098 17336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbec07350ad04c49afd42e5d88dda6ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:23.197341 18221 tablet_bootstrap.cc:492] T 81b54f3bcf34485eb11b1bc954a2bd8f P 002cc3411acb4fdfa0b67c6f57f687a6: No bootstrap required, opened a new log
I20240503 02:00:23.197418 18221 ts_tablet_manager.cc:1367] T 81b54f3bcf34485eb11b1bc954a2bd8f P 002cc3411acb4fdfa0b67c6f57f687a6: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240503 02:00:23.197558 18222 tablet_bootstrap.cc:492] T fbec07350ad04c49afd42e5d88dda6ad P c843193b801b4f1a88f630ed3ed0f375: Bootstrap starting.
I20240503 02:00:23.197851 18221 raft_consensus.cc:348] T 81b54f3bcf34485eb11b1bc954a2bd8f P 002cc3411acb4fdfa0b67c6f57f687a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:23.197966 18221 raft_consensus.cc:374] T 81b54f3bcf34485eb11b1bc954a2bd8f P 002cc3411acb4fdfa0b67c6f57f687a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:23.198009 18221 raft_consensus.cc:729] T 81b54f3bcf34485eb11b1bc954a2bd8f P 002cc3411acb4fdfa0b67c6f57f687a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 002cc3411acb4fdfa0b67c6f57f687a6, State: Initialized, Role: FOLLOWER
I20240503 02:00:23.198132 18221 consensus_queue.cc:260] T 81b54f3bcf34485eb11b1bc954a2bd8f P 002cc3411acb4fdfa0b67c6f57f687a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:23.198372 18221 ts_tablet_manager.cc:1398] T 81b54f3bcf34485eb11b1bc954a2bd8f P 002cc3411acb4fdfa0b67c6f57f687a6: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:23.198633 18234 tablet_bootstrap.cc:654] T 81b54f3bcf34485eb11b1bc954a2bd8f P d43ed6b8663947d99a72cd922cc323aa: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:23.200219 18234 log.cc:824] T 81b54f3bcf34485eb11b1bc954a2bd8f P d43ed6b8663947d99a72cd922cc323aa: Log is configured to *not* fsync() on all Append() calls
I20240503 02:00:23.201617 18222 tablet_bootstrap.cc:654] T fbec07350ad04c49afd42e5d88dda6ad P c843193b801b4f1a88f630ed3ed0f375: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:23.201617 18227 ts_tablet_manager.cc:1367] T acfd773c1f384c219d776878407d1144 P 5b8291b26a7f448b8636187d1caf86d0: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.000s
I20240503 02:00:23.201989 18234 tablet_bootstrap.cc:492] T 81b54f3bcf34485eb11b1bc954a2bd8f P d43ed6b8663947d99a72cd922cc323aa: No bootstrap required, opened a new log
I20240503 02:00:23.202159 18234 ts_tablet_manager.cc:1367] T 81b54f3bcf34485eb11b1bc954a2bd8f P d43ed6b8663947d99a72cd922cc323aa: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240503 02:00:23.202415 18222 log.cc:824] T fbec07350ad04c49afd42e5d88dda6ad P c843193b801b4f1a88f630ed3ed0f375: Log is configured to *not* fsync() on all Append() calls
I20240503 02:00:23.203609 18225 tablet_bootstrap.cc:492] T fbec07350ad04c49afd42e5d88dda6ad P 89dd09f8aaee4b249ae3ce924a422a67: No bootstrap required, opened a new log
I20240503 02:00:23.203778 18225 ts_tablet_manager.cc:1367] T fbec07350ad04c49afd42e5d88dda6ad P 89dd09f8aaee4b249ae3ce924a422a67: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.000s
I20240503 02:00:23.203595 18223 raft_consensus.cc:348] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:23.204099 18223 raft_consensus.cc:374] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:23.204198 18223 raft_consensus.cc:729] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16d437fe629643948c95d628926d5ffa, State: Initialized, Role: FOLLOWER
I20240503 02:00:23.204372 18224 tablet_bootstrap.cc:492] T acfd773c1f384c219d776878407d1144 P 6398c8993c3c45dcaab25b42142d5671: No bootstrap required, opened a new log
I20240503 02:00:23.204591 18224 ts_tablet_manager.cc:1367] T acfd773c1f384c219d776878407d1144 P 6398c8993c3c45dcaab25b42142d5671: Time spent bootstrapping tablet: real 0.036s user 0.006s sys 0.000s
I20240503 02:00:23.204896 18223 consensus_queue.cc:260] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:23.205356 18234 raft_consensus.cc:348] T 81b54f3bcf34485eb11b1bc954a2bd8f P d43ed6b8663947d99a72cd922cc323aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:23.205545 18234 raft_consensus.cc:374] T 81b54f3bcf34485eb11b1bc954a2bd8f P d43ed6b8663947d99a72cd922cc323aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:23.205608 18234 raft_consensus.cc:729] T 81b54f3bcf34485eb11b1bc954a2bd8f P d43ed6b8663947d99a72cd922cc323aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d43ed6b8663947d99a72cd922cc323aa, State: Initialized, Role: FOLLOWER
I20240503 02:00:23.206017 18223 ts_tablet_manager.cc:1398] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20240503 02:00:23.206007 18234 consensus_queue.cc:260] T 81b54f3bcf34485eb11b1bc954a2bd8f P d43ed6b8663947d99a72cd922cc323aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:23.206496 18223 tablet_bootstrap.cc:492] T 81b54f3bcf34485eb11b1bc954a2bd8f P 16d437fe629643948c95d628926d5ffa: Bootstrap starting.
I20240503 02:00:23.206751 18234 ts_tablet_manager.cc:1398] T 81b54f3bcf34485eb11b1bc954a2bd8f P d43ed6b8663947d99a72cd922cc323aa: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240503 02:00:23.207123 18234 tablet_bootstrap.cc:492] T fbec07350ad04c49afd42e5d88dda6ad P d43ed6b8663947d99a72cd922cc323aa: Bootstrap starting.
I20240503 02:00:23.207713 18223 tablet_bootstrap.cc:654] T 81b54f3bcf34485eb11b1bc954a2bd8f P 16d437fe629643948c95d628926d5ffa: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:23.208149 18234 tablet_bootstrap.cc:654] T fbec07350ad04c49afd42e5d88dda6ad P d43ed6b8663947d99a72cd922cc323aa: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:23.209591 18223 tablet_bootstrap.cc:492] T 81b54f3bcf34485eb11b1bc954a2bd8f P 16d437fe629643948c95d628926d5ffa: No bootstrap required, opened a new log
I20240503 02:00:23.209671 18223 ts_tablet_manager.cc:1367] T 81b54f3bcf34485eb11b1bc954a2bd8f P 16d437fe629643948c95d628926d5ffa: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:23.210239 18223 raft_consensus.cc:348] T 81b54f3bcf34485eb11b1bc954a2bd8f P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:23.210367 18223 raft_consensus.cc:374] T 81b54f3bcf34485eb11b1bc954a2bd8f P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:23.210402 18223 raft_consensus.cc:729] T 81b54f3bcf34485eb11b1bc954a2bd8f P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16d437fe629643948c95d628926d5ffa, State: Initialized, Role: FOLLOWER
I20240503 02:00:23.210362 18224 raft_consensus.cc:348] T acfd773c1f384c219d776878407d1144 P 6398c8993c3c45dcaab25b42142d5671 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:23.210534 18223 consensus_queue.cc:260] T 81b54f3bcf34485eb11b1bc954a2bd8f P 16d437fe629643948c95d628926d5ffa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:23.210639 18224 raft_consensus.cc:374] T acfd773c1f384c219d776878407d1144 P 6398c8993c3c45dcaab25b42142d5671 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:23.210701 18224 raft_consensus.cc:729] T acfd773c1f384c219d776878407d1144 P 6398c8993c3c45dcaab25b42142d5671 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6398c8993c3c45dcaab25b42142d5671, State: Initialized, Role: FOLLOWER
I20240503 02:00:23.210804 18223 ts_tablet_manager.cc:1398] T 81b54f3bcf34485eb11b1bc954a2bd8f P 16d437fe629643948c95d628926d5ffa: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:23.210965 18223 tablet_bootstrap.cc:492] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa: Bootstrap starting.
I20240503 02:00:23.211139 18224 consensus_queue.cc:260] T acfd773c1f384c219d776878407d1144 P 6398c8993c3c45dcaab25b42142d5671 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:23.211921 18223 tablet_bootstrap.cc:654] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:23.211963 18224 ts_tablet_manager.cc:1398] T acfd773c1f384c219d776878407d1144 P 6398c8993c3c45dcaab25b42142d5671: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20240503 02:00:23.212304 18224 tablet_bootstrap.cc:492] T fbec07350ad04c49afd42e5d88dda6ad P 6398c8993c3c45dcaab25b42142d5671: Bootstrap starting.
I20240503 02:00:23.212355 18225 raft_consensus.cc:348] T fbec07350ad04c49afd42e5d88dda6ad P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } }
I20240503 02:00:23.212569 18225 raft_consensus.cc:374] T fbec07350ad04c49afd42e5d88dda6ad P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:23.212622 18225 raft_consensus.cc:729] T fbec07350ad04c49afd42e5d88dda6ad P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89dd09f8aaee4b249ae3ce924a422a67, State: Initialized, Role: FOLLOWER
I20240503 02:00:23.213086 18225 consensus_queue.cc:260] T fbec07350ad04c49afd42e5d88dda6ad P 89dd09f8aaee4b249ae3ce924a422a67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } }
I20240503 02:00:23.213344 18224 tablet_bootstrap.cc:654] T fbec07350ad04c49afd42e5d88dda6ad P 6398c8993c3c45dcaab25b42142d5671: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:23.213928 18225 ts_tablet_manager.cc:1398] T fbec07350ad04c49afd42e5d88dda6ad P 89dd09f8aaee4b249ae3ce924a422a67: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20240503 02:00:23.214320 18225 tablet_bootstrap.cc:492] T acfd773c1f384c219d776878407d1144 P 89dd09f8aaee4b249ae3ce924a422a67: Bootstrap starting.
I20240503 02:00:23.215286 18225 tablet_bootstrap.cc:654] T acfd773c1f384c219d776878407d1144 P 89dd09f8aaee4b249ae3ce924a422a67: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:23.216471 18234 tablet_bootstrap.cc:492] T fbec07350ad04c49afd42e5d88dda6ad P d43ed6b8663947d99a72cd922cc323aa: No bootstrap required, opened a new log
I20240503 02:00:23.216471 18225 tablet_bootstrap.cc:492] T acfd773c1f384c219d776878407d1144 P 89dd09f8aaee4b249ae3ce924a422a67: No bootstrap required, opened a new log
I20240503 02:00:23.216584 18234 ts_tablet_manager.cc:1367] T fbec07350ad04c49afd42e5d88dda6ad P d43ed6b8663947d99a72cd922cc323aa: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.000s
I20240503 02:00:23.216693 18225 ts_tablet_manager.cc:1367] T acfd773c1f384c219d776878407d1144 P 89dd09f8aaee4b249ae3ce924a422a67: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:23.217200 18234 raft_consensus.cc:348] T fbec07350ad04c49afd42e5d88dda6ad P d43ed6b8663947d99a72cd922cc323aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } }
I20240503 02:00:23.217234 18225 raft_consensus.cc:348] T acfd773c1f384c219d776878407d1144 P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:23.217339 18234 raft_consensus.cc:374] T fbec07350ad04c49afd42e5d88dda6ad P d43ed6b8663947d99a72cd922cc323aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:23.217381 18234 raft_consensus.cc:729] T fbec07350ad04c49afd42e5d88dda6ad P d43ed6b8663947d99a72cd922cc323aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d43ed6b8663947d99a72cd922cc323aa, State: Initialized, Role: FOLLOWER
I20240503 02:00:23.217279 18227 raft_consensus.cc:348] T acfd773c1f384c219d776878407d1144 P 5b8291b26a7f448b8636187d1caf86d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:23.217453 18225 raft_consensus.cc:374] T acfd773c1f384c219d776878407d1144 P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:23.217528 18227 raft_consensus.cc:374] T acfd773c1f384c219d776878407d1144 P 5b8291b26a7f448b8636187d1caf86d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:23.217593 18227 raft_consensus.cc:729] T acfd773c1f384c219d776878407d1144 P 5b8291b26a7f448b8636187d1caf86d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b8291b26a7f448b8636187d1caf86d0, State: Initialized, Role: FOLLOWER
I20240503 02:00:23.217528 18234 consensus_queue.cc:260] T fbec07350ad04c49afd42e5d88dda6ad P d43ed6b8663947d99a72cd922cc323aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } }
I20240503 02:00:23.217646 18222 tablet_bootstrap.cc:492] T fbec07350ad04c49afd42e5d88dda6ad P c843193b801b4f1a88f630ed3ed0f375: No bootstrap required, opened a new log
I20240503 02:00:23.217558 18225 raft_consensus.cc:729] T acfd773c1f384c219d776878407d1144 P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89dd09f8aaee4b249ae3ce924a422a67, State: Initialized, Role: FOLLOWER
I20240503 02:00:23.217832 18234 ts_tablet_manager.cc:1398] T fbec07350ad04c49afd42e5d88dda6ad P d43ed6b8663947d99a72cd922cc323aa: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20240503 02:00:23.217861 18225 consensus_queue.cc:260] T acfd773c1f384c219d776878407d1144 P 89dd09f8aaee4b249ae3ce924a422a67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:23.218036 18234 tablet_bootstrap.cc:492] T acfd773c1f384c219d776878407d1144 P d43ed6b8663947d99a72cd922cc323aa: Bootstrap starting.
I20240503 02:00:23.218041 18227 consensus_queue.cc:260] T acfd773c1f384c219d776878407d1144 P 5b8291b26a7f448b8636187d1caf86d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:23.218444 18225 ts_tablet_manager.cc:1398] T acfd773c1f384c219d776878407d1144 P 89dd09f8aaee4b249ae3ce924a422a67: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:23.218926 18227 ts_tablet_manager.cc:1398] T acfd773c1f384c219d776878407d1144 P 5b8291b26a7f448b8636187d1caf86d0: Time spent starting tablet: real 0.017s user 0.006s sys 0.000s
I20240503 02:00:23.218971 18234 tablet_bootstrap.cc:654] T acfd773c1f384c219d776878407d1144 P d43ed6b8663947d99a72cd922cc323aa: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:23.219266 18227 tablet_bootstrap.cc:492] T 81b54f3bcf34485eb11b1bc954a2bd8f P 5b8291b26a7f448b8636187d1caf86d0: Bootstrap starting.
I20240503 02:00:23.220147 18225 tablet_bootstrap.cc:492] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67: Bootstrap starting.
I20240503 02:00:23.220445 18227 tablet_bootstrap.cc:654] T 81b54f3bcf34485eb11b1bc954a2bd8f P 5b8291b26a7f448b8636187d1caf86d0: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:23.217896 18222 ts_tablet_manager.cc:1367] T fbec07350ad04c49afd42e5d88dda6ad P c843193b801b4f1a88f630ed3ed0f375: Time spent bootstrapping tablet: real 0.021s user 0.005s sys 0.000s
I20240503 02:00:23.221246 18225 tablet_bootstrap.cc:654] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:23.221889 18227 tablet_bootstrap.cc:492] T 81b54f3bcf34485eb11b1bc954a2bd8f P 5b8291b26a7f448b8636187d1caf86d0: No bootstrap required, opened a new log
I20240503 02:00:23.222026 18227 ts_tablet_manager.cc:1367] T 81b54f3bcf34485eb11b1bc954a2bd8f P 5b8291b26a7f448b8636187d1caf86d0: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 02:00:23.222638 18227 raft_consensus.cc:348] T 81b54f3bcf34485eb11b1bc954a2bd8f P 5b8291b26a7f448b8636187d1caf86d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:23.222762 18227 raft_consensus.cc:374] T 81b54f3bcf34485eb11b1bc954a2bd8f P 5b8291b26a7f448b8636187d1caf86d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:23.222798 18227 raft_consensus.cc:729] T 81b54f3bcf34485eb11b1bc954a2bd8f P 5b8291b26a7f448b8636187d1caf86d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b8291b26a7f448b8636187d1caf86d0, State: Initialized, Role: FOLLOWER
I20240503 02:00:23.222956 18227 consensus_queue.cc:260] T 81b54f3bcf34485eb11b1bc954a2bd8f P 5b8291b26a7f448b8636187d1caf86d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:23.223273 18227 ts_tablet_manager.cc:1398] T 81b54f3bcf34485eb11b1bc954a2bd8f P 5b8291b26a7f448b8636187d1caf86d0: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20240503 02:00:23.223438 18227 tablet_bootstrap.cc:492] T fbec07350ad04c49afd42e5d88dda6ad P 5b8291b26a7f448b8636187d1caf86d0: Bootstrap starting.
I20240503 02:00:23.225101 18227 tablet_bootstrap.cc:654] T fbec07350ad04c49afd42e5d88dda6ad P 5b8291b26a7f448b8636187d1caf86d0: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:23.225422 18225 tablet_bootstrap.cc:492] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67: No bootstrap required, opened a new log
I20240503 02:00:23.225546 18225 ts_tablet_manager.cc:1367] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240503 02:00:23.226305 18224 tablet_bootstrap.cc:492] T fbec07350ad04c49afd42e5d88dda6ad P 6398c8993c3c45dcaab25b42142d5671: No bootstrap required, opened a new log
I20240503 02:00:23.226392 18234 tablet_bootstrap.cc:492] T acfd773c1f384c219d776878407d1144 P d43ed6b8663947d99a72cd922cc323aa: No bootstrap required, opened a new log
I20240503 02:00:23.226419 18224 ts_tablet_manager.cc:1367] T fbec07350ad04c49afd42e5d88dda6ad P 6398c8993c3c45dcaab25b42142d5671: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240503 02:00:23.226341 18225 raft_consensus.cc:348] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:23.226464 18225 raft_consensus.cc:374] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:23.226481 18234 ts_tablet_manager.cc:1367] T acfd773c1f384c219d776878407d1144 P d43ed6b8663947d99a72cd922cc323aa: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.002s
I20240503 02:00:23.226500 18225 raft_consensus.cc:729] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89dd09f8aaee4b249ae3ce924a422a67, State: Initialized, Role: FOLLOWER
I20240503 02:00:23.226660 18225 consensus_queue.cc:260] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:23.227022 18225 ts_tablet_manager.cc:1398] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:23.227035 18224 raft_consensus.cc:348] T fbec07350ad04c49afd42e5d88dda6ad P 6398c8993c3c45dcaab25b42142d5671 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } }
I20240503 02:00:23.227191 18224 raft_consensus.cc:374] T fbec07350ad04c49afd42e5d88dda6ad P 6398c8993c3c45dcaab25b42142d5671 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:23.227252 18224 raft_consensus.cc:729] T fbec07350ad04c49afd42e5d88dda6ad P 6398c8993c3c45dcaab25b42142d5671 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6398c8993c3c45dcaab25b42142d5671, State: Initialized, Role: FOLLOWER
I20240503 02:00:23.227053 18222 raft_consensus.cc:348] T fbec07350ad04c49afd42e5d88dda6ad P c843193b801b4f1a88f630ed3ed0f375 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } }
I20240503 02:00:23.227351 18222 raft_consensus.cc:374] T fbec07350ad04c49afd42e5d88dda6ad P c843193b801b4f1a88f630ed3ed0f375 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:23.227409 18222 raft_consensus.cc:729] T fbec07350ad04c49afd42e5d88dda6ad P c843193b801b4f1a88f630ed3ed0f375 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c843193b801b4f1a88f630ed3ed0f375, State: Initialized, Role: FOLLOWER
I20240503 02:00:23.227397 18224 consensus_queue.cc:260] T fbec07350ad04c49afd42e5d88dda6ad P 6398c8993c3c45dcaab25b42142d5671 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } }
I20240503 02:00:23.227486 18234 raft_consensus.cc:348] T acfd773c1f384c219d776878407d1144 P d43ed6b8663947d99a72cd922cc323aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:23.227613 18234 raft_consensus.cc:374] T acfd773c1f384c219d776878407d1144 P d43ed6b8663947d99a72cd922cc323aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:23.227654 18234 raft_consensus.cc:729] T acfd773c1f384c219d776878407d1144 P d43ed6b8663947d99a72cd922cc323aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d43ed6b8663947d99a72cd922cc323aa, State: Initialized, Role: FOLLOWER
I20240503 02:00:23.227723 18224 ts_tablet_manager.cc:1398] T fbec07350ad04c49afd42e5d88dda6ad P 6398c8993c3c45dcaab25b42142d5671: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 02:00:23.227905 18224 tablet_bootstrap.cc:492] T 81b54f3bcf34485eb11b1bc954a2bd8f P 6398c8993c3c45dcaab25b42142d5671: Bootstrap starting.
I20240503 02:00:23.227838 18234 consensus_queue.cc:260] T acfd773c1f384c219d776878407d1144 P d43ed6b8663947d99a72cd922cc323aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:23.228003 18222 consensus_queue.cc:260] T fbec07350ad04c49afd42e5d88dda6ad P c843193b801b4f1a88f630ed3ed0f375 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } }
I20240503 02:00:23.228988 18222 ts_tablet_manager.cc:1398] T fbec07350ad04c49afd42e5d88dda6ad P c843193b801b4f1a88f630ed3ed0f375: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20240503 02:00:23.229349 18222 tablet_bootstrap.cc:492] T 81b54f3bcf34485eb11b1bc954a2bd8f P c843193b801b4f1a88f630ed3ed0f375: Bootstrap starting.
I20240503 02:00:23.229424 18224 tablet_bootstrap.cc:654] T 81b54f3bcf34485eb11b1bc954a2bd8f P 6398c8993c3c45dcaab25b42142d5671: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:23.229583 18234 ts_tablet_manager.cc:1398] T acfd773c1f384c219d776878407d1144 P d43ed6b8663947d99a72cd922cc323aa: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 02:00:23.230337 18222 tablet_bootstrap.cc:654] T 81b54f3bcf34485eb11b1bc954a2bd8f P c843193b801b4f1a88f630ed3ed0f375: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:23.230515 18223 tablet_bootstrap.cc:492] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa: No bootstrap required, opened a new log
I20240503 02:00:23.230629 18223 ts_tablet_manager.cc:1367] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.000s
I20240503 02:00:23.231154 18223 raft_consensus.cc:348] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } }
I20240503 02:00:23.231729 18223 raft_consensus.cc:374] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:23.231783 18223 raft_consensus.cc:729] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16d437fe629643948c95d628926d5ffa, State: Initialized, Role: FOLLOWER
I20240503 02:00:23.231956 18223 consensus_queue.cc:260] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } }
I20240503 02:00:23.232333 18223 ts_tablet_manager.cc:1398] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:23.233430 18224 tablet_bootstrap.cc:492] T 81b54f3bcf34485eb11b1bc954a2bd8f P 6398c8993c3c45dcaab25b42142d5671: No bootstrap required, opened a new log
I20240503 02:00:23.233680 18227 tablet_bootstrap.cc:492] T fbec07350ad04c49afd42e5d88dda6ad P 5b8291b26a7f448b8636187d1caf86d0: No bootstrap required, opened a new log
I20240503 02:00:23.233799 18227 ts_tablet_manager.cc:1367] T fbec07350ad04c49afd42e5d88dda6ad P 5b8291b26a7f448b8636187d1caf86d0: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240503 02:00:23.234275 18227 raft_consensus.cc:348] T fbec07350ad04c49afd42e5d88dda6ad P 5b8291b26a7f448b8636187d1caf86d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } }
I20240503 02:00:23.234407 18227 raft_consensus.cc:374] T fbec07350ad04c49afd42e5d88dda6ad P 5b8291b26a7f448b8636187d1caf86d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:23.234442 18227 raft_consensus.cc:729] T fbec07350ad04c49afd42e5d88dda6ad P 5b8291b26a7f448b8636187d1caf86d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b8291b26a7f448b8636187d1caf86d0, State: Initialized, Role: FOLLOWER
I20240503 02:00:23.234580 18227 consensus_queue.cc:260] T fbec07350ad04c49afd42e5d88dda6ad P 5b8291b26a7f448b8636187d1caf86d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } }
I20240503 02:00:23.234902 18227 ts_tablet_manager.cc:1398] T fbec07350ad04c49afd42e5d88dda6ad P 5b8291b26a7f448b8636187d1caf86d0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:23.233711 18224 ts_tablet_manager.cc:1367] T 81b54f3bcf34485eb11b1bc954a2bd8f P 6398c8993c3c45dcaab25b42142d5671: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20240503 02:00:23.235517 18222 tablet_bootstrap.cc:492] T 81b54f3bcf34485eb11b1bc954a2bd8f P c843193b801b4f1a88f630ed3ed0f375: No bootstrap required, opened a new log
I20240503 02:00:23.235448 18224 raft_consensus.cc:348] T 81b54f3bcf34485eb11b1bc954a2bd8f P 6398c8993c3c45dcaab25b42142d5671 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:23.235571 18224 raft_consensus.cc:374] T 81b54f3bcf34485eb11b1bc954a2bd8f P 6398c8993c3c45dcaab25b42142d5671 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:23.235594 18222 ts_tablet_manager.cc:1367] T 81b54f3bcf34485eb11b1bc954a2bd8f P c843193b801b4f1a88f630ed3ed0f375: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 02:00:23.235605 18224 raft_consensus.cc:729] T 81b54f3bcf34485eb11b1bc954a2bd8f P 6398c8993c3c45dcaab25b42142d5671 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6398c8993c3c45dcaab25b42142d5671, State: Initialized, Role: FOLLOWER
I20240503 02:00:23.235738 18224 consensus_queue.cc:260] T 81b54f3bcf34485eb11b1bc954a2bd8f P 6398c8993c3c45dcaab25b42142d5671 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:23.236056 18224 ts_tablet_manager.cc:1398] T 81b54f3bcf34485eb11b1bc954a2bd8f P 6398c8993c3c45dcaab25b42142d5671: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:23.236248 18222 raft_consensus.cc:348] T 81b54f3bcf34485eb11b1bc954a2bd8f P c843193b801b4f1a88f630ed3ed0f375 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:23.236351 18222 raft_consensus.cc:374] T 81b54f3bcf34485eb11b1bc954a2bd8f P c843193b801b4f1a88f630ed3ed0f375 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:23.236390 18222 raft_consensus.cc:729] T 81b54f3bcf34485eb11b1bc954a2bd8f P c843193b801b4f1a88f630ed3ed0f375 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c843193b801b4f1a88f630ed3ed0f375, State: Initialized, Role: FOLLOWER
I20240503 02:00:23.236518 18222 consensus_queue.cc:260] T 81b54f3bcf34485eb11b1bc954a2bd8f P c843193b801b4f1a88f630ed3ed0f375 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:23.236835 18222 ts_tablet_manager.cc:1398] T 81b54f3bcf34485eb11b1bc954a2bd8f P c843193b801b4f1a88f630ed3ed0f375: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:23.237010 18222 tablet_bootstrap.cc:492] T acfd773c1f384c219d776878407d1144 P c843193b801b4f1a88f630ed3ed0f375: Bootstrap starting.
I20240503 02:00:23.237958 18222 tablet_bootstrap.cc:654] T acfd773c1f384c219d776878407d1144 P c843193b801b4f1a88f630ed3ed0f375: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:23.239427 18222 tablet_bootstrap.cc:492] T acfd773c1f384c219d776878407d1144 P c843193b801b4f1a88f630ed3ed0f375: No bootstrap required, opened a new log
I20240503 02:00:23.239540 18222 ts_tablet_manager.cc:1367] T acfd773c1f384c219d776878407d1144 P c843193b801b4f1a88f630ed3ed0f375: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:23.239943 18222 raft_consensus.cc:348] T acfd773c1f384c219d776878407d1144 P c843193b801b4f1a88f630ed3ed0f375 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:23.240034 18222 raft_consensus.cc:374] T acfd773c1f384c219d776878407d1144 P c843193b801b4f1a88f630ed3ed0f375 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:23.240056 18222 raft_consensus.cc:729] T acfd773c1f384c219d776878407d1144 P c843193b801b4f1a88f630ed3ed0f375 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c843193b801b4f1a88f630ed3ed0f375, State: Initialized, Role: FOLLOWER
I20240503 02:00:23.240168 18222 consensus_queue.cc:260] T acfd773c1f384c219d776878407d1144 P c843193b801b4f1a88f630ed3ed0f375 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:23.240401 18222 ts_tablet_manager.cc:1398] T acfd773c1f384c219d776878407d1144 P c843193b801b4f1a88f630ed3ed0f375: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20240503 02:00:23.242101 17139 tablet.cc:2369] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:00:23.253099 18243 raft_consensus.cc:482] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:23.253326 18243 raft_consensus.cc:504] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:23.255033 18243 leader_election.cc:288] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6398c8993c3c45dcaab25b42142d5671 (127.0.99.129:36983), 5b8291b26a7f448b8636187d1caf86d0 (127.0.99.131:44479), c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134:44343), d43ed6b8663947d99a72cd922cc323aa (127.0.99.132:40023), 16d437fe629643948c95d628926d5ffa (127.0.99.133:46581), 002cc3411acb4fdfa0b67c6f57f687a6 (127.0.99.135:38989)
I20240503 02:00:23.255086 18237 raft_consensus.cc:482] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:23.255278 18237 raft_consensus.cc:504] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:23.256989 18260 raft_consensus.cc:482] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:23.257027 18237 leader_election.cc:288] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6398c8993c3c45dcaab25b42142d5671 (127.0.99.129:36983), 89dd09f8aaee4b249ae3ce924a422a67 (127.0.99.130:38889), 5b8291b26a7f448b8636187d1caf86d0 (127.0.99.131:44479), d43ed6b8663947d99a72cd922cc323aa (127.0.99.132:40023), c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134:44343), 002cc3411acb4fdfa0b67c6f57f687a6 (127.0.99.135:38989)
I20240503 02:00:23.257092 18260 raft_consensus.cc:504] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } }
I20240503 02:00:23.257711 18260 leader_election.cc:288] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6398c8993c3c45dcaab25b42142d5671 (127.0.99.129:36983), 89dd09f8aaee4b249ae3ce924a422a67 (127.0.99.130:38889), 5b8291b26a7f448b8636187d1caf86d0 (127.0.99.131:44479), c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134:44343), d43ed6b8663947d99a72cd922cc323aa (127.0.99.132:40023), 002cc3411acb4fdfa0b67c6f57f687a6 (127.0.99.135:38989)
I20240503 02:00:23.262895 17224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81b54f3bcf34485eb11b1bc954a2bd8f" candidate_uuid: "89dd09f8aaee4b249ae3ce924a422a67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b8291b26a7f448b8636187d1caf86d0" is_pre_election: true
I20240503 02:00:23.263285 17224 raft_consensus.cc:2454] T 81b54f3bcf34485eb11b1bc954a2bd8f P 5b8291b26a7f448b8636187d1caf86d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89dd09f8aaee4b249ae3ce924a422a67 in term 0.
I20240503 02:00:23.265153 16960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81b54f3bcf34485eb11b1bc954a2bd8f" candidate_uuid: "89dd09f8aaee4b249ae3ce924a422a67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6398c8993c3c45dcaab25b42142d5671" is_pre_election: true
I20240503 02:00:23.265502 16960 raft_consensus.cc:2454] T 81b54f3bcf34485eb11b1bc954a2bd8f P 6398c8993c3c45dcaab25b42142d5671 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89dd09f8aaee4b249ae3ce924a422a67 in term 0.
I20240503 02:00:23.266379 17488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81b54f3bcf34485eb11b1bc954a2bd8f" candidate_uuid: "89dd09f8aaee4b249ae3ce924a422a67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16d437fe629643948c95d628926d5ffa" is_pre_election: true
I20240503 02:00:23.266608 17488 raft_consensus.cc:2454] T 81b54f3bcf34485eb11b1bc954a2bd8f P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89dd09f8aaee4b249ae3ce924a422a67 in term 0.
I20240503 02:00:23.267107 17025 leader_election.cc:302] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 16d437fe629643948c95d628926d5ffa, 5b8291b26a7f448b8636187d1caf86d0, 6398c8993c3c45dcaab25b42142d5671, 89dd09f8aaee4b249ae3ce924a422a67; no voters:
I20240503 02:00:23.267524 18243 raft_consensus.cc:2781] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:23.267628 18243 raft_consensus.cc:482] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:23.267688 18243 raft_consensus.cc:3037] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:23.268687 17752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81b54f3bcf34485eb11b1bc954a2bd8f" candidate_uuid: "89dd09f8aaee4b249ae3ce924a422a67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" is_pre_election: true
I20240503 02:00:23.268968 17752 raft_consensus.cc:2454] T 81b54f3bcf34485eb11b1bc954a2bd8f P 002cc3411acb4fdfa0b67c6f57f687a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89dd09f8aaee4b249ae3ce924a422a67 in term 0.
I20240503 02:00:23.269121 18243 raft_consensus.cc:504] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:23.269781 18243 leader_election.cc:288] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 [CANDIDATE]: Term 1 election: Requested vote from peers 6398c8993c3c45dcaab25b42142d5671 (127.0.99.129:36983), 5b8291b26a7f448b8636187d1caf86d0 (127.0.99.131:44479), c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134:44343), d43ed6b8663947d99a72cd922cc323aa (127.0.99.132:40023), 16d437fe629643948c95d628926d5ffa (127.0.99.133:46581), 002cc3411acb4fdfa0b67c6f57f687a6 (127.0.99.135:38989)
I20240503 02:00:23.269958 16960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81b54f3bcf34485eb11b1bc954a2bd8f" candidate_uuid: "89dd09f8aaee4b249ae3ce924a422a67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6398c8993c3c45dcaab25b42142d5671"
I20240503 02:00:23.270100 16960 raft_consensus.cc:3037] T 81b54f3bcf34485eb11b1bc954a2bd8f P 6398c8993c3c45dcaab25b42142d5671 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:23.270169 17356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81b54f3bcf34485eb11b1bc954a2bd8f" candidate_uuid: "89dd09f8aaee4b249ae3ce924a422a67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d43ed6b8663947d99a72cd922cc323aa" is_pre_election: true
I20240503 02:00:23.270424 17355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81b54f3bcf34485eb11b1bc954a2bd8f" candidate_uuid: "89dd09f8aaee4b249ae3ce924a422a67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d43ed6b8663947d99a72cd922cc323aa"
I20240503 02:00:23.270529 17356 raft_consensus.cc:2454] T 81b54f3bcf34485eb11b1bc954a2bd8f P d43ed6b8663947d99a72cd922cc323aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89dd09f8aaee4b249ae3ce924a422a67 in term 0.
I20240503 02:00:23.271477 17224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81b54f3bcf34485eb11b1bc954a2bd8f" candidate_uuid: "89dd09f8aaee4b249ae3ce924a422a67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b8291b26a7f448b8636187d1caf86d0"
I20240503 02:00:23.271620 17224 raft_consensus.cc:3037] T 81b54f3bcf34485eb11b1bc954a2bd8f P 5b8291b26a7f448b8636187d1caf86d0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:23.271883 16960 raft_consensus.cc:2454] T 81b54f3bcf34485eb11b1bc954a2bd8f P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89dd09f8aaee4b249ae3ce924a422a67 in term 1.
I20240503 02:00:23.272728 17488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81b54f3bcf34485eb11b1bc954a2bd8f" candidate_uuid: "89dd09f8aaee4b249ae3ce924a422a67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16d437fe629643948c95d628926d5ffa"
I20240503 02:00:23.272897 17488 raft_consensus.cc:3037] T 81b54f3bcf34485eb11b1bc954a2bd8f P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:23.272994 17224 raft_consensus.cc:2454] T 81b54f3bcf34485eb11b1bc954a2bd8f P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89dd09f8aaee4b249ae3ce924a422a67 in term 1.
I20240503 02:00:23.273116 17752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81b54f3bcf34485eb11b1bc954a2bd8f" candidate_uuid: "89dd09f8aaee4b249ae3ce924a422a67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "002cc3411acb4fdfa0b67c6f57f687a6"
I20240503 02:00:23.273249 17752 raft_consensus.cc:3037] T 81b54f3bcf34485eb11b1bc954a2bd8f P 002cc3411acb4fdfa0b67c6f57f687a6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:23.273795 17223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbec07350ad04c49afd42e5d88dda6ad" candidate_uuid: "16d437fe629643948c95d628926d5ffa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b8291b26a7f448b8636187d1caf86d0" is_pre_election: true
I20240503 02:00:23.273926 17223 raft_consensus.cc:2454] T fbec07350ad04c49afd42e5d88dda6ad P 5b8291b26a7f448b8636187d1caf86d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16d437fe629643948c95d628926d5ffa in term 0.
I20240503 02:00:23.274135 17488 raft_consensus.cc:2454] T 81b54f3bcf34485eb11b1bc954a2bd8f P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89dd09f8aaee4b249ae3ce924a422a67 in term 1.
I20240503 02:00:23.274480 17752 raft_consensus.cc:2454] T 81b54f3bcf34485eb11b1bc954a2bd8f P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89dd09f8aaee4b249ae3ce924a422a67 in term 1.
I20240503 02:00:23.274541 17224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acfd773c1f384c219d776878407d1144" candidate_uuid: "16d437fe629643948c95d628926d5ffa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b8291b26a7f448b8636187d1caf86d0" is_pre_election: true
I20240503 02:00:23.274654 17224 raft_consensus.cc:2454] T acfd773c1f384c219d776878407d1144 P 5b8291b26a7f448b8636187d1caf86d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16d437fe629643948c95d628926d5ffa in term 0.
I20240503 02:00:23.274799 16959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbec07350ad04c49afd42e5d88dda6ad" candidate_uuid: "16d437fe629643948c95d628926d5ffa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6398c8993c3c45dcaab25b42142d5671" is_pre_election: true
I20240503 02:00:23.274931 16959 raft_consensus.cc:2454] T fbec07350ad04c49afd42e5d88dda6ad P 6398c8993c3c45dcaab25b42142d5671 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16d437fe629643948c95d628926d5ffa in term 0.
I20240503 02:00:23.275022 17025 leader_election.cc:302] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 5 responses out of 7 voters: 4 yes votes; 1 no votes. yes voters: 002cc3411acb4fdfa0b67c6f57f687a6, 5b8291b26a7f448b8636187d1caf86d0, 6398c8993c3c45dcaab25b42142d5671, 89dd09f8aaee4b249ae3ce924a422a67; no voters: d43ed6b8663947d99a72cd922cc323aa
I20240503 02:00:23.275223 18243 raft_consensus.cc:2781] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:23.275319 18243 raft_consensus.cc:686] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 LEADER]: Becoming Leader. State: Replica: 89dd09f8aaee4b249ae3ce924a422a67, State: Running, Role: LEADER
I20240503 02:00:23.275367 16960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acfd773c1f384c219d776878407d1144" candidate_uuid: "16d437fe629643948c95d628926d5ffa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6398c8993c3c45dcaab25b42142d5671" is_pre_election: true
I20240503 02:00:23.275485 16960 raft_consensus.cc:2454] T acfd773c1f384c219d776878407d1144 P 6398c8993c3c45dcaab25b42142d5671 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16d437fe629643948c95d628926d5ffa in term 0.
I20240503 02:00:23.275733 18243 consensus_queue.cc:237] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:23.276504 17355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acfd773c1f384c219d776878407d1144" candidate_uuid: "16d437fe629643948c95d628926d5ffa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d43ed6b8663947d99a72cd922cc323aa" is_pre_election: true
I20240503 02:00:23.276690 17355 raft_consensus.cc:2454] T acfd773c1f384c219d776878407d1144 P d43ed6b8663947d99a72cd922cc323aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16d437fe629643948c95d628926d5ffa in term 0.
I20240503 02:00:23.277024 17422 leader_election.cc:302] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 16d437fe629643948c95d628926d5ffa, 5b8291b26a7f448b8636187d1caf86d0, 6398c8993c3c45dcaab25b42142d5671, d43ed6b8663947d99a72cd922cc323aa; no voters:
I20240503 02:00:23.277323 18260 raft_consensus.cc:2781] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:23.277374 18260 raft_consensus.cc:482] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:23.277408 18260 raft_consensus.cc:3037] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:23.278456 17356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbec07350ad04c49afd42e5d88dda6ad" candidate_uuid: "16d437fe629643948c95d628926d5ffa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d43ed6b8663947d99a72cd922cc323aa" is_pre_election: true
I20240503 02:00:23.278457 18260 raft_consensus.cc:504] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:23.278594 17356 raft_consensus.cc:2454] T fbec07350ad04c49afd42e5d88dda6ad P d43ed6b8663947d99a72cd922cc323aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16d437fe629643948c95d628926d5ffa in term 0.
I20240503 02:00:23.278623 17620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acfd773c1f384c219d776878407d1144" candidate_uuid: "16d437fe629643948c95d628926d5ffa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c843193b801b4f1a88f630ed3ed0f375" is_pre_election: true
I20240503 02:00:23.278874 17619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbec07350ad04c49afd42e5d88dda6ad" candidate_uuid: "16d437fe629643948c95d628926d5ffa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c843193b801b4f1a88f630ed3ed0f375" is_pre_election: true
I20240503 02:00:23.278914 17620 raft_consensus.cc:2454] T acfd773c1f384c219d776878407d1144 P c843193b801b4f1a88f630ed3ed0f375 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16d437fe629643948c95d628926d5ffa in term 0.
I20240503 02:00:23.278986 17619 raft_consensus.cc:2454] T fbec07350ad04c49afd42e5d88dda6ad P c843193b801b4f1a88f630ed3ed0f375 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16d437fe629643948c95d628926d5ffa in term 0.
I20240503 02:00:23.279103 18260 leader_election.cc:288] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa [CANDIDATE]: Term 1 election: Requested vote from peers 6398c8993c3c45dcaab25b42142d5671 (127.0.99.129:36983), 89dd09f8aaee4b249ae3ce924a422a67 (127.0.99.130:38889), 5b8291b26a7f448b8636187d1caf86d0 (127.0.99.131:44479), d43ed6b8663947d99a72cd922cc323aa (127.0.99.132:40023), c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134:44343), 002cc3411acb4fdfa0b67c6f57f687a6 (127.0.99.135:38989)
I20240503 02:00:23.279662 16960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acfd773c1f384c219d776878407d1144" candidate_uuid: "16d437fe629643948c95d628926d5ffa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6398c8993c3c45dcaab25b42142d5671"
I20240503 02:00:23.279788 16960 raft_consensus.cc:3037] T acfd773c1f384c219d776878407d1144 P 6398c8993c3c45dcaab25b42142d5671 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:23.279778 17421 leader_election.cc:302] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 16d437fe629643948c95d628926d5ffa, 5b8291b26a7f448b8636187d1caf86d0, 6398c8993c3c45dcaab25b42142d5671, c843193b801b4f1a88f630ed3ed0f375; no voters:
I20240503 02:00:23.279904 17224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acfd773c1f384c219d776878407d1144" candidate_uuid: "16d437fe629643948c95d628926d5ffa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b8291b26a7f448b8636187d1caf86d0"
I20240503 02:00:23.280017 17224 raft_consensus.cc:3037] T acfd773c1f384c219d776878407d1144 P 5b8291b26a7f448b8636187d1caf86d0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:23.280117 17620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acfd773c1f384c219d776878407d1144" candidate_uuid: "16d437fe629643948c95d628926d5ffa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c843193b801b4f1a88f630ed3ed0f375"
I20240503 02:00:23.280193 17356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acfd773c1f384c219d776878407d1144" candidate_uuid: "16d437fe629643948c95d628926d5ffa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d43ed6b8663947d99a72cd922cc323aa"
I20240503 02:00:23.280234 17620 raft_consensus.cc:3037] T acfd773c1f384c219d776878407d1144 P c843193b801b4f1a88f630ed3ed0f375 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:23.280306 17356 raft_consensus.cc:3037] T acfd773c1f384c219d776878407d1144 P d43ed6b8663947d99a72cd922cc323aa [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:23.280733 18260 raft_consensus.cc:2781] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:23.280793 18260 raft_consensus.cc:482] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:23.280833 18260 raft_consensus.cc:3037] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:23.280819 16960 raft_consensus.cc:2454] T acfd773c1f384c219d776878407d1144 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16d437fe629643948c95d628926d5ffa in term 1.
I20240503 02:00:23.281517 17620 raft_consensus.cc:2454] T acfd773c1f384c219d776878407d1144 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16d437fe629643948c95d628926d5ffa in term 1.
I20240503 02:00:23.281562 17356 raft_consensus.cc:2454] T acfd773c1f384c219d776878407d1144 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16d437fe629643948c95d628926d5ffa in term 1.
I20240503 02:00:23.281769 18260 raft_consensus.cc:504] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } }
I20240503 02:00:23.281955 17421 leader_election.cc:302] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 16d437fe629643948c95d628926d5ffa, 6398c8993c3c45dcaab25b42142d5671, c843193b801b4f1a88f630ed3ed0f375, d43ed6b8663947d99a72cd922cc323aa; no voters:
I20240503 02:00:23.282354 17620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81b54f3bcf34485eb11b1bc954a2bd8f" candidate_uuid: "89dd09f8aaee4b249ae3ce924a422a67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c843193b801b4f1a88f630ed3ed0f375" is_pre_election: true
I20240503 02:00:23.282394 17619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81b54f3bcf34485eb11b1bc954a2bd8f" candidate_uuid: "89dd09f8aaee4b249ae3ce924a422a67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c843193b801b4f1a88f630ed3ed0f375"
I20240503 02:00:23.282433 18260 leader_election.cc:288] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa [CANDIDATE]: Term 1 election: Requested vote from peers 6398c8993c3c45dcaab25b42142d5671 (127.0.99.129:36983), 89dd09f8aaee4b249ae3ce924a422a67 (127.0.99.130:38889), 5b8291b26a7f448b8636187d1caf86d0 (127.0.99.131:44479), c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134:44343), d43ed6b8663947d99a72cd922cc323aa (127.0.99.132:40023), 002cc3411acb4fdfa0b67c6f57f687a6 (127.0.99.135:38989)
I20240503 02:00:23.282480 17620 raft_consensus.cc:2454] T 81b54f3bcf34485eb11b1bc954a2bd8f P c843193b801b4f1a88f630ed3ed0f375 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89dd09f8aaee4b249ae3ce924a422a67 in term 0.
I20240503 02:00:23.282541 18260 raft_consensus.cc:2781] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:23.282615 18260 raft_consensus.cc:686] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa [term 1 LEADER]: Becoming Leader. State: Replica: 16d437fe629643948c95d628926d5ffa, State: Running, Role: LEADER
I20240503 02:00:23.283398 18260 consensus_queue.cc:237] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:23.283586 17619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbec07350ad04c49afd42e5d88dda6ad" candidate_uuid: "16d437fe629643948c95d628926d5ffa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c843193b801b4f1a88f630ed3ed0f375"
I20240503 02:00:23.283707 17619 raft_consensus.cc:3037] T fbec07350ad04c49afd42e5d88dda6ad P c843193b801b4f1a88f630ed3ed0f375 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:23.284768 17619 raft_consensus.cc:2454] T fbec07350ad04c49afd42e5d88dda6ad P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16d437fe629643948c95d628926d5ffa in term 1.
I20240503 02:00:23.284904 16816 catalog_manager.cc:5484] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 reported cstate change: term changed from 0 to 1, leader changed from <none> to 89dd09f8aaee4b249ae3ce924a422a67 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "89dd09f8aaee4b249ae3ce924a422a67" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:23.285940 17092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acfd773c1f384c219d776878407d1144" candidate_uuid: "16d437fe629643948c95d628926d5ffa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89dd09f8aaee4b249ae3ce924a422a67" is_pre_election: true
I20240503 02:00:23.286144 17092 raft_consensus.cc:2454] T acfd773c1f384c219d776878407d1144 P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16d437fe629643948c95d628926d5ffa in term 0.
I20240503 02:00:23.286413 17091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbec07350ad04c49afd42e5d88dda6ad" candidate_uuid: "16d437fe629643948c95d628926d5ffa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89dd09f8aaee4b249ae3ce924a422a67" is_pre_election: true
I20240503 02:00:23.286516 17091 raft_consensus.cc:2454] T fbec07350ad04c49afd42e5d88dda6ad P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16d437fe629643948c95d628926d5ffa in term 0.
I20240503 02:00:23.286585 17356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbec07350ad04c49afd42e5d88dda6ad" candidate_uuid: "16d437fe629643948c95d628926d5ffa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d43ed6b8663947d99a72cd922cc323aa"
I20240503 02:00:23.286695 17356 raft_consensus.cc:3037] T fbec07350ad04c49afd42e5d88dda6ad P d43ed6b8663947d99a72cd922cc323aa [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:23.286695 17090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acfd773c1f384c219d776878407d1144" candidate_uuid: "16d437fe629643948c95d628926d5ffa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89dd09f8aaee4b249ae3ce924a422a67"
I20240503 02:00:23.286810 17090 raft_consensus.cc:3037] T acfd773c1f384c219d776878407d1144 P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:23.286949 17224 raft_consensus.cc:2454] T acfd773c1f384c219d776878407d1144 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16d437fe629643948c95d628926d5ffa in term 1.
I20240503 02:00:23.287735 17356 raft_consensus.cc:2454] T fbec07350ad04c49afd42e5d88dda6ad P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16d437fe629643948c95d628926d5ffa in term 1.
W20240503 02:00:23.287797 17535 tablet.cc:2369] T 81b54f3bcf34485eb11b1bc954a2bd8f P 16d437fe629643948c95d628926d5ffa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:00:23.287983 17090 raft_consensus.cc:2454] T acfd773c1f384c219d776878407d1144 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16d437fe629643948c95d628926d5ffa in term 1.
I20240503 02:00:23.288187 17089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbec07350ad04c49afd42e5d88dda6ad" candidate_uuid: "16d437fe629643948c95d628926d5ffa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89dd09f8aaee4b249ae3ce924a422a67"
I20240503 02:00:23.288292 17089 raft_consensus.cc:3037] T fbec07350ad04c49afd42e5d88dda6ad P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:23.288261 16960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbec07350ad04c49afd42e5d88dda6ad" candidate_uuid: "16d437fe629643948c95d628926d5ffa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6398c8993c3c45dcaab25b42142d5671"
I20240503 02:00:23.288380 16960 raft_consensus.cc:3037] T fbec07350ad04c49afd42e5d88dda6ad P 6398c8993c3c45dcaab25b42142d5671 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:23.289216 17752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acfd773c1f384c219d776878407d1144" candidate_uuid: "16d437fe629643948c95d628926d5ffa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" is_pre_election: true
I20240503 02:00:23.289381 17752 raft_consensus.cc:2454] T acfd773c1f384c219d776878407d1144 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16d437fe629643948c95d628926d5ffa in term 0.
I20240503 02:00:23.289395 16960 raft_consensus.cc:2454] T fbec07350ad04c49afd42e5d88dda6ad P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16d437fe629643948c95d628926d5ffa in term 1.
I20240503 02:00:23.289646 17751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbec07350ad04c49afd42e5d88dda6ad" candidate_uuid: "16d437fe629643948c95d628926d5ffa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" is_pre_election: true
I20240503 02:00:23.289690 17423 leader_election.cc:302] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 16d437fe629643948c95d628926d5ffa, 6398c8993c3c45dcaab25b42142d5671, c843193b801b4f1a88f630ed3ed0f375, d43ed6b8663947d99a72cd922cc323aa; no voters:
I20240503 02:00:23.289816 17751 raft_consensus.cc:2454] T fbec07350ad04c49afd42e5d88dda6ad P 002cc3411acb4fdfa0b67c6f57f687a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16d437fe629643948c95d628926d5ffa in term 0.
I20240503 02:00:23.289898 18237 raft_consensus.cc:2781] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:23.290048 17750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acfd773c1f384c219d776878407d1144" candidate_uuid: "16d437fe629643948c95d628926d5ffa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "002cc3411acb4fdfa0b67c6f57f687a6"
I20240503 02:00:23.290125 18237 raft_consensus.cc:686] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa [term 1 LEADER]: Becoming Leader. State: Replica: 16d437fe629643948c95d628926d5ffa, State: Running, Role: LEADER
I20240503 02:00:23.290150 17750 raft_consensus.cc:3037] T acfd773c1f384c219d776878407d1144 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:23.290282 18237 consensus_queue.cc:237] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } }
I20240503 02:00:23.291280 17089 raft_consensus.cc:2454] T fbec07350ad04c49afd42e5d88dda6ad P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16d437fe629643948c95d628926d5ffa in term 1.
I20240503 02:00:23.291327 17750 raft_consensus.cc:2454] T acfd773c1f384c219d776878407d1144 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16d437fe629643948c95d628926d5ffa in term 1.
I20240503 02:00:23.291582 17224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbec07350ad04c49afd42e5d88dda6ad" candidate_uuid: "16d437fe629643948c95d628926d5ffa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b8291b26a7f448b8636187d1caf86d0"
I20240503 02:00:23.291740 17224 raft_consensus.cc:3037] T fbec07350ad04c49afd42e5d88dda6ad P 5b8291b26a7f448b8636187d1caf86d0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:23.292898 17224 raft_consensus.cc:2454] T fbec07350ad04c49afd42e5d88dda6ad P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16d437fe629643948c95d628926d5ffa in term 1.
W20240503 02:00:23.293406 17007 tablet.cc:2369] T 81b54f3bcf34485eb11b1bc954a2bd8f P 6398c8993c3c45dcaab25b42142d5671: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:00:23.289393 17749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbec07350ad04c49afd42e5d88dda6ad" candidate_uuid: "16d437fe629643948c95d628926d5ffa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "002cc3411acb4fdfa0b67c6f57f687a6"
I20240503 02:00:23.293586 17749 raft_consensus.cc:3037] T fbec07350ad04c49afd42e5d88dda6ad P 002cc3411acb4fdfa0b67c6f57f687a6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:23.294162 16816 catalog_manager.cc:5484] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa reported cstate change: term changed from 0 to 1, leader changed from <none> to 16d437fe629643948c95d628926d5ffa (127.0.99.133). New cstate: current_term: 1 leader_uuid: "16d437fe629643948c95d628926d5ffa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:23.294585 17749 raft_consensus.cc:2454] T fbec07350ad04c49afd42e5d88dda6ad P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16d437fe629643948c95d628926d5ffa in term 1.
I20240503 02:00:23.294579 16816 catalog_manager.cc:5484] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa reported cstate change: term changed from 0 to 1, leader changed from <none> to 16d437fe629643948c95d628926d5ffa (127.0.99.133). New cstate: current_term: 1 leader_uuid: "16d437fe629643948c95d628926d5ffa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } health_report { overall_health: HEALTHY } } }
W20240503 02:00:23.360814 17403 tablet.cc:2369] T 81b54f3bcf34485eb11b1bc954a2bd8f P d43ed6b8663947d99a72cd922cc323aa: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 02:00:23.431876 17799 tablet.cc:2369] T 81b54f3bcf34485eb11b1bc954a2bd8f P 002cc3411acb4fdfa0b67c6f57f687a6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 02:00:23.434619 17271 tablet.cc:2369] T 81b54f3bcf34485eb11b1bc954a2bd8f P 5b8291b26a7f448b8636187d1caf86d0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 02:00:23.437013 18279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:00:23.437248 18279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:00:23.440168 18279 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
W20240503 02:00:23.477861 17667 tablet.cc:2369] T 81b54f3bcf34485eb11b1bc954a2bd8f P c843193b801b4f1a88f630ed3ed0f375: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:00:23.532708 17749 raft_consensus.cc:1261] T acfd773c1f384c219d776878407d1144 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:23.533857 17089 raft_consensus.cc:1261] T fbec07350ad04c49afd42e5d88dda6ad P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:23.533991 16960 raft_consensus.cc:1261] T fbec07350ad04c49afd42e5d88dda6ad P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:23.534273 17356 raft_consensus.cc:1261] T fbec07350ad04c49afd42e5d88dda6ad P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:23.534525 17488 raft_consensus.cc:1261] T 81b54f3bcf34485eb11b1bc954a2bd8f P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:23.534822 17356 raft_consensus.cc:3037] T 81b54f3bcf34485eb11b1bc954a2bd8f P d43ed6b8663947d99a72cd922cc323aa [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:23.535470 17224 raft_consensus.cc:1261] T fbec07350ad04c49afd42e5d88dda6ad P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:23.535938 17356 raft_consensus.cc:1261] T 81b54f3bcf34485eb11b1bc954a2bd8f P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:23.536335 18243 consensus_queue.cc:1035] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:23.537220 17749 raft_consensus.cc:1261] T 81b54f3bcf34485eb11b1bc954a2bd8f P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:23.537547 17619 raft_consensus.cc:3037] T 81b54f3bcf34485eb11b1bc954a2bd8f P c843193b801b4f1a88f630ed3ed0f375 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:23.537861 18276 consensus_queue.cc:1035] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:23.538048 18260 consensus_queue.cc:1035] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:23.538362 18260 consensus_queue.cc:1035] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:23.538973 17620 raft_consensus.cc:1261] T fbec07350ad04c49afd42e5d88dda6ad P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:23.539433 18237 consensus_queue.cc:1035] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:23.539466 17749 raft_consensus.cc:1261] T fbec07350ad04c49afd42e5d88dda6ad P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:23.539772 18237 consensus_queue.cc:1035] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:23.544023 17619 raft_consensus.cc:1261] T 81b54f3bcf34485eb11b1bc954a2bd8f P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:23.546329 18260 consensus_queue.cc:1035] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:23.546681 18260 consensus_queue.cc:1035] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:23.547593 17223 raft_consensus.cc:1261] T acfd773c1f384c219d776878407d1144 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:23.547967 18237 consensus_queue.cc:1035] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:23.549401 17620 raft_consensus.cc:1261] T acfd773c1f384c219d776878407d1144 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:23.550619 16959 raft_consensus.cc:1261] T acfd773c1f384c219d776878407d1144 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:23.551002 18237 consensus_queue.cc:1035] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:23.551285 18237 consensus_queue.cc:1035] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:23.553699 17354 raft_consensus.cc:1261] T acfd773c1f384c219d776878407d1144 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:23.554157 18276 consensus_queue.cc:1035] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:23.564924 18243 consensus_queue.cc:1035] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:23.565819 18247 consensus_queue.cc:1035] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 02:00:23.566421 18247 consensus_queue.cc:1035] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:23.566109 17090 raft_consensus.cc:1261] T acfd773c1f384c219d776878407d1144 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:23.567070 18260 consensus_queue.cc:1035] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:23.569325 18305 mvcc.cc:205] Tried to move back new op lower bound from 7023417850007023616 to 7023417848999854080. Current Snapshot: MvccSnapshot[applied={T|T < 7023417849986453504}]
I20240503 02:00:23.581245 18319 mvcc.cc:205] Tried to move back new op lower bound from 7023417850007023616 to 7023417848999854080. Current Snapshot: MvccSnapshot[applied={T|T < 7023417849986453504}]
I20240503 02:00:23.581861 18312 mvcc.cc:205] Tried to move back new op lower bound from 7023417850007023616 to 7023417848999854080. Current Snapshot: MvccSnapshot[applied={T|T < 7023417849986453504}]
I20240503 02:00:23.582363 16960 raft_consensus.cc:1261] T 81b54f3bcf34485eb11b1bc954a2bd8f P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:23.582974 18247 consensus_queue.cc:1035] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:23.583874 17224 raft_consensus.cc:1261] T 81b54f3bcf34485eb11b1bc954a2bd8f P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:23.584277 18247 consensus_queue.cc:1035] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:23.585124 18309 mvcc.cc:205] Tried to move back new op lower bound from 7023417850007023616 to 7023417848999854080. Current Snapshot: MvccSnapshot[applied={T|T < 7023417849986453504}]
I20240503 02:00:23.604415 18329 mvcc.cc:205] Tried to move back new op lower bound from 7023417850007023616 to 7023417848999854080. Current Snapshot: MvccSnapshot[applied={T|T < 7023417849986453504}]
I20240503 02:00:23.621035 18325 mvcc.cc:205] Tried to move back new op lower bound from 7023417849983528960 to 7023417848942923776. Current Snapshot: MvccSnapshot[applied={T|T < 7023417849983528960}]
I20240503 02:00:23.656934 18311 mvcc.cc:205] Tried to move back new op lower bound from 7023417850007023616 to 7023417848999854080. Current Snapshot: MvccSnapshot[applied={T|T < 7023417850007023616 or (T in {7023417850007023616})}]
INSERT report
rows total: 2000
time total: 228.265 ms
time per row: 0.114133 ms
I20240503 02:00:23.713737 16818 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39916:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 7
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240503 02:00:23.714335 16818 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_1 in case of a server failure: 8 tablet servers would be needed, 7 are available. Consider bringing up more tablet servers.
I20240503 02:00:23.720237 17072 tablet_service.cc:1467] Processing CreateTablet for tablet 885fe0c3a5e04f408e5b02c3c133248a (DEFAULT_TABLE table=rebalance_test_table_1 [id=38eeb06923e14f2ead405ffb039fd231]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:23.720299 16939 tablet_service.cc:1467] Processing CreateTablet for tablet 885fe0c3a5e04f408e5b02c3c133248a (DEFAULT_TABLE table=rebalance_test_table_1 [id=38eeb06923e14f2ead405ffb039fd231]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:23.720587 17072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 885fe0c3a5e04f408e5b02c3c133248a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:23.720587 16939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 885fe0c3a5e04f408e5b02c3c133248a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:23.720818 17468 tablet_service.cc:1467] Processing CreateTablet for tablet 885fe0c3a5e04f408e5b02c3c133248a (DEFAULT_TABLE table=rebalance_test_table_1 [id=38eeb06923e14f2ead405ffb039fd231]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:23.721139 17468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 885fe0c3a5e04f408e5b02c3c133248a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:23.723269 18224 tablet_bootstrap.cc:492] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671: Bootstrap starting.
I20240503 02:00:23.723542 17598 tablet_service.cc:1467] Processing CreateTablet for tablet 885fe0c3a5e04f408e5b02c3c133248a (DEFAULT_TABLE table=rebalance_test_table_1 [id=38eeb06923e14f2ead405ffb039fd231]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:23.723618 17467 tablet_service.cc:1467] Processing CreateTablet for tablet 5833b4f9a5434734817f154d660aa9f7 (DEFAULT_TABLE table=rebalance_test_table_1 [id=38eeb06923e14f2ead405ffb039fd231]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:23.723843 17598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 885fe0c3a5e04f408e5b02c3c133248a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:23.723855 17467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5833b4f9a5434734817f154d660aa9f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:23.723906 17466 tablet_service.cc:1467] Processing CreateTablet for tablet 30abfada8ab44449b9a4237c57093b73 (DEFAULT_TABLE table=rebalance_test_table_1 [id=38eeb06923e14f2ead405ffb039fd231]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:23.724112 17466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30abfada8ab44449b9a4237c57093b73. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:23.724180 18224 tablet_bootstrap.cc:654] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:23.724978 17731 tablet_service.cc:1467] Processing CreateTablet for tablet 885fe0c3a5e04f408e5b02c3c133248a (DEFAULT_TABLE table=rebalance_test_table_1 [id=38eeb06923e14f2ead405ffb039fd231]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:23.725260 17731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 885fe0c3a5e04f408e5b02c3c133248a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:23.726140 18225 tablet_bootstrap.cc:492] T 885fe0c3a5e04f408e5b02c3c133248a P 89dd09f8aaee4b249ae3ce924a422a67: Bootstrap starting.
I20240503 02:00:23.726512 18222 tablet_bootstrap.cc:492] T 885fe0c3a5e04f408e5b02c3c133248a P c843193b801b4f1a88f630ed3ed0f375: Bootstrap starting.
I20240503 02:00:23.727082 18225 tablet_bootstrap.cc:654] T 885fe0c3a5e04f408e5b02c3c133248a P 89dd09f8aaee4b249ae3ce924a422a67: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:23.727312 18222 tablet_bootstrap.cc:654] T 885fe0c3a5e04f408e5b02c3c133248a P c843193b801b4f1a88f630ed3ed0f375: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:23.727669 18223 tablet_bootstrap.cc:492] T 885fe0c3a5e04f408e5b02c3c133248a P 16d437fe629643948c95d628926d5ffa: Bootstrap starting.
I20240503 02:00:23.727969 17732 tablet_service.cc:1467] Processing CreateTablet for tablet 30abfada8ab44449b9a4237c57093b73 (DEFAULT_TABLE table=rebalance_test_table_1 [id=38eeb06923e14f2ead405ffb039fd231]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:23.728120 18225 tablet_bootstrap.cc:492] T 885fe0c3a5e04f408e5b02c3c133248a P 89dd09f8aaee4b249ae3ce924a422a67: No bootstrap required, opened a new log
I20240503 02:00:23.728157 17732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30abfada8ab44449b9a4237c57093b73. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:23.728199 18225 ts_tablet_manager.cc:1367] T 885fe0c3a5e04f408e5b02c3c133248a P 89dd09f8aaee4b249ae3ce924a422a67: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 02:00:23.728516 18223 tablet_bootstrap.cc:654] T 885fe0c3a5e04f408e5b02c3c133248a P 16d437fe629643948c95d628926d5ffa: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:23.728812 18225 raft_consensus.cc:348] T 885fe0c3a5e04f408e5b02c3c133248a P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:23.728937 18225 raft_consensus.cc:374] T 885fe0c3a5e04f408e5b02c3c133248a P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:23.728992 18225 raft_consensus.cc:729] T 885fe0c3a5e04f408e5b02c3c133248a P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89dd09f8aaee4b249ae3ce924a422a67, State: Initialized, Role: FOLLOWER
I20240503 02:00:23.729121 18225 consensus_queue.cc:260] T 885fe0c3a5e04f408e5b02c3c133248a P 89dd09f8aaee4b249ae3ce924a422a67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:23.729343 17600 tablet_service.cc:1467] Processing CreateTablet for tablet 30abfada8ab44449b9a4237c57093b73 (DEFAULT_TABLE table=rebalance_test_table_1 [id=38eeb06923e14f2ead405ffb039fd231]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:23.729416 18225 ts_tablet_manager.cc:1398] T 885fe0c3a5e04f408e5b02c3c133248a P 89dd09f8aaee4b249ae3ce924a422a67: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:23.729591 17600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30abfada8ab44449b9a4237c57093b73. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:23.730028 17730 tablet_service.cc:1467] Processing CreateTablet for tablet 5833b4f9a5434734817f154d660aa9f7 (DEFAULT_TABLE table=rebalance_test_table_1 [id=38eeb06923e14f2ead405ffb039fd231]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:23.730260 17730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5833b4f9a5434734817f154d660aa9f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:23.732090 17334 tablet_service.cc:1467] Processing CreateTablet for tablet 885fe0c3a5e04f408e5b02c3c133248a (DEFAULT_TABLE table=rebalance_test_table_1 [id=38eeb06923e14f2ead405ffb039fd231]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:23.732172 17599 tablet_service.cc:1467] Processing CreateTablet for tablet 5833b4f9a5434734817f154d660aa9f7 (DEFAULT_TABLE table=rebalance_test_table_1 [id=38eeb06923e14f2ead405ffb039fd231]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:23.732216 17335 tablet_service.cc:1467] Processing CreateTablet for tablet 5833b4f9a5434734817f154d660aa9f7 (DEFAULT_TABLE table=rebalance_test_table_1 [id=38eeb06923e14f2ead405ffb039fd231]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:23.732378 17599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5833b4f9a5434734817f154d660aa9f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:23.732378 17334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 885fe0c3a5e04f408e5b02c3c133248a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:23.732507 16939 tablet_service.cc:1467] Processing CreateTablet for tablet 5833b4f9a5434734817f154d660aa9f7 (DEFAULT_TABLE table=rebalance_test_table_1 [id=38eeb06923e14f2ead405ffb039fd231]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:23.732693 17336 tablet_service.cc:1467] Processing CreateTablet for tablet 30abfada8ab44449b9a4237c57093b73 (DEFAULT_TABLE table=rebalance_test_table_1 [id=38eeb06923e14f2ead405ffb039fd231]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:23.732898 17336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30abfada8ab44449b9a4237c57093b73. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:23.732898 16939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5833b4f9a5434734817f154d660aa9f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:23.734493 16938 tablet_service.cc:1467] Processing CreateTablet for tablet 30abfada8ab44449b9a4237c57093b73 (DEFAULT_TABLE table=rebalance_test_table_1 [id=38eeb06923e14f2ead405ffb039fd231]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:23.734757 16938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30abfada8ab44449b9a4237c57093b73. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:23.735097 17335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5833b4f9a5434734817f154d660aa9f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:23.735163 18373 tablet_bootstrap.cc:492] T 885fe0c3a5e04f408e5b02c3c133248a P d43ed6b8663947d99a72cd922cc323aa: Bootstrap starting.
I20240503 02:00:23.736083 18373 tablet_bootstrap.cc:654] T 885fe0c3a5e04f408e5b02c3c133248a P d43ed6b8663947d99a72cd922cc323aa: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:23.736227 18369 tablet_bootstrap.cc:492] T 885fe0c3a5e04f408e5b02c3c133248a P 002cc3411acb4fdfa0b67c6f57f687a6: Bootstrap starting.
I20240503 02:00:23.737246 18223 tablet_bootstrap.cc:492] T 885fe0c3a5e04f408e5b02c3c133248a P 16d437fe629643948c95d628926d5ffa: No bootstrap required, opened a new log
I20240503 02:00:23.737331 18223 ts_tablet_manager.cc:1367] T 885fe0c3a5e04f408e5b02c3c133248a P 16d437fe629643948c95d628926d5ffa: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.000s
I20240503 02:00:23.737795 18223 raft_consensus.cc:348] T 885fe0c3a5e04f408e5b02c3c133248a P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:23.737941 18223 raft_consensus.cc:374] T 885fe0c3a5e04f408e5b02c3c133248a P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:23.737974 18223 raft_consensus.cc:729] T 885fe0c3a5e04f408e5b02c3c133248a P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16d437fe629643948c95d628926d5ffa, State: Initialized, Role: FOLLOWER
I20240503 02:00:23.738126 18223 consensus_queue.cc:260] T 885fe0c3a5e04f408e5b02c3c133248a P 16d437fe629643948c95d628926d5ffa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:23.738430 18223 ts_tablet_manager.cc:1398] T 885fe0c3a5e04f408e5b02c3c133248a P 16d437fe629643948c95d628926d5ffa: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:23.738565 18223 tablet_bootstrap.cc:492] T 5833b4f9a5434734817f154d660aa9f7 P 16d437fe629643948c95d628926d5ffa: Bootstrap starting.
I20240503 02:00:23.739442 18223 tablet_bootstrap.cc:654] T 5833b4f9a5434734817f154d660aa9f7 P 16d437fe629643948c95d628926d5ffa: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:23.740160 18373 tablet_bootstrap.cc:492] T 885fe0c3a5e04f408e5b02c3c133248a P d43ed6b8663947d99a72cd922cc323aa: No bootstrap required, opened a new log
I20240503 02:00:23.740705 18222 tablet_bootstrap.cc:492] T 885fe0c3a5e04f408e5b02c3c133248a P c843193b801b4f1a88f630ed3ed0f375: No bootstrap required, opened a new log
I20240503 02:00:23.740788 18222 ts_tablet_manager.cc:1367] T 885fe0c3a5e04f408e5b02c3c133248a P c843193b801b4f1a88f630ed3ed0f375: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240503 02:00:23.740938 18369 tablet_bootstrap.cc:654] T 885fe0c3a5e04f408e5b02c3c133248a P 002cc3411acb4fdfa0b67c6f57f687a6: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:23.741278 18222 raft_consensus.cc:348] T 885fe0c3a5e04f408e5b02c3c133248a P c843193b801b4f1a88f630ed3ed0f375 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:23.740231 18373 ts_tablet_manager.cc:1367] T 885fe0c3a5e04f408e5b02c3c133248a P d43ed6b8663947d99a72cd922cc323aa: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 02:00:23.741294 17204 tablet_service.cc:1467] Processing CreateTablet for tablet 5833b4f9a5434734817f154d660aa9f7 (DEFAULT_TABLE table=rebalance_test_table_1 [id=38eeb06923e14f2ead405ffb039fd231]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:23.741389 18222 raft_consensus.cc:374] T 885fe0c3a5e04f408e5b02c3c133248a P c843193b801b4f1a88f630ed3ed0f375 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:23.741425 18222 raft_consensus.cc:729] T 885fe0c3a5e04f408e5b02c3c133248a P c843193b801b4f1a88f630ed3ed0f375 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c843193b801b4f1a88f630ed3ed0f375, State: Initialized, Role: FOLLOWER
I20240503 02:00:23.741590 17204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5833b4f9a5434734817f154d660aa9f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:23.741559 18222 consensus_queue.cc:260] T 885fe0c3a5e04f408e5b02c3c133248a P c843193b801b4f1a88f630ed3ed0f375 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:23.741808 18373 raft_consensus.cc:348] T 885fe0c3a5e04f408e5b02c3c133248a P d43ed6b8663947d99a72cd922cc323aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:23.741914 18222 ts_tablet_manager.cc:1398] T 885fe0c3a5e04f408e5b02c3c133248a P c843193b801b4f1a88f630ed3ed0f375: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:23.741938 18373 raft_consensus.cc:374] T 885fe0c3a5e04f408e5b02c3c133248a P d43ed6b8663947d99a72cd922cc323aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:23.742033 18223 tablet_bootstrap.cc:492] T 5833b4f9a5434734817f154d660aa9f7 P 16d437fe629643948c95d628926d5ffa: No bootstrap required, opened a new log
I20240503 02:00:23.742108 18223 ts_tablet_manager.cc:1367] T 5833b4f9a5434734817f154d660aa9f7 P 16d437fe629643948c95d628926d5ffa: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:00:23.742117 18222 tablet_bootstrap.cc:492] T 30abfada8ab44449b9a4237c57093b73 P c843193b801b4f1a88f630ed3ed0f375: Bootstrap starting.
I20240503 02:00:23.742514 18223 raft_consensus.cc:348] T 5833b4f9a5434734817f154d660aa9f7 P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:23.742630 18223 raft_consensus.cc:374] T 5833b4f9a5434734817f154d660aa9f7 P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:23.742661 18223 raft_consensus.cc:729] T 5833b4f9a5434734817f154d660aa9f7 P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16d437fe629643948c95d628926d5ffa, State: Initialized, Role: FOLLOWER
I20240503 02:00:23.742774 18223 consensus_queue.cc:260] T 5833b4f9a5434734817f154d660aa9f7 P 16d437fe629643948c95d628926d5ffa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:23.742990 18222 tablet_bootstrap.cc:654] T 30abfada8ab44449b9a4237c57093b73 P c843193b801b4f1a88f630ed3ed0f375: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:23.743047 18223 ts_tablet_manager.cc:1398] T 5833b4f9a5434734817f154d660aa9f7 P 16d437fe629643948c95d628926d5ffa: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:23.743214 18223 tablet_bootstrap.cc:492] T 30abfada8ab44449b9a4237c57093b73 P 16d437fe629643948c95d628926d5ffa: Bootstrap starting.
I20240503 02:00:23.741989 18373 raft_consensus.cc:729] T 885fe0c3a5e04f408e5b02c3c133248a P d43ed6b8663947d99a72cd922cc323aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d43ed6b8663947d99a72cd922cc323aa, State: Initialized, Role: FOLLOWER
I20240503 02:00:23.743819 17202 tablet_service.cc:1467] Processing CreateTablet for tablet 885fe0c3a5e04f408e5b02c3c133248a (DEFAULT_TABLE table=rebalance_test_table_1 [id=38eeb06923e14f2ead405ffb039fd231]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:23.743885 18373 consensus_queue.cc:260] T 885fe0c3a5e04f408e5b02c3c133248a P d43ed6b8663947d99a72cd922cc323aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:23.744030 18223 tablet_bootstrap.cc:654] T 30abfada8ab44449b9a4237c57093b73 P 16d437fe629643948c95d628926d5ffa: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:23.744051 17202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 885fe0c3a5e04f408e5b02c3c133248a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:23.744165 17203 tablet_service.cc:1467] Processing CreateTablet for tablet 30abfada8ab44449b9a4237c57093b73 (DEFAULT_TABLE table=rebalance_test_table_1 [id=38eeb06923e14f2ead405ffb039fd231]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:23.744269 18373 ts_tablet_manager.cc:1398] T 885fe0c3a5e04f408e5b02c3c133248a P d43ed6b8663947d99a72cd922cc323aa: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 02:00:23.744392 17203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30abfada8ab44449b9a4237c57093b73. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:23.744426 18373 tablet_bootstrap.cc:492] T 30abfada8ab44449b9a4237c57093b73 P d43ed6b8663947d99a72cd922cc323aa: Bootstrap starting.
I20240503 02:00:23.745294 18373 tablet_bootstrap.cc:654] T 30abfada8ab44449b9a4237c57093b73 P d43ed6b8663947d99a72cd922cc323aa: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:23.746284 18373 tablet_bootstrap.cc:492] T 30abfada8ab44449b9a4237c57093b73 P d43ed6b8663947d99a72cd922cc323aa: No bootstrap required, opened a new log
I20240503 02:00:23.746361 18373 ts_tablet_manager.cc:1367] T 30abfada8ab44449b9a4237c57093b73 P d43ed6b8663947d99a72cd922cc323aa: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:23.746909 18223 tablet_bootstrap.cc:492] T 30abfada8ab44449b9a4237c57093b73 P 16d437fe629643948c95d628926d5ffa: No bootstrap required, opened a new log
I20240503 02:00:23.746990 18223 ts_tablet_manager.cc:1367] T 30abfada8ab44449b9a4237c57093b73 P 16d437fe629643948c95d628926d5ffa: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 02:00:23.747126 17071 tablet_service.cc:1467] Processing CreateTablet for tablet 30abfada8ab44449b9a4237c57093b73 (DEFAULT_TABLE table=rebalance_test_table_1 [id=38eeb06923e14f2ead405ffb039fd231]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:23.747155 18373 raft_consensus.cc:348] T 30abfada8ab44449b9a4237c57093b73 P d43ed6b8663947d99a72cd922cc323aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:23.747253 18373 raft_consensus.cc:374] T 30abfada8ab44449b9a4237c57093b73 P d43ed6b8663947d99a72cd922cc323aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:23.747282 18373 raft_consensus.cc:729] T 30abfada8ab44449b9a4237c57093b73 P d43ed6b8663947d99a72cd922cc323aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d43ed6b8663947d99a72cd922cc323aa, State: Initialized, Role: FOLLOWER
I20240503 02:00:23.747401 17071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30abfada8ab44449b9a4237c57093b73. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:23.747424 17072 tablet_service.cc:1467] Processing CreateTablet for tablet 5833b4f9a5434734817f154d660aa9f7 (DEFAULT_TABLE table=rebalance_test_table_1 [id=38eeb06923e14f2ead405ffb039fd231]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:23.747434 18373 consensus_queue.cc:260] T 30abfada8ab44449b9a4237c57093b73 P d43ed6b8663947d99a72cd922cc323aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:23.747432 18223 raft_consensus.cc:348] T 30abfada8ab44449b9a4237c57093b73 P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:23.747537 18223 raft_consensus.cc:374] T 30abfada8ab44449b9a4237c57093b73 P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:23.747570 18223 raft_consensus.cc:729] T 30abfada8ab44449b9a4237c57093b73 P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16d437fe629643948c95d628926d5ffa, State: Initialized, Role: FOLLOWER
I20240503 02:00:23.747680 17072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5833b4f9a5434734817f154d660aa9f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:23.747715 18373 ts_tablet_manager.cc:1398] T 30abfada8ab44449b9a4237c57093b73 P d43ed6b8663947d99a72cd922cc323aa: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:23.747694 18223 consensus_queue.cc:260] T 30abfada8ab44449b9a4237c57093b73 P 16d437fe629643948c95d628926d5ffa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:23.747963 18223 ts_tablet_manager.cc:1398] T 30abfada8ab44449b9a4237c57093b73 P 16d437fe629643948c95d628926d5ffa: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:23.748258 18378 tablet_bootstrap.cc:492] T 5833b4f9a5434734817f154d660aa9f7 P 5b8291b26a7f448b8636187d1caf86d0: Bootstrap starting.
I20240503 02:00:23.749171 18378 tablet_bootstrap.cc:654] T 5833b4f9a5434734817f154d660aa9f7 P 5b8291b26a7f448b8636187d1caf86d0: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:23.749706 18225 tablet_bootstrap.cc:492] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67: Bootstrap starting.
I20240503 02:00:23.750255 18369 tablet_bootstrap.cc:492] T 885fe0c3a5e04f408e5b02c3c133248a P 002cc3411acb4fdfa0b67c6f57f687a6: No bootstrap required, opened a new log
I20240503 02:00:23.750346 18369 ts_tablet_manager.cc:1367] T 885fe0c3a5e04f408e5b02c3c133248a P 002cc3411acb4fdfa0b67c6f57f687a6: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240503 02:00:23.750608 18225 tablet_bootstrap.cc:654] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:23.750906 18369 raft_consensus.cc:348] T 885fe0c3a5e04f408e5b02c3c133248a P 002cc3411acb4fdfa0b67c6f57f687a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:23.751061 18369 raft_consensus.cc:374] T 885fe0c3a5e04f408e5b02c3c133248a P 002cc3411acb4fdfa0b67c6f57f687a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:23.751111 18369 raft_consensus.cc:729] T 885fe0c3a5e04f408e5b02c3c133248a P 002cc3411acb4fdfa0b67c6f57f687a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 002cc3411acb4fdfa0b67c6f57f687a6, State: Initialized, Role: FOLLOWER
I20240503 02:00:23.751271 18369 consensus_queue.cc:260] T 885fe0c3a5e04f408e5b02c3c133248a P 002cc3411acb4fdfa0b67c6f57f687a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:23.751636 18369 ts_tablet_manager.cc:1398] T 885fe0c3a5e04f408e5b02c3c133248a P 002cc3411acb4fdfa0b67c6f57f687a6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:23.751827 18369 tablet_bootstrap.cc:492] T 30abfada8ab44449b9a4237c57093b73 P 002cc3411acb4fdfa0b67c6f57f687a6: Bootstrap starting.
I20240503 02:00:23.752106 18373 tablet_bootstrap.cc:492] T 5833b4f9a5434734817f154d660aa9f7 P d43ed6b8663947d99a72cd922cc323aa: Bootstrap starting.
I20240503 02:00:23.752835 18369 tablet_bootstrap.cc:654] T 30abfada8ab44449b9a4237c57093b73 P 002cc3411acb4fdfa0b67c6f57f687a6: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:23.752951 18373 tablet_bootstrap.cc:654] T 5833b4f9a5434734817f154d660aa9f7 P d43ed6b8663947d99a72cd922cc323aa: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:23.753479 18225 tablet_bootstrap.cc:492] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67: No bootstrap required, opened a new log
I20240503 02:00:23.753576 18225 ts_tablet_manager.cc:1367] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:00:23.753954 18225 raft_consensus.cc:348] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:23.754065 18225 raft_consensus.cc:374] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:23.754101 18225 raft_consensus.cc:729] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89dd09f8aaee4b249ae3ce924a422a67, State: Initialized, Role: FOLLOWER
I20240503 02:00:23.754228 18225 consensus_queue.cc:260] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:23.754479 18225 ts_tablet_manager.cc:1398] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:23.754642 18225 tablet_bootstrap.cc:492] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67: Bootstrap starting.
I20240503 02:00:23.755822 18225 tablet_bootstrap.cc:654] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:23.756860 18225 tablet_bootstrap.cc:492] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67: No bootstrap required, opened a new log
I20240503 02:00:23.756991 18225 ts_tablet_manager.cc:1367] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:23.757480 18225 raft_consensus.cc:348] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:23.757668 18225 raft_consensus.cc:374] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:23.757750 18225 raft_consensus.cc:729] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89dd09f8aaee4b249ae3ce924a422a67, State: Initialized, Role: FOLLOWER
I20240503 02:00:23.757781 18373 tablet_bootstrap.cc:492] T 5833b4f9a5434734817f154d660aa9f7 P d43ed6b8663947d99a72cd922cc323aa: No bootstrap required, opened a new log
I20240503 02:00:23.757862 18373 ts_tablet_manager.cc:1367] T 5833b4f9a5434734817f154d660aa9f7 P d43ed6b8663947d99a72cd922cc323aa: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240503 02:00:23.758009 18225 consensus_queue.cc:260] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:23.758249 18373 raft_consensus.cc:348] T 5833b4f9a5434734817f154d660aa9f7 P d43ed6b8663947d99a72cd922cc323aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:23.758355 18373 raft_consensus.cc:374] T 5833b4f9a5434734817f154d660aa9f7 P d43ed6b8663947d99a72cd922cc323aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:23.758388 18373 raft_consensus.cc:729] T 5833b4f9a5434734817f154d660aa9f7 P d43ed6b8663947d99a72cd922cc323aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d43ed6b8663947d99a72cd922cc323aa, State: Initialized, Role: FOLLOWER
I20240503 02:00:23.758495 18225 ts_tablet_manager.cc:1398] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:23.758512 18373 consensus_queue.cc:260] T 5833b4f9a5434734817f154d660aa9f7 P d43ed6b8663947d99a72cd922cc323aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:23.758780 18373 ts_tablet_manager.cc:1398] T 5833b4f9a5434734817f154d660aa9f7 P d43ed6b8663947d99a72cd922cc323aa: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:23.761305 18224 tablet_bootstrap.cc:492] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671: No bootstrap required, opened a new log
I20240503 02:00:23.761412 18224 ts_tablet_manager.cc:1367] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671: Time spent bootstrapping tablet: real 0.038s user 0.001s sys 0.000s
I20240503 02:00:23.761817 18224 raft_consensus.cc:348] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:23.761942 18224 raft_consensus.cc:374] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:23.761994 18224 raft_consensus.cc:729] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6398c8993c3c45dcaab25b42142d5671, State: Initialized, Role: FOLLOWER
I20240503 02:00:23.762120 18224 consensus_queue.cc:260] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:23.762418 18224 ts_tablet_manager.cc:1398] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:23.762574 18224 tablet_bootstrap.cc:492] T 5833b4f9a5434734817f154d660aa9f7 P 6398c8993c3c45dcaab25b42142d5671: Bootstrap starting.
I20240503 02:00:23.762733 18222 tablet_bootstrap.cc:492] T 30abfada8ab44449b9a4237c57093b73 P c843193b801b4f1a88f630ed3ed0f375: No bootstrap required, opened a new log
I20240503 02:00:23.762836 18222 ts_tablet_manager.cc:1367] T 30abfada8ab44449b9a4237c57093b73 P c843193b801b4f1a88f630ed3ed0f375: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.000s
I20240503 02:00:23.763306 18224 tablet_bootstrap.cc:654] T 5833b4f9a5434734817f154d660aa9f7 P 6398c8993c3c45dcaab25b42142d5671: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:23.763280 18222 raft_consensus.cc:348] T 30abfada8ab44449b9a4237c57093b73 P c843193b801b4f1a88f630ed3ed0f375 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:23.763381 18222 raft_consensus.cc:374] T 30abfada8ab44449b9a4237c57093b73 P c843193b801b4f1a88f630ed3ed0f375 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:23.763413 18222 raft_consensus.cc:729] T 30abfada8ab44449b9a4237c57093b73 P c843193b801b4f1a88f630ed3ed0f375 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c843193b801b4f1a88f630ed3ed0f375, State: Initialized, Role: FOLLOWER
I20240503 02:00:23.763550 18222 consensus_queue.cc:260] T 30abfada8ab44449b9a4237c57093b73 P c843193b801b4f1a88f630ed3ed0f375 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:23.763829 18222 ts_tablet_manager.cc:1398] T 30abfada8ab44449b9a4237c57093b73 P c843193b801b4f1a88f630ed3ed0f375: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:23.764024 18222 tablet_bootstrap.cc:492] T 5833b4f9a5434734817f154d660aa9f7 P c843193b801b4f1a88f630ed3ed0f375: Bootstrap starting.
I20240503 02:00:23.764364 18224 tablet_bootstrap.cc:492] T 5833b4f9a5434734817f154d660aa9f7 P 6398c8993c3c45dcaab25b42142d5671: No bootstrap required, opened a new log
I20240503 02:00:23.764462 18224 ts_tablet_manager.cc:1367] T 5833b4f9a5434734817f154d660aa9f7 P 6398c8993c3c45dcaab25b42142d5671: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:23.764947 18224 raft_consensus.cc:348] T 5833b4f9a5434734817f154d660aa9f7 P 6398c8993c3c45dcaab25b42142d5671 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:23.765172 18222 tablet_bootstrap.cc:654] T 5833b4f9a5434734817f154d660aa9f7 P c843193b801b4f1a88f630ed3ed0f375: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:23.765053 18224 raft_consensus.cc:374] T 5833b4f9a5434734817f154d660aa9f7 P 6398c8993c3c45dcaab25b42142d5671 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:23.765468 18224 raft_consensus.cc:729] T 5833b4f9a5434734817f154d660aa9f7 P 6398c8993c3c45dcaab25b42142d5671 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6398c8993c3c45dcaab25b42142d5671, State: Initialized, Role: FOLLOWER
I20240503 02:00:23.765708 18346 raft_consensus.cc:482] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:23.765688 18224 consensus_queue.cc:260] T 5833b4f9a5434734817f154d660aa9f7 P 6398c8993c3c45dcaab25b42142d5671 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:23.765797 18346 raft_consensus.cc:504] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:23.766132 18224 ts_tablet_manager.cc:1398] T 5833b4f9a5434734817f154d660aa9f7 P 6398c8993c3c45dcaab25b42142d5671: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:23.766316 18224 tablet_bootstrap.cc:492] T 30abfada8ab44449b9a4237c57093b73 P 6398c8993c3c45dcaab25b42142d5671: Bootstrap starting.
I20240503 02:00:23.766497 18346 leader_election.cc:288] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6398c8993c3c45dcaab25b42142d5671 (127.0.99.129:36983), 5b8291b26a7f448b8636187d1caf86d0 (127.0.99.131:44479), c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134:44343), d43ed6b8663947d99a72cd922cc323aa (127.0.99.132:40023), 16d437fe629643948c95d628926d5ffa (127.0.99.133:46581), 002cc3411acb4fdfa0b67c6f57f687a6 (127.0.99.135:38989)
I20240503 02:00:23.767645 17355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30abfada8ab44449b9a4237c57093b73" candidate_uuid: "89dd09f8aaee4b249ae3ce924a422a67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d43ed6b8663947d99a72cd922cc323aa" is_pre_election: true
I20240503 02:00:23.767753 18224 tablet_bootstrap.cc:654] T 30abfada8ab44449b9a4237c57093b73 P 6398c8993c3c45dcaab25b42142d5671: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:23.767764 17355 raft_consensus.cc:2454] T 30abfada8ab44449b9a4237c57093b73 P d43ed6b8663947d99a72cd922cc323aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89dd09f8aaee4b249ae3ce924a422a67 in term 0.
I20240503 02:00:23.768152 17749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30abfada8ab44449b9a4237c57093b73" candidate_uuid: "89dd09f8aaee4b249ae3ce924a422a67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" is_pre_election: true
I20240503 02:00:23.768160 17224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30abfada8ab44449b9a4237c57093b73" candidate_uuid: "89dd09f8aaee4b249ae3ce924a422a67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b8291b26a7f448b8636187d1caf86d0" is_pre_election: true
I20240503 02:00:23.768543 17618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30abfada8ab44449b9a4237c57093b73" candidate_uuid: "89dd09f8aaee4b249ae3ce924a422a67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c843193b801b4f1a88f630ed3ed0f375" is_pre_election: true
I20240503 02:00:23.768697 17618 raft_consensus.cc:2454] T 30abfada8ab44449b9a4237c57093b73 P c843193b801b4f1a88f630ed3ed0f375 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89dd09f8aaee4b249ae3ce924a422a67 in term 0.
I20240503 02:00:23.768850 17488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30abfada8ab44449b9a4237c57093b73" candidate_uuid: "89dd09f8aaee4b249ae3ce924a422a67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16d437fe629643948c95d628926d5ffa" is_pre_election: true
I20240503 02:00:23.769012 17488 raft_consensus.cc:2454] T 30abfada8ab44449b9a4237c57093b73 P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89dd09f8aaee4b249ae3ce924a422a67 in term 0.
I20240503 02:00:23.769366 16960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30abfada8ab44449b9a4237c57093b73" candidate_uuid: "89dd09f8aaee4b249ae3ce924a422a67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6398c8993c3c45dcaab25b42142d5671" is_pre_election: true
W20240503 02:00:23.769512 17026 leader_election.cc:341] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b8291b26a7f448b8636187d1caf86d0 (127.0.99.131:44479): Illegal state: must be running to vote when last-logged opid is not known
W20240503 02:00:23.769726 17027 leader_election.cc:341] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6398c8993c3c45dcaab25b42142d5671 (127.0.99.129:36983): Illegal state: must be running to vote when last-logged opid is not known
W20240503 02:00:23.769879 17025 leader_election.cc:341] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 002cc3411acb4fdfa0b67c6f57f687a6 (127.0.99.135:38989): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:00:23.770056 17026 leader_election.cc:302] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 7 responses out of 7 voters: 4 yes votes; 3 no votes. yes voters: 16d437fe629643948c95d628926d5ffa, 89dd09f8aaee4b249ae3ce924a422a67, c843193b801b4f1a88f630ed3ed0f375, d43ed6b8663947d99a72cd922cc323aa; no voters: 002cc3411acb4fdfa0b67c6f57f687a6, 5b8291b26a7f448b8636187d1caf86d0, 6398c8993c3c45dcaab25b42142d5671
I20240503 02:00:23.770248 18346 raft_consensus.cc:2781] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:23.770298 18346 raft_consensus.cc:482] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:23.770340 18346 raft_consensus.cc:3037] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:23.771392 18346 raft_consensus.cc:504] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:23.771898 18378 tablet_bootstrap.cc:492] T 5833b4f9a5434734817f154d660aa9f7 P 5b8291b26a7f448b8636187d1caf86d0: No bootstrap required, opened a new log
I20240503 02:00:23.771991 18378 ts_tablet_manager.cc:1367] T 5833b4f9a5434734817f154d660aa9f7 P 5b8291b26a7f448b8636187d1caf86d0: Time spent bootstrapping tablet: real 0.024s user 0.001s sys 0.000s
I20240503 02:00:23.771992 18346 leader_election.cc:288] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 [CANDIDATE]: Term 1 election: Requested vote from peers 6398c8993c3c45dcaab25b42142d5671 (127.0.99.129:36983), 5b8291b26a7f448b8636187d1caf86d0 (127.0.99.131:44479), c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134:44343), d43ed6b8663947d99a72cd922cc323aa (127.0.99.132:40023), 16d437fe629643948c95d628926d5ffa (127.0.99.133:46581), 002cc3411acb4fdfa0b67c6f57f687a6 (127.0.99.135:38989)
I20240503 02:00:23.772503 18378 raft_consensus.cc:348] T 5833b4f9a5434734817f154d660aa9f7 P 5b8291b26a7f448b8636187d1caf86d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:23.772637 18378 raft_consensus.cc:374] T 5833b4f9a5434734817f154d660aa9f7 P 5b8291b26a7f448b8636187d1caf86d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:23.772791 18378 raft_consensus.cc:729] T 5833b4f9a5434734817f154d660aa9f7 P 5b8291b26a7f448b8636187d1caf86d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b8291b26a7f448b8636187d1caf86d0, State: Initialized, Role: FOLLOWER
I20240503 02:00:23.772944 18378 consensus_queue.cc:260] T 5833b4f9a5434734817f154d660aa9f7 P 5b8291b26a7f448b8636187d1caf86d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:23.772995 17355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30abfada8ab44449b9a4237c57093b73" candidate_uuid: "89dd09f8aaee4b249ae3ce924a422a67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d43ed6b8663947d99a72cd922cc323aa"
I20240503 02:00:23.773090 17355 raft_consensus.cc:3037] T 30abfada8ab44449b9a4237c57093b73 P d43ed6b8663947d99a72cd922cc323aa [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:23.773296 18378 ts_tablet_manager.cc:1398] T 5833b4f9a5434734817f154d660aa9f7 P 5b8291b26a7f448b8636187d1caf86d0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:23.773502 18378 tablet_bootstrap.cc:492] T 885fe0c3a5e04f408e5b02c3c133248a P 5b8291b26a7f448b8636187d1caf86d0: Bootstrap starting.
I20240503 02:00:23.774111 17355 raft_consensus.cc:2454] T 30abfada8ab44449b9a4237c57093b73 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89dd09f8aaee4b249ae3ce924a422a67 in term 1.
I20240503 02:00:23.774577 16960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30abfada8ab44449b9a4237c57093b73" candidate_uuid: "89dd09f8aaee4b249ae3ce924a422a67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6398c8993c3c45dcaab25b42142d5671"
I20240503 02:00:23.774693 18378 tablet_bootstrap.cc:654] T 885fe0c3a5e04f408e5b02c3c133248a P 5b8291b26a7f448b8636187d1caf86d0: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:23.774900 17224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30abfada8ab44449b9a4237c57093b73" candidate_uuid: "89dd09f8aaee4b249ae3ce924a422a67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b8291b26a7f448b8636187d1caf86d0"
I20240503 02:00:23.775065 18369 tablet_bootstrap.cc:492] T 30abfada8ab44449b9a4237c57093b73 P 002cc3411acb4fdfa0b67c6f57f687a6: No bootstrap required, opened a new log
I20240503 02:00:23.775159 18369 ts_tablet_manager.cc:1367] T 30abfada8ab44449b9a4237c57093b73 P 002cc3411acb4fdfa0b67c6f57f687a6: Time spent bootstrapping tablet: real 0.023s user 0.000s sys 0.002s
W20240503 02:00:23.775230 17026 leader_election.cc:341] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5b8291b26a7f448b8636187d1caf86d0 (127.0.99.131:44479): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:00:23.775612 18369 raft_consensus.cc:348] T 30abfada8ab44449b9a4237c57093b73 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:23.775760 18369 raft_consensus.cc:374] T 30abfada8ab44449b9a4237c57093b73 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:23.775801 18369 raft_consensus.cc:729] T 30abfada8ab44449b9a4237c57093b73 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 002cc3411acb4fdfa0b67c6f57f687a6, State: Initialized, Role: FOLLOWER
I20240503 02:00:23.775846 17488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30abfada8ab44449b9a4237c57093b73" candidate_uuid: "89dd09f8aaee4b249ae3ce924a422a67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16d437fe629643948c95d628926d5ffa"
I20240503 02:00:23.775846 17618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30abfada8ab44449b9a4237c57093b73" candidate_uuid: "89dd09f8aaee4b249ae3ce924a422a67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c843193b801b4f1a88f630ed3ed0f375"
I20240503 02:00:23.775954 17618 raft_consensus.cc:3037] T 30abfada8ab44449b9a4237c57093b73 P c843193b801b4f1a88f630ed3ed0f375 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:23.775967 17488 raft_consensus.cc:3037] T 30abfada8ab44449b9a4237c57093b73 P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:23.775930 18369 consensus_queue.cc:260] T 30abfada8ab44449b9a4237c57093b73 P 002cc3411acb4fdfa0b67c6f57f687a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:23.776063 17749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30abfada8ab44449b9a4237c57093b73" candidate_uuid: "89dd09f8aaee4b249ae3ce924a422a67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "002cc3411acb4fdfa0b67c6f57f687a6"
I20240503 02:00:23.776144 17749 raft_consensus.cc:3037] T 30abfada8ab44449b9a4237c57093b73 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:23.776229 18369 ts_tablet_manager.cc:1398] T 30abfada8ab44449b9a4237c57093b73 P 002cc3411acb4fdfa0b67c6f57f687a6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:23.776382 18369 tablet_bootstrap.cc:492] T 5833b4f9a5434734817f154d660aa9f7 P 002cc3411acb4fdfa0b67c6f57f687a6: Bootstrap starting.
I20240503 02:00:23.776965 17618 raft_consensus.cc:2454] T 30abfada8ab44449b9a4237c57093b73 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89dd09f8aaee4b249ae3ce924a422a67 in term 1.
I20240503 02:00:23.776965 17488 raft_consensus.cc:2454] T 30abfada8ab44449b9a4237c57093b73 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89dd09f8aaee4b249ae3ce924a422a67 in term 1.
I20240503 02:00:23.777411 17025 leader_election.cc:302] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 5 responses out of 7 voters: 4 yes votes; 1 no votes. yes voters: 16d437fe629643948c95d628926d5ffa, 89dd09f8aaee4b249ae3ce924a422a67, c843193b801b4f1a88f630ed3ed0f375, d43ed6b8663947d99a72cd922cc323aa; no voters: 5b8291b26a7f448b8636187d1caf86d0
I20240503 02:00:23.777576 18369 tablet_bootstrap.cc:654] T 5833b4f9a5434734817f154d660aa9f7 P 002cc3411acb4fdfa0b67c6f57f687a6: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:23.777586 17749 raft_consensus.cc:2454] T 30abfada8ab44449b9a4237c57093b73 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89dd09f8aaee4b249ae3ce924a422a67 in term 1.
W20240503 02:00:23.778009 17027 leader_election.cc:341] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6398c8993c3c45dcaab25b42142d5671 (127.0.99.129:36983): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:00:23.778435 18346 raft_consensus.cc:2781] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:23.778520 18346 raft_consensus.cc:686] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 LEADER]: Becoming Leader. State: Replica: 89dd09f8aaee4b249ae3ce924a422a67, State: Running, Role: LEADER
I20240503 02:00:23.778693 18224 tablet_bootstrap.cc:492] T 30abfada8ab44449b9a4237c57093b73 P 6398c8993c3c45dcaab25b42142d5671: No bootstrap required, opened a new log
I20240503 02:00:23.778661 18346 consensus_queue.cc:237] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:23.778769 18224 ts_tablet_manager.cc:1367] T 30abfada8ab44449b9a4237c57093b73 P 6398c8993c3c45dcaab25b42142d5671: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240503 02:00:23.779191 18224 raft_consensus.cc:348] T 30abfada8ab44449b9a4237c57093b73 P 6398c8993c3c45dcaab25b42142d5671 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:23.779314 18224 raft_consensus.cc:374] T 30abfada8ab44449b9a4237c57093b73 P 6398c8993c3c45dcaab25b42142d5671 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:23.779343 18224 raft_consensus.cc:729] T 30abfada8ab44449b9a4237c57093b73 P 6398c8993c3c45dcaab25b42142d5671 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6398c8993c3c45dcaab25b42142d5671, State: Initialized, Role: FOLLOWER
I20240503 02:00:23.779460 18224 consensus_queue.cc:260] T 30abfada8ab44449b9a4237c57093b73 P 6398c8993c3c45dcaab25b42142d5671 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:23.779572 18222 tablet_bootstrap.cc:492] T 5833b4f9a5434734817f154d660aa9f7 P c843193b801b4f1a88f630ed3ed0f375: No bootstrap required, opened a new log
I20240503 02:00:23.779666 18222 ts_tablet_manager.cc:1367] T 5833b4f9a5434734817f154d660aa9f7 P c843193b801b4f1a88f630ed3ed0f375: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20240503 02:00:23.779850 18224 ts_tablet_manager.cc:1398] T 30abfada8ab44449b9a4237c57093b73 P 6398c8993c3c45dcaab25b42142d5671: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:23.780117 18222 raft_consensus.cc:348] T 5833b4f9a5434734817f154d660aa9f7 P c843193b801b4f1a88f630ed3ed0f375 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:23.780241 18222 raft_consensus.cc:374] T 5833b4f9a5434734817f154d660aa9f7 P c843193b801b4f1a88f630ed3ed0f375 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:23.780289 18222 raft_consensus.cc:729] T 5833b4f9a5434734817f154d660aa9f7 P c843193b801b4f1a88f630ed3ed0f375 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c843193b801b4f1a88f630ed3ed0f375, State: Initialized, Role: FOLLOWER
I20240503 02:00:23.780416 18222 consensus_queue.cc:260] T 5833b4f9a5434734817f154d660aa9f7 P c843193b801b4f1a88f630ed3ed0f375 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:23.780745 18222 ts_tablet_manager.cc:1398] T 5833b4f9a5434734817f154d660aa9f7 P c843193b801b4f1a88f630ed3ed0f375: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:23.781550 18369 tablet_bootstrap.cc:492] T 5833b4f9a5434734817f154d660aa9f7 P 002cc3411acb4fdfa0b67c6f57f687a6: No bootstrap required, opened a new log
I20240503 02:00:23.781643 18369 ts_tablet_manager.cc:1367] T 5833b4f9a5434734817f154d660aa9f7 P 002cc3411acb4fdfa0b67c6f57f687a6: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 02:00:23.781577 16818 catalog_manager.cc:5484] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 reported cstate change: term changed from 0 to 1, leader changed from <none> to 89dd09f8aaee4b249ae3ce924a422a67 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "89dd09f8aaee4b249ae3ce924a422a67" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:23.782302 18369 raft_consensus.cc:348] T 5833b4f9a5434734817f154d660aa9f7 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:23.782444 18369 raft_consensus.cc:374] T 5833b4f9a5434734817f154d660aa9f7 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:23.782480 18369 raft_consensus.cc:729] T 5833b4f9a5434734817f154d660aa9f7 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 002cc3411acb4fdfa0b67c6f57f687a6, State: Initialized, Role: FOLLOWER
I20240503 02:00:23.782613 18369 consensus_queue.cc:260] T 5833b4f9a5434734817f154d660aa9f7 P 002cc3411acb4fdfa0b67c6f57f687a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:23.782737 18378 tablet_bootstrap.cc:492] T 885fe0c3a5e04f408e5b02c3c133248a P 5b8291b26a7f448b8636187d1caf86d0: No bootstrap required, opened a new log
I20240503 02:00:23.782827 18378 ts_tablet_manager.cc:1367] T 885fe0c3a5e04f408e5b02c3c133248a P 5b8291b26a7f448b8636187d1caf86d0: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 02:00:23.782949 18369 ts_tablet_manager.cc:1398] T 5833b4f9a5434734817f154d660aa9f7 P 002cc3411acb4fdfa0b67c6f57f687a6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:23.783258 18378 raft_consensus.cc:348] T 885fe0c3a5e04f408e5b02c3c133248a P 5b8291b26a7f448b8636187d1caf86d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:23.783403 18378 raft_consensus.cc:374] T 885fe0c3a5e04f408e5b02c3c133248a P 5b8291b26a7f448b8636187d1caf86d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:23.783440 18378 raft_consensus.cc:729] T 885fe0c3a5e04f408e5b02c3c133248a P 5b8291b26a7f448b8636187d1caf86d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b8291b26a7f448b8636187d1caf86d0, State: Initialized, Role: FOLLOWER
I20240503 02:00:23.783561 18378 consensus_queue.cc:260] T 885fe0c3a5e04f408e5b02c3c133248a P 5b8291b26a7f448b8636187d1caf86d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:23.783854 18378 ts_tablet_manager.cc:1398] T 885fe0c3a5e04f408e5b02c3c133248a P 5b8291b26a7f448b8636187d1caf86d0: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:23.784018 18378 tablet_bootstrap.cc:492] T 30abfada8ab44449b9a4237c57093b73 P 5b8291b26a7f448b8636187d1caf86d0: Bootstrap starting.
I20240503 02:00:23.784930 18378 tablet_bootstrap.cc:654] T 30abfada8ab44449b9a4237c57093b73 P 5b8291b26a7f448b8636187d1caf86d0: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:23.786509 18378 tablet_bootstrap.cc:492] T 30abfada8ab44449b9a4237c57093b73 P 5b8291b26a7f448b8636187d1caf86d0: No bootstrap required, opened a new log
I20240503 02:00:23.786581 18378 ts_tablet_manager.cc:1367] T 30abfada8ab44449b9a4237c57093b73 P 5b8291b26a7f448b8636187d1caf86d0: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:23.786958 18378 raft_consensus.cc:348] T 30abfada8ab44449b9a4237c57093b73 P 5b8291b26a7f448b8636187d1caf86d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:23.787062 18378 raft_consensus.cc:374] T 30abfada8ab44449b9a4237c57093b73 P 5b8291b26a7f448b8636187d1caf86d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:23.787086 18378 raft_consensus.cc:729] T 30abfada8ab44449b9a4237c57093b73 P 5b8291b26a7f448b8636187d1caf86d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b8291b26a7f448b8636187d1caf86d0, State: Initialized, Role: FOLLOWER
I20240503 02:00:23.787209 18378 consensus_queue.cc:260] T 30abfada8ab44449b9a4237c57093b73 P 5b8291b26a7f448b8636187d1caf86d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:23.787492 18378 ts_tablet_manager.cc:1398] T 30abfada8ab44449b9a4237c57093b73 P 5b8291b26a7f448b8636187d1caf86d0: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:23.806948 18346 raft_consensus.cc:482] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:23.807077 18346 raft_consensus.cc:504] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:23.807680 18346 leader_election.cc:288] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6398c8993c3c45dcaab25b42142d5671 (127.0.99.129:36983), 5b8291b26a7f448b8636187d1caf86d0 (127.0.99.131:44479), c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134:44343), 16d437fe629643948c95d628926d5ffa (127.0.99.133:46581), d43ed6b8663947d99a72cd922cc323aa (127.0.99.132:40023), 002cc3411acb4fdfa0b67c6f57f687a6 (127.0.99.135:38989)
I20240503 02:00:23.808028 16960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5833b4f9a5434734817f154d660aa9f7" candidate_uuid: "89dd09f8aaee4b249ae3ce924a422a67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6398c8993c3c45dcaab25b42142d5671" is_pre_election: true
I20240503 02:00:23.808070 17488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5833b4f9a5434734817f154d660aa9f7" candidate_uuid: "89dd09f8aaee4b249ae3ce924a422a67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16d437fe629643948c95d628926d5ffa" is_pre_election: true
I20240503 02:00:23.808070 17749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5833b4f9a5434734817f154d660aa9f7" candidate_uuid: "89dd09f8aaee4b249ae3ce924a422a67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" is_pre_election: true
I20240503 02:00:23.808105 17355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5833b4f9a5434734817f154d660aa9f7" candidate_uuid: "89dd09f8aaee4b249ae3ce924a422a67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d43ed6b8663947d99a72cd922cc323aa" is_pre_election: true
I20240503 02:00:23.808171 16960 raft_consensus.cc:2454] T 5833b4f9a5434734817f154d660aa9f7 P 6398c8993c3c45dcaab25b42142d5671 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89dd09f8aaee4b249ae3ce924a422a67 in term 0.
I20240503 02:00:23.808198 17749 raft_consensus.cc:2454] T 5833b4f9a5434734817f154d660aa9f7 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89dd09f8aaee4b249ae3ce924a422a67 in term 0.
I20240503 02:00:23.808198 17488 raft_consensus.cc:2454] T 5833b4f9a5434734817f154d660aa9f7 P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89dd09f8aaee4b249ae3ce924a422a67 in term 0.
I20240503 02:00:23.808228 17355 raft_consensus.cc:2454] T 5833b4f9a5434734817f154d660aa9f7 P d43ed6b8663947d99a72cd922cc323aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89dd09f8aaee4b249ae3ce924a422a67 in term 0.
I20240503 02:00:23.808496 17224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5833b4f9a5434734817f154d660aa9f7" candidate_uuid: "89dd09f8aaee4b249ae3ce924a422a67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b8291b26a7f448b8636187d1caf86d0" is_pre_election: true
I20240503 02:00:23.808635 17224 raft_consensus.cc:2454] T 5833b4f9a5434734817f154d660aa9f7 P 5b8291b26a7f448b8636187d1caf86d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89dd09f8aaee4b249ae3ce924a422a67 in term 0.
I20240503 02:00:23.808722 17025 leader_election.cc:302] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 002cc3411acb4fdfa0b67c6f57f687a6, 16d437fe629643948c95d628926d5ffa, 89dd09f8aaee4b249ae3ce924a422a67, d43ed6b8663947d99a72cd922cc323aa; no voters:
I20240503 02:00:23.808877 18346 raft_consensus.cc:2781] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:23.808943 18346 raft_consensus.cc:482] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:23.808990 18346 raft_consensus.cc:3037] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:23.808933 17618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5833b4f9a5434734817f154d660aa9f7" candidate_uuid: "89dd09f8aaee4b249ae3ce924a422a67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c843193b801b4f1a88f630ed3ed0f375" is_pre_election: true
I20240503 02:00:23.809077 17618 raft_consensus.cc:2454] T 5833b4f9a5434734817f154d660aa9f7 P c843193b801b4f1a88f630ed3ed0f375 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89dd09f8aaee4b249ae3ce924a422a67 in term 0.
I20240503 02:00:23.810139 18346 raft_consensus.cc:504] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:23.810671 18346 leader_election.cc:288] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 [CANDIDATE]: Term 1 election: Requested vote from peers 6398c8993c3c45dcaab25b42142d5671 (127.0.99.129:36983), 5b8291b26a7f448b8636187d1caf86d0 (127.0.99.131:44479), c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134:44343), 16d437fe629643948c95d628926d5ffa (127.0.99.133:46581), d43ed6b8663947d99a72cd922cc323aa (127.0.99.132:40023), 002cc3411acb4fdfa0b67c6f57f687a6 (127.0.99.135:38989)
I20240503 02:00:23.810858 16960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5833b4f9a5434734817f154d660aa9f7" candidate_uuid: "89dd09f8aaee4b249ae3ce924a422a67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6398c8993c3c45dcaab25b42142d5671"
I20240503 02:00:23.810973 16960 raft_consensus.cc:3037] T 5833b4f9a5434734817f154d660aa9f7 P 6398c8993c3c45dcaab25b42142d5671 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:23.810983 17355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5833b4f9a5434734817f154d660aa9f7" candidate_uuid: "89dd09f8aaee4b249ae3ce924a422a67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d43ed6b8663947d99a72cd922cc323aa"
I20240503 02:00:23.810993 17749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5833b4f9a5434734817f154d660aa9f7" candidate_uuid: "89dd09f8aaee4b249ae3ce924a422a67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "002cc3411acb4fdfa0b67c6f57f687a6"
I20240503 02:00:23.811079 17355 raft_consensus.cc:3037] T 5833b4f9a5434734817f154d660aa9f7 P d43ed6b8663947d99a72cd922cc323aa [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:23.811106 17749 raft_consensus.cc:3037] T 5833b4f9a5434734817f154d660aa9f7 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:23.811254 17488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5833b4f9a5434734817f154d660aa9f7" candidate_uuid: "89dd09f8aaee4b249ae3ce924a422a67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16d437fe629643948c95d628926d5ffa"
I20240503 02:00:23.811358 17488 raft_consensus.cc:3037] T 5833b4f9a5434734817f154d660aa9f7 P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:23.811955 16960 raft_consensus.cc:2454] T 5833b4f9a5434734817f154d660aa9f7 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89dd09f8aaee4b249ae3ce924a422a67 in term 1.
I20240503 02:00:23.811952 17355 raft_consensus.cc:2454] T 5833b4f9a5434734817f154d660aa9f7 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89dd09f8aaee4b249ae3ce924a422a67 in term 1.
I20240503 02:00:23.812054 17749 raft_consensus.cc:2454] T 5833b4f9a5434734817f154d660aa9f7 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89dd09f8aaee4b249ae3ce924a422a67 in term 1.
I20240503 02:00:23.812304 17488 raft_consensus.cc:2454] T 5833b4f9a5434734817f154d660aa9f7 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89dd09f8aaee4b249ae3ce924a422a67 in term 1.
I20240503 02:00:23.812399 17027 leader_election.cc:302] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 002cc3411acb4fdfa0b67c6f57f687a6, 6398c8993c3c45dcaab25b42142d5671, 89dd09f8aaee4b249ae3ce924a422a67, d43ed6b8663947d99a72cd922cc323aa; no voters:
I20240503 02:00:23.812572 18346 raft_consensus.cc:2781] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:23.812754 18346 raft_consensus.cc:686] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 LEADER]: Becoming Leader. State: Replica: 89dd09f8aaee4b249ae3ce924a422a67, State: Running, Role: LEADER
I20240503 02:00:23.812726 17224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5833b4f9a5434734817f154d660aa9f7" candidate_uuid: "89dd09f8aaee4b249ae3ce924a422a67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b8291b26a7f448b8636187d1caf86d0"
I20240503 02:00:23.812759 17618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5833b4f9a5434734817f154d660aa9f7" candidate_uuid: "89dd09f8aaee4b249ae3ce924a422a67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c843193b801b4f1a88f630ed3ed0f375"
I20240503 02:00:23.812836 17224 raft_consensus.cc:3037] T 5833b4f9a5434734817f154d660aa9f7 P 5b8291b26a7f448b8636187d1caf86d0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:23.812880 17618 raft_consensus.cc:3037] T 5833b4f9a5434734817f154d660aa9f7 P c843193b801b4f1a88f630ed3ed0f375 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:23.812919 18346 consensus_queue.cc:237] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:23.813711 17618 raft_consensus.cc:2454] T 5833b4f9a5434734817f154d660aa9f7 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89dd09f8aaee4b249ae3ce924a422a67 in term 1.
I20240503 02:00:23.813796 17224 raft_consensus.cc:2454] T 5833b4f9a5434734817f154d660aa9f7 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89dd09f8aaee4b249ae3ce924a422a67 in term 1.
I20240503 02:00:23.815053 16818 catalog_manager.cc:5484] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 reported cstate change: term changed from 0 to 1, leader changed from <none> to 89dd09f8aaee4b249ae3ce924a422a67 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "89dd09f8aaee4b249ae3ce924a422a67" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:23.860934 18248 raft_consensus.cc:482] T 30abfada8ab44449b9a4237c57093b73 P 5b8291b26a7f448b8636187d1caf86d0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:23.861086 18248 raft_consensus.cc:504] T 30abfada8ab44449b9a4237c57093b73 P 5b8291b26a7f448b8636187d1caf86d0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:23.862756 18248 leader_election.cc:288] T 30abfada8ab44449b9a4237c57093b73 P 5b8291b26a7f448b8636187d1caf86d0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6398c8993c3c45dcaab25b42142d5671 (127.0.99.129:36983), 89dd09f8aaee4b249ae3ce924a422a67 (127.0.99.130:38889), c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134:44343), d43ed6b8663947d99a72cd922cc323aa (127.0.99.132:40023), 16d437fe629643948c95d628926d5ffa (127.0.99.133:46581), 002cc3411acb4fdfa0b67c6f57f687a6 (127.0.99.135:38989)
I20240503 02:00:23.868208 16960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30abfada8ab44449b9a4237c57093b73" candidate_uuid: "5b8291b26a7f448b8636187d1caf86d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6398c8993c3c45dcaab25b42142d5671" is_pre_election: true
I20240503 02:00:23.868391 16960 raft_consensus.cc:2454] T 30abfada8ab44449b9a4237c57093b73 P 6398c8993c3c45dcaab25b42142d5671 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b8291b26a7f448b8636187d1caf86d0 in term 0.
I20240503 02:00:23.868553 17090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30abfada8ab44449b9a4237c57093b73" candidate_uuid: "5b8291b26a7f448b8636187d1caf86d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89dd09f8aaee4b249ae3ce924a422a67" is_pre_election: true
I20240503 02:00:23.868813 17618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30abfada8ab44449b9a4237c57093b73" candidate_uuid: "5b8291b26a7f448b8636187d1caf86d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c843193b801b4f1a88f630ed3ed0f375" is_pre_election: true
I20240503 02:00:23.868994 17618 raft_consensus.cc:2379] T 30abfada8ab44449b9a4237c57093b73 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5b8291b26a7f448b8636187d1caf86d0 in current term 1: Already voted for candidate 89dd09f8aaee4b249ae3ce924a422a67 in this term.
I20240503 02:00:23.870587 17488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30abfada8ab44449b9a4237c57093b73" candidate_uuid: "5b8291b26a7f448b8636187d1caf86d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16d437fe629643948c95d628926d5ffa" is_pre_election: true
I20240503 02:00:23.870633 17355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30abfada8ab44449b9a4237c57093b73" candidate_uuid: "5b8291b26a7f448b8636187d1caf86d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d43ed6b8663947d99a72cd922cc323aa" is_pre_election: true
I20240503 02:00:23.870782 17488 raft_consensus.cc:2379] T 30abfada8ab44449b9a4237c57093b73 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5b8291b26a7f448b8636187d1caf86d0 in current term 1: Already voted for candidate 89dd09f8aaee4b249ae3ce924a422a67 in this term.
I20240503 02:00:23.870792 17355 raft_consensus.cc:2379] T 30abfada8ab44449b9a4237c57093b73 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5b8291b26a7f448b8636187d1caf86d0 in current term 1: Already voted for candidate 89dd09f8aaee4b249ae3ce924a422a67 in this term.
I20240503 02:00:23.871125 17157 leader_election.cc:302] T 30abfada8ab44449b9a4237c57093b73 P 5b8291b26a7f448b8636187d1caf86d0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 6 responses out of 7 voters: 2 yes votes; 4 no votes. yes voters: 5b8291b26a7f448b8636187d1caf86d0, 6398c8993c3c45dcaab25b42142d5671; no voters: 16d437fe629643948c95d628926d5ffa, 89dd09f8aaee4b249ae3ce924a422a67, c843193b801b4f1a88f630ed3ed0f375, d43ed6b8663947d99a72cd922cc323aa
I20240503 02:00:23.871389 18248 raft_consensus.cc:3037] T 30abfada8ab44449b9a4237c57093b73 P 5b8291b26a7f448b8636187d1caf86d0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:23.872442 18248 raft_consensus.cc:2726] T 30abfada8ab44449b9a4237c57093b73 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:00:23.872960 17749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30abfada8ab44449b9a4237c57093b73" candidate_uuid: "5b8291b26a7f448b8636187d1caf86d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" is_pre_election: true
I20240503 02:00:23.873148 17749 raft_consensus.cc:2379] T 30abfada8ab44449b9a4237c57093b73 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5b8291b26a7f448b8636187d1caf86d0 in current term 1: Already voted for candidate 89dd09f8aaee4b249ae3ce924a422a67 in this term.
I20240503 02:00:23.908334 18241 raft_consensus.cc:482] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:23.908506 18241 raft_consensus.cc:504] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:23.909862 18241 leader_election.cc:288] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89dd09f8aaee4b249ae3ce924a422a67 (127.0.99.130:38889), 5b8291b26a7f448b8636187d1caf86d0 (127.0.99.131:44479), d43ed6b8663947d99a72cd922cc323aa (127.0.99.132:40023), 16d437fe629643948c95d628926d5ffa (127.0.99.133:46581), c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134:44343), 002cc3411acb4fdfa0b67c6f57f687a6 (127.0.99.135:38989)
I20240503 02:00:23.914973 17090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "885fe0c3a5e04f408e5b02c3c133248a" candidate_uuid: "6398c8993c3c45dcaab25b42142d5671" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89dd09f8aaee4b249ae3ce924a422a67" is_pre_election: true
I20240503 02:00:23.914973 17488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "885fe0c3a5e04f408e5b02c3c133248a" candidate_uuid: "6398c8993c3c45dcaab25b42142d5671" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16d437fe629643948c95d628926d5ffa" is_pre_election: true
I20240503 02:00:23.915129 17090 raft_consensus.cc:2454] T 885fe0c3a5e04f408e5b02c3c133248a P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6398c8993c3c45dcaab25b42142d5671 in term 0.
I20240503 02:00:23.915129 17488 raft_consensus.cc:2454] T 885fe0c3a5e04f408e5b02c3c133248a P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6398c8993c3c45dcaab25b42142d5671 in term 0.
I20240503 02:00:23.915813 17618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "885fe0c3a5e04f408e5b02c3c133248a" candidate_uuid: "6398c8993c3c45dcaab25b42142d5671" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c843193b801b4f1a88f630ed3ed0f375" is_pre_election: true
I20240503 02:00:23.915952 17618 raft_consensus.cc:2454] T 885fe0c3a5e04f408e5b02c3c133248a P c843193b801b4f1a88f630ed3ed0f375 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6398c8993c3c45dcaab25b42142d5671 in term 0.
I20240503 02:00:23.915977 17355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "885fe0c3a5e04f408e5b02c3c133248a" candidate_uuid: "6398c8993c3c45dcaab25b42142d5671" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d43ed6b8663947d99a72cd922cc323aa" is_pre_election: true
I20240503 02:00:23.916096 17355 raft_consensus.cc:2454] T 885fe0c3a5e04f408e5b02c3c133248a P d43ed6b8663947d99a72cd922cc323aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6398c8993c3c45dcaab25b42142d5671 in term 0.
I20240503 02:00:23.916700 16893 leader_election.cc:302] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 16d437fe629643948c95d628926d5ffa, 6398c8993c3c45dcaab25b42142d5671, 89dd09f8aaee4b249ae3ce924a422a67, c843193b801b4f1a88f630ed3ed0f375; no voters:
I20240503 02:00:23.916955 18241 raft_consensus.cc:2781] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:23.917021 18241 raft_consensus.cc:482] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:23.917066 18241 raft_consensus.cc:3037] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:23.917989 18241 raft_consensus.cc:504] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:23.918658 18241 leader_election.cc:288] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671 [CANDIDATE]: Term 1 election: Requested vote from peers 89dd09f8aaee4b249ae3ce924a422a67 (127.0.99.130:38889), 5b8291b26a7f448b8636187d1caf86d0 (127.0.99.131:44479), d43ed6b8663947d99a72cd922cc323aa (127.0.99.132:40023), 16d437fe629643948c95d628926d5ffa (127.0.99.133:46581), c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134:44343), 002cc3411acb4fdfa0b67c6f57f687a6 (127.0.99.135:38989)
I20240503 02:00:23.918853 17749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "885fe0c3a5e04f408e5b02c3c133248a" candidate_uuid: "6398c8993c3c45dcaab25b42142d5671" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" is_pre_election: true
I20240503 02:00:23.918979 17749 raft_consensus.cc:2454] T 885fe0c3a5e04f408e5b02c3c133248a P 002cc3411acb4fdfa0b67c6f57f687a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6398c8993c3c45dcaab25b42142d5671 in term 0.
I20240503 02:00:23.919034 17750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "885fe0c3a5e04f408e5b02c3c133248a" candidate_uuid: "6398c8993c3c45dcaab25b42142d5671" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "002cc3411acb4fdfa0b67c6f57f687a6"
I20240503 02:00:23.919142 17750 raft_consensus.cc:3037] T 885fe0c3a5e04f408e5b02c3c133248a P 002cc3411acb4fdfa0b67c6f57f687a6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:23.919189 17488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "885fe0c3a5e04f408e5b02c3c133248a" candidate_uuid: "6398c8993c3c45dcaab25b42142d5671" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16d437fe629643948c95d628926d5ffa"
I20240503 02:00:23.919253 17090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "885fe0c3a5e04f408e5b02c3c133248a" candidate_uuid: "6398c8993c3c45dcaab25b42142d5671" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89dd09f8aaee4b249ae3ce924a422a67"
I20240503 02:00:23.919299 17488 raft_consensus.cc:3037] T 885fe0c3a5e04f408e5b02c3c133248a P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:23.919358 17090 raft_consensus.cc:3037] T 885fe0c3a5e04f408e5b02c3c133248a P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:23.920229 17090 raft_consensus.cc:2454] T 885fe0c3a5e04f408e5b02c3c133248a P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6398c8993c3c45dcaab25b42142d5671 in term 1.
I20240503 02:00:23.920229 17488 raft_consensus.cc:2454] T 885fe0c3a5e04f408e5b02c3c133248a P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6398c8993c3c45dcaab25b42142d5671 in term 1.
I20240503 02:00:23.920234 17750 raft_consensus.cc:2454] T 885fe0c3a5e04f408e5b02c3c133248a P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6398c8993c3c45dcaab25b42142d5671 in term 1.
I20240503 02:00:23.920483 17618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "885fe0c3a5e04f408e5b02c3c133248a" candidate_uuid: "6398c8993c3c45dcaab25b42142d5671" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c843193b801b4f1a88f630ed3ed0f375"
I20240503 02:00:23.920612 17618 raft_consensus.cc:3037] T 885fe0c3a5e04f408e5b02c3c133248a P c843193b801b4f1a88f630ed3ed0f375 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:23.920603 17355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "885fe0c3a5e04f408e5b02c3c133248a" candidate_uuid: "6398c8993c3c45dcaab25b42142d5671" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d43ed6b8663947d99a72cd922cc323aa"
I20240503 02:00:23.920753 17355 raft_consensus.cc:3037] T 885fe0c3a5e04f408e5b02c3c133248a P d43ed6b8663947d99a72cd922cc323aa [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:23.920965 17224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "885fe0c3a5e04f408e5b02c3c133248a" candidate_uuid: "6398c8993c3c45dcaab25b42142d5671" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b8291b26a7f448b8636187d1caf86d0" is_pre_election: true
I20240503 02:00:23.921103 17224 raft_consensus.cc:2454] T 885fe0c3a5e04f408e5b02c3c133248a P 5b8291b26a7f448b8636187d1caf86d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6398c8993c3c45dcaab25b42142d5671 in term 0.
I20240503 02:00:23.921279 17223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "885fe0c3a5e04f408e5b02c3c133248a" candidate_uuid: "6398c8993c3c45dcaab25b42142d5671" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b8291b26a7f448b8636187d1caf86d0"
I20240503 02:00:23.921375 17223 raft_consensus.cc:3037] T 885fe0c3a5e04f408e5b02c3c133248a P 5b8291b26a7f448b8636187d1caf86d0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:23.921598 17618 raft_consensus.cc:2454] T 885fe0c3a5e04f408e5b02c3c133248a P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6398c8993c3c45dcaab25b42142d5671 in term 1.
I20240503 02:00:23.921633 17355 raft_consensus.cc:2454] T 885fe0c3a5e04f408e5b02c3c133248a P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6398c8993c3c45dcaab25b42142d5671 in term 1.
I20240503 02:00:23.921869 16896 leader_election.cc:302] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 002cc3411acb4fdfa0b67c6f57f687a6, 16d437fe629643948c95d628926d5ffa, 6398c8993c3c45dcaab25b42142d5671, 89dd09f8aaee4b249ae3ce924a422a67; no voters:
I20240503 02:00:23.922062 18241 raft_consensus.cc:2781] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:23.922250 18241 raft_consensus.cc:686] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671 [term 1 LEADER]: Becoming Leader. State: Replica: 6398c8993c3c45dcaab25b42142d5671, State: Running, Role: LEADER
I20240503 02:00:23.922565 18241 consensus_queue.cc:237] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:23.922869 17223 raft_consensus.cc:2454] T 885fe0c3a5e04f408e5b02c3c133248a P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6398c8993c3c45dcaab25b42142d5671 in term 1.
I20240503 02:00:23.925467 16818 catalog_manager.cc:5484] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6398c8993c3c45dcaab25b42142d5671 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "6398c8993c3c45dcaab25b42142d5671" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } health_report { overall_health: UNKNOWN } } }
W20240503 02:00:24.052074 18413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:00:24.052321 18413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:00:24.055322 18413 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240503 02:00:24.141960 17488 raft_consensus.cc:1261] T 30abfada8ab44449b9a4237c57093b73 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:24.142102 17355 raft_consensus.cc:1261] T 30abfada8ab44449b9a4237c57093b73 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:24.142488 17750 raft_consensus.cc:1261] T 30abfada8ab44449b9a4237c57093b73 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:24.142587 17618 raft_consensus.cc:1261] T 30abfada8ab44449b9a4237c57093b73 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:24.143029 16960 raft_consensus.cc:3037] T 30abfada8ab44449b9a4237c57093b73 P 6398c8993c3c45dcaab25b42142d5671 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:24.143059 18346 consensus_queue.cc:1035] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:24.143128 17223 raft_consensus.cc:1261] T 30abfada8ab44449b9a4237c57093b73 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:24.143226 18365 consensus_queue.cc:1035] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:24.143440 18365 consensus_queue.cc:1035] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:24.143687 18365 consensus_queue.cc:1035] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:24.143958 18365 consensus_queue.cc:1035] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:24.144191 16960 raft_consensus.cc:1261] T 30abfada8ab44449b9a4237c57093b73 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:24.147761 18346 consensus_queue.cc:1035] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:24.155576 17620 raft_consensus.cc:1261] T 885fe0c3a5e04f408e5b02c3c133248a P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer 6398c8993c3c45dcaab25b42142d5671: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:24.156075 18241 consensus_queue.cc:1035] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:24.156198 17354 raft_consensus.cc:1261] T 885fe0c3a5e04f408e5b02c3c133248a P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Refusing update from remote peer 6398c8993c3c45dcaab25b42142d5671: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:24.156291 17487 raft_consensus.cc:1261] T 885fe0c3a5e04f408e5b02c3c133248a P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Refusing update from remote peer 6398c8993c3c45dcaab25b42142d5671: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:24.156606 18241 consensus_queue.cc:1035] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:24.157291 17750 raft_consensus.cc:1261] T 885fe0c3a5e04f408e5b02c3c133248a P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer 6398c8993c3c45dcaab25b42142d5671: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:24.162196 17090 raft_consensus.cc:1261] T 885fe0c3a5e04f408e5b02c3c133248a P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer 6398c8993c3c45dcaab25b42142d5671: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:24.162536 17224 raft_consensus.cc:1261] T 885fe0c3a5e04f408e5b02c3c133248a P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer 6398c8993c3c45dcaab25b42142d5671: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:24.162953 18241 consensus_queue.cc:1035] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:24.163529 18241 consensus_queue.cc:1035] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:24.165436 17356 raft_consensus.cc:1261] T 5833b4f9a5434734817f154d660aa9f7 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:24.169257 17223 raft_consensus.cc:1261] T 5833b4f9a5434734817f154d660aa9f7 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:24.169765 18362 consensus_queue.cc:1035] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:24.171379 16959 raft_consensus.cc:1261] T 5833b4f9a5434734817f154d660aa9f7 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:24.172266 17487 raft_consensus.cc:1261] T 5833b4f9a5434734817f154d660aa9f7 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:24.172858 17618 raft_consensus.cc:1261] T 5833b4f9a5434734817f154d660aa9f7 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:24.174422 17749 raft_consensus.cc:1261] T 5833b4f9a5434734817f154d660aa9f7 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:24.174968 18346 consensus_queue.cc:1035] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:24.181934 18362 consensus_queue.cc:1035] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:24.186170 18241 consensus_queue.cc:1035] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671 [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:24.186460 18241 consensus_queue.cc:1035] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:24.187230 18362 consensus_queue.cc:1035] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:24.187693 18362 consensus_queue.cc:1035] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:24.187949 18362 consensus_queue.cc:1035] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 202.408 ms
time per row: 0.101204 ms
I20240503 02:00:24.301956 16813 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39916:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 7
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240503 02:00:24.302515 16813 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_2 in case of a server failure: 8 tablet servers would be needed, 7 are available. Consider bringing up more tablet servers.
I20240503 02:00:24.308094 16938 tablet_service.cc:1467] Processing CreateTablet for tablet 78f81fe4c6234ff0b5fcabd974779a66 (DEFAULT_TABLE table=rebalance_test_table_2 [id=9f4cafd5cdb041c1bbe1592f04370c0b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:24.308167 17072 tablet_service.cc:1467] Processing CreateTablet for tablet 78f81fe4c6234ff0b5fcabd974779a66 (DEFAULT_TABLE table=rebalance_test_table_2 [id=9f4cafd5cdb041c1bbe1592f04370c0b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:24.308378 16938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78f81fe4c6234ff0b5fcabd974779a66. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:24.308439 17072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78f81fe4c6234ff0b5fcabd974779a66. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:24.311013 18490 tablet_bootstrap.cc:492] T 78f81fe4c6234ff0b5fcabd974779a66 P 6398c8993c3c45dcaab25b42142d5671: Bootstrap starting.
I20240503 02:00:24.311890 18490 tablet_bootstrap.cc:654] T 78f81fe4c6234ff0b5fcabd974779a66 P 6398c8993c3c45dcaab25b42142d5671: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:24.312891 18490 tablet_bootstrap.cc:492] T 78f81fe4c6234ff0b5fcabd974779a66 P 6398c8993c3c45dcaab25b42142d5671: No bootstrap required, opened a new log
I20240503 02:00:24.312966 18490 ts_tablet_manager.cc:1367] T 78f81fe4c6234ff0b5fcabd974779a66 P 6398c8993c3c45dcaab25b42142d5671: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:24.313328 18490 raft_consensus.cc:348] T 78f81fe4c6234ff0b5fcabd974779a66 P 6398c8993c3c45dcaab25b42142d5671 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:24.313429 18490 raft_consensus.cc:374] T 78f81fe4c6234ff0b5fcabd974779a66 P 6398c8993c3c45dcaab25b42142d5671 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:24.313453 18490 raft_consensus.cc:729] T 78f81fe4c6234ff0b5fcabd974779a66 P 6398c8993c3c45dcaab25b42142d5671 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6398c8993c3c45dcaab25b42142d5671, State: Initialized, Role: FOLLOWER
I20240503 02:00:24.313555 18490 consensus_queue.cc:260] T 78f81fe4c6234ff0b5fcabd974779a66 P 6398c8993c3c45dcaab25b42142d5671 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:24.313827 18490 ts_tablet_manager.cc:1398] T 78f81fe4c6234ff0b5fcabd974779a66 P 6398c8993c3c45dcaab25b42142d5671: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:24.314332 17466 tablet_service.cc:1467] Processing CreateTablet for tablet 78f81fe4c6234ff0b5fcabd974779a66 (DEFAULT_TABLE table=rebalance_test_table_2 [id=9f4cafd5cdb041c1bbe1592f04370c0b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:24.314625 17466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78f81fe4c6234ff0b5fcabd974779a66. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:24.315786 16939 tablet_service.cc:1467] Processing CreateTablet for tablet 5d2104698d5f452c8350cce10afd3898 (DEFAULT_TABLE table=rebalance_test_table_2 [id=9f4cafd5cdb041c1bbe1592f04370c0b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:24.316015 16939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d2104698d5f452c8350cce10afd3898. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:24.316635 18492 tablet_bootstrap.cc:492] T 78f81fe4c6234ff0b5fcabd974779a66 P 16d437fe629643948c95d628926d5ffa: Bootstrap starting.
I20240503 02:00:24.317540 18492 tablet_bootstrap.cc:654] T 78f81fe4c6234ff0b5fcabd974779a66 P 16d437fe629643948c95d628926d5ffa: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:24.317849 18490 tablet_bootstrap.cc:492] T 5d2104698d5f452c8350cce10afd3898 P 6398c8993c3c45dcaab25b42142d5671: Bootstrap starting.
I20240503 02:00:24.318043 16938 tablet_service.cc:1467] Processing CreateTablet for tablet e245c9cf642043cfbb3b87114beeb002 (DEFAULT_TABLE table=rebalance_test_table_2 [id=9f4cafd5cdb041c1bbe1592f04370c0b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:24.318236 16938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e245c9cf642043cfbb3b87114beeb002. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:24.318570 18492 tablet_bootstrap.cc:492] T 78f81fe4c6234ff0b5fcabd974779a66 P 16d437fe629643948c95d628926d5ffa: No bootstrap required, opened a new log
I20240503 02:00:24.318620 18490 tablet_bootstrap.cc:654] T 5d2104698d5f452c8350cce10afd3898 P 6398c8993c3c45dcaab25b42142d5671: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:24.318645 18492 ts_tablet_manager.cc:1367] T 78f81fe4c6234ff0b5fcabd974779a66 P 16d437fe629643948c95d628926d5ffa: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:24.319041 18492 raft_consensus.cc:348] T 78f81fe4c6234ff0b5fcabd974779a66 P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:24.319157 18492 raft_consensus.cc:374] T 78f81fe4c6234ff0b5fcabd974779a66 P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:24.319201 18492 raft_consensus.cc:729] T 78f81fe4c6234ff0b5fcabd974779a66 P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16d437fe629643948c95d628926d5ffa, State: Initialized, Role: FOLLOWER
I20240503 02:00:24.319361 18492 consensus_queue.cc:260] T 78f81fe4c6234ff0b5fcabd974779a66 P 16d437fe629643948c95d628926d5ffa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:24.319572 18490 tablet_bootstrap.cc:492] T 5d2104698d5f452c8350cce10afd3898 P 6398c8993c3c45dcaab25b42142d5671: No bootstrap required, opened a new log
I20240503 02:00:24.319649 18490 ts_tablet_manager.cc:1367] T 5d2104698d5f452c8350cce10afd3898 P 6398c8993c3c45dcaab25b42142d5671: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:24.319741 18492 ts_tablet_manager.cc:1398] T 78f81fe4c6234ff0b5fcabd974779a66 P 16d437fe629643948c95d628926d5ffa: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:24.320263 18490 raft_consensus.cc:348] T 5d2104698d5f452c8350cce10afd3898 P 6398c8993c3c45dcaab25b42142d5671 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:24.320380 18490 raft_consensus.cc:374] T 5d2104698d5f452c8350cce10afd3898 P 6398c8993c3c45dcaab25b42142d5671 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:24.320325 17731 tablet_service.cc:1467] Processing CreateTablet for tablet 5d2104698d5f452c8350cce10afd3898 (DEFAULT_TABLE table=rebalance_test_table_2 [id=9f4cafd5cdb041c1bbe1592f04370c0b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:24.320420 18490 raft_consensus.cc:729] T 5d2104698d5f452c8350cce10afd3898 P 6398c8993c3c45dcaab25b42142d5671 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6398c8993c3c45dcaab25b42142d5671, State: Initialized, Role: FOLLOWER
I20240503 02:00:24.320622 17731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d2104698d5f452c8350cce10afd3898. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:24.320561 18490 consensus_queue.cc:260] T 5d2104698d5f452c8350cce10afd3898 P 6398c8993c3c45dcaab25b42142d5671 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:24.321000 17467 tablet_service.cc:1467] Processing CreateTablet for tablet e245c9cf642043cfbb3b87114beeb002 (DEFAULT_TABLE table=rebalance_test_table_2 [id=9f4cafd5cdb041c1bbe1592f04370c0b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:24.321097 18490 ts_tablet_manager.cc:1398] T 5d2104698d5f452c8350cce10afd3898 P 6398c8993c3c45dcaab25b42142d5671: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 02:00:24.321200 17467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e245c9cf642043cfbb3b87114beeb002. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:24.321218 18490 tablet_bootstrap.cc:492] T e245c9cf642043cfbb3b87114beeb002 P 6398c8993c3c45dcaab25b42142d5671: Bootstrap starting.
I20240503 02:00:24.321887 18490 tablet_bootstrap.cc:654] T e245c9cf642043cfbb3b87114beeb002 P 6398c8993c3c45dcaab25b42142d5671: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:24.322731 18490 tablet_bootstrap.cc:492] T e245c9cf642043cfbb3b87114beeb002 P 6398c8993c3c45dcaab25b42142d5671: No bootstrap required, opened a new log
I20240503 02:00:24.322803 18490 ts_tablet_manager.cc:1367] T e245c9cf642043cfbb3b87114beeb002 P 6398c8993c3c45dcaab25b42142d5671: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:24.322912 18496 tablet_bootstrap.cc:492] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6: Bootstrap starting.
I20240503 02:00:24.323192 18490 raft_consensus.cc:348] T e245c9cf642043cfbb3b87114beeb002 P 6398c8993c3c45dcaab25b42142d5671 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:24.323303 18492 tablet_bootstrap.cc:492] T e245c9cf642043cfbb3b87114beeb002 P 16d437fe629643948c95d628926d5ffa: Bootstrap starting.
I20240503 02:00:24.323325 18490 raft_consensus.cc:374] T e245c9cf642043cfbb3b87114beeb002 P 6398c8993c3c45dcaab25b42142d5671 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:24.323264 17732 tablet_service.cc:1467] Processing CreateTablet for tablet e245c9cf642043cfbb3b87114beeb002 (DEFAULT_TABLE table=rebalance_test_table_2 [id=9f4cafd5cdb041c1bbe1592f04370c0b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:24.323354 18490 raft_consensus.cc:729] T e245c9cf642043cfbb3b87114beeb002 P 6398c8993c3c45dcaab25b42142d5671 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6398c8993c3c45dcaab25b42142d5671, State: Initialized, Role: FOLLOWER
I20240503 02:00:24.323482 17732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e245c9cf642043cfbb3b87114beeb002. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:24.323465 18490 consensus_queue.cc:260] T e245c9cf642043cfbb3b87114beeb002 P 6398c8993c3c45dcaab25b42142d5671 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:24.323774 18496 tablet_bootstrap.cc:654] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:24.323817 18490 ts_tablet_manager.cc:1398] T e245c9cf642043cfbb3b87114beeb002 P 6398c8993c3c45dcaab25b42142d5671: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:24.324107 18492 tablet_bootstrap.cc:654] T e245c9cf642043cfbb3b87114beeb002 P 16d437fe629643948c95d628926d5ffa: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:24.324661 18496 tablet_bootstrap.cc:492] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6: No bootstrap required, opened a new log
I20240503 02:00:24.324752 18496 ts_tablet_manager.cc:1367] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:24.324910 17468 tablet_service.cc:1467] Processing CreateTablet for tablet 5d2104698d5f452c8350cce10afd3898 (DEFAULT_TABLE table=rebalance_test_table_2 [id=9f4cafd5cdb041c1bbe1592f04370c0b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:24.325078 18496 raft_consensus.cc:348] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:24.325130 17468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d2104698d5f452c8350cce10afd3898. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:24.325160 18496 raft_consensus.cc:374] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:24.325179 18496 raft_consensus.cc:729] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 002cc3411acb4fdfa0b67c6f57f687a6, State: Initialized, Role: FOLLOWER
I20240503 02:00:24.325284 18496 consensus_queue.cc:260] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:24.325623 17730 tablet_service.cc:1467] Processing CreateTablet for tablet 78f81fe4c6234ff0b5fcabd974779a66 (DEFAULT_TABLE table=rebalance_test_table_2 [id=9f4cafd5cdb041c1bbe1592f04370c0b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:24.325853 17730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78f81fe4c6234ff0b5fcabd974779a66. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:24.325878 18496 ts_tablet_manager.cc:1398] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:24.325999 18496 tablet_bootstrap.cc:492] T e245c9cf642043cfbb3b87114beeb002 P 002cc3411acb4fdfa0b67c6f57f687a6: Bootstrap starting.
I20240503 02:00:24.326623 18496 tablet_bootstrap.cc:654] T e245c9cf642043cfbb3b87114beeb002 P 002cc3411acb4fdfa0b67c6f57f687a6: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:24.327888 18496 tablet_bootstrap.cc:492] T e245c9cf642043cfbb3b87114beeb002 P 002cc3411acb4fdfa0b67c6f57f687a6: No bootstrap required, opened a new log
I20240503 02:00:24.327952 18496 ts_tablet_manager.cc:1367] T e245c9cf642043cfbb3b87114beeb002 P 002cc3411acb4fdfa0b67c6f57f687a6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:24.327977 17598 tablet_service.cc:1467] Processing CreateTablet for tablet 5d2104698d5f452c8350cce10afd3898 (DEFAULT_TABLE table=rebalance_test_table_2 [id=9f4cafd5cdb041c1bbe1592f04370c0b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:24.328197 17598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d2104698d5f452c8350cce10afd3898. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:24.328245 18496 raft_consensus.cc:348] T e245c9cf642043cfbb3b87114beeb002 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:24.328317 18496 raft_consensus.cc:374] T e245c9cf642043cfbb3b87114beeb002 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:24.328336 18496 raft_consensus.cc:729] T e245c9cf642043cfbb3b87114beeb002 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 002cc3411acb4fdfa0b67c6f57f687a6, State: Initialized, Role: FOLLOWER
I20240503 02:00:24.328434 18496 consensus_queue.cc:260] T e245c9cf642043cfbb3b87114beeb002 P 002cc3411acb4fdfa0b67c6f57f687a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:24.328671 18496 ts_tablet_manager.cc:1398] T e245c9cf642043cfbb3b87114beeb002 P 002cc3411acb4fdfa0b67c6f57f687a6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:24.328810 18496 tablet_bootstrap.cc:492] T 78f81fe4c6234ff0b5fcabd974779a66 P 002cc3411acb4fdfa0b67c6f57f687a6: Bootstrap starting.
I20240503 02:00:24.328919 17599 tablet_service.cc:1467] Processing CreateTablet for tablet 78f81fe4c6234ff0b5fcabd974779a66 (DEFAULT_TABLE table=rebalance_test_table_2 [id=9f4cafd5cdb041c1bbe1592f04370c0b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:24.329140 17599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78f81fe4c6234ff0b5fcabd974779a66. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:24.329421 18496 tablet_bootstrap.cc:654] T 78f81fe4c6234ff0b5fcabd974779a66 P 002cc3411acb4fdfa0b67c6f57f687a6: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:24.330266 18496 tablet_bootstrap.cc:492] T 78f81fe4c6234ff0b5fcabd974779a66 P 002cc3411acb4fdfa0b67c6f57f687a6: No bootstrap required, opened a new log
I20240503 02:00:24.330339 18496 ts_tablet_manager.cc:1367] T 78f81fe4c6234ff0b5fcabd974779a66 P 002cc3411acb4fdfa0b67c6f57f687a6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:24.330365 18501 tablet_bootstrap.cc:492] T 5d2104698d5f452c8350cce10afd3898 P c843193b801b4f1a88f630ed3ed0f375: Bootstrap starting.
I20240503 02:00:24.330459 17600 tablet_service.cc:1467] Processing CreateTablet for tablet e245c9cf642043cfbb3b87114beeb002 (DEFAULT_TABLE table=rebalance_test_table_2 [id=9f4cafd5cdb041c1bbe1592f04370c0b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:24.330706 17600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e245c9cf642043cfbb3b87114beeb002. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:24.330746 18496 raft_consensus.cc:348] T 78f81fe4c6234ff0b5fcabd974779a66 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:24.330859 18496 raft_consensus.cc:374] T 78f81fe4c6234ff0b5fcabd974779a66 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:24.330896 18496 raft_consensus.cc:729] T 78f81fe4c6234ff0b5fcabd974779a66 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 002cc3411acb4fdfa0b67c6f57f687a6, State: Initialized, Role: FOLLOWER
I20240503 02:00:24.331028 18496 consensus_queue.cc:260] T 78f81fe4c6234ff0b5fcabd974779a66 P 002cc3411acb4fdfa0b67c6f57f687a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:24.331312 18496 ts_tablet_manager.cc:1398] T 78f81fe4c6234ff0b5fcabd974779a66 P 002cc3411acb4fdfa0b67c6f57f687a6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:24.331384 18501 tablet_bootstrap.cc:654] T 5d2104698d5f452c8350cce10afd3898 P c843193b801b4f1a88f630ed3ed0f375: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:24.332499 18501 tablet_bootstrap.cc:492] T 5d2104698d5f452c8350cce10afd3898 P c843193b801b4f1a88f630ed3ed0f375: No bootstrap required, opened a new log
I20240503 02:00:24.332576 18501 ts_tablet_manager.cc:1367] T 5d2104698d5f452c8350cce10afd3898 P c843193b801b4f1a88f630ed3ed0f375: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:24.333400 17203 tablet_service.cc:1467] Processing CreateTablet for tablet 78f81fe4c6234ff0b5fcabd974779a66 (DEFAULT_TABLE table=rebalance_test_table_2 [id=9f4cafd5cdb041c1bbe1592f04370c0b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:24.333393 17335 tablet_service.cc:1467] Processing CreateTablet for tablet 78f81fe4c6234ff0b5fcabd974779a66 (DEFAULT_TABLE table=rebalance_test_table_2 [id=9f4cafd5cdb041c1bbe1592f04370c0b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:24.333439 17204 tablet_service.cc:1467] Processing CreateTablet for tablet 5d2104698d5f452c8350cce10afd3898 (DEFAULT_TABLE table=rebalance_test_table_2 [id=9f4cafd5cdb041c1bbe1592f04370c0b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:24.333671 17203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78f81fe4c6234ff0b5fcabd974779a66. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:24.333671 17335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78f81fe4c6234ff0b5fcabd974779a66. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:24.334873 18492 tablet_bootstrap.cc:492] T e245c9cf642043cfbb3b87114beeb002 P 16d437fe629643948c95d628926d5ffa: No bootstrap required, opened a new log
I20240503 02:00:24.334973 18492 ts_tablet_manager.cc:1367] T e245c9cf642043cfbb3b87114beeb002 P 16d437fe629643948c95d628926d5ffa: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240503 02:00:24.335462 18492 raft_consensus.cc:348] T e245c9cf642043cfbb3b87114beeb002 P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:24.335626 18492 raft_consensus.cc:374] T e245c9cf642043cfbb3b87114beeb002 P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:24.335664 18492 raft_consensus.cc:729] T e245c9cf642043cfbb3b87114beeb002 P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16d437fe629643948c95d628926d5ffa, State: Initialized, Role: FOLLOWER
I20240503 02:00:24.335772 18492 consensus_queue.cc:260] T e245c9cf642043cfbb3b87114beeb002 P 16d437fe629643948c95d628926d5ffa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:24.335896 17336 tablet_service.cc:1467] Processing CreateTablet for tablet e245c9cf642043cfbb3b87114beeb002 (DEFAULT_TABLE table=rebalance_test_table_2 [id=9f4cafd5cdb041c1bbe1592f04370c0b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:24.336050 18492 ts_tablet_manager.cc:1398] T e245c9cf642043cfbb3b87114beeb002 P 16d437fe629643948c95d628926d5ffa: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:24.336123 17336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e245c9cf642043cfbb3b87114beeb002. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:24.336205 18492 tablet_bootstrap.cc:492] T 5d2104698d5f452c8350cce10afd3898 P 16d437fe629643948c95d628926d5ffa: Bootstrap starting.
I20240503 02:00:24.337126 18492 tablet_bootstrap.cc:654] T 5d2104698d5f452c8350cce10afd3898 P 16d437fe629643948c95d628926d5ffa: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:24.337918 18489 tablet_bootstrap.cc:492] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67: Bootstrap starting.
I20240503 02:00:24.338132 17204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d2104698d5f452c8350cce10afd3898. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:24.338217 17202 tablet_service.cc:1467] Processing CreateTablet for tablet e245c9cf642043cfbb3b87114beeb002 (DEFAULT_TABLE table=rebalance_test_table_2 [id=9f4cafd5cdb041c1bbe1592f04370c0b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:24.338441 17202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e245c9cf642043cfbb3b87114beeb002. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:24.339040 18489 tablet_bootstrap.cc:654] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:24.340183 17334 tablet_service.cc:1467] Processing CreateTablet for tablet 5d2104698d5f452c8350cce10afd3898 (DEFAULT_TABLE table=rebalance_test_table_2 [id=9f4cafd5cdb041c1bbe1592f04370c0b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:24.340445 17334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d2104698d5f452c8350cce10afd3898. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:24.341320 17071 tablet_service.cc:1467] Processing CreateTablet for tablet 5d2104698d5f452c8350cce10afd3898 (DEFAULT_TABLE table=rebalance_test_table_2 [id=9f4cafd5cdb041c1bbe1592f04370c0b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:24.341603 17071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d2104698d5f452c8350cce10afd3898. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:24.341711 18501 raft_consensus.cc:348] T 5d2104698d5f452c8350cce10afd3898 P c843193b801b4f1a88f630ed3ed0f375 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:24.341842 18501 raft_consensus.cc:374] T 5d2104698d5f452c8350cce10afd3898 P c843193b801b4f1a88f630ed3ed0f375 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:24.341886 18501 raft_consensus.cc:729] T 5d2104698d5f452c8350cce10afd3898 P c843193b801b4f1a88f630ed3ed0f375 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c843193b801b4f1a88f630ed3ed0f375, State: Initialized, Role: FOLLOWER
I20240503 02:00:24.342022 18501 consensus_queue.cc:260] T 5d2104698d5f452c8350cce10afd3898 P c843193b801b4f1a88f630ed3ed0f375 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:24.342428 17072 tablet_service.cc:1467] Processing CreateTablet for tablet e245c9cf642043cfbb3b87114beeb002 (DEFAULT_TABLE table=rebalance_test_table_2 [id=9f4cafd5cdb041c1bbe1592f04370c0b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:24.342618 17072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e245c9cf642043cfbb3b87114beeb002. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:24.343036 18501 ts_tablet_manager.cc:1398] T 5d2104698d5f452c8350cce10afd3898 P c843193b801b4f1a88f630ed3ed0f375: Time spent starting tablet: real 0.010s user 0.001s sys 0.000s
I20240503 02:00:24.343204 18501 tablet_bootstrap.cc:492] T 78f81fe4c6234ff0b5fcabd974779a66 P c843193b801b4f1a88f630ed3ed0f375: Bootstrap starting.
I20240503 02:00:24.344305 18501 tablet_bootstrap.cc:654] T 78f81fe4c6234ff0b5fcabd974779a66 P c843193b801b4f1a88f630ed3ed0f375: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:24.344902 18503 tablet_bootstrap.cc:492] T 78f81fe4c6234ff0b5fcabd974779a66 P d43ed6b8663947d99a72cd922cc323aa: Bootstrap starting.
I20240503 02:00:24.345870 18503 tablet_bootstrap.cc:654] T 78f81fe4c6234ff0b5fcabd974779a66 P d43ed6b8663947d99a72cd922cc323aa: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:24.345952 18501 tablet_bootstrap.cc:492] T 78f81fe4c6234ff0b5fcabd974779a66 P c843193b801b4f1a88f630ed3ed0f375: No bootstrap required, opened a new log
I20240503 02:00:24.346045 18501 ts_tablet_manager.cc:1367] T 78f81fe4c6234ff0b5fcabd974779a66 P c843193b801b4f1a88f630ed3ed0f375: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:24.346513 18501 raft_consensus.cc:348] T 78f81fe4c6234ff0b5fcabd974779a66 P c843193b801b4f1a88f630ed3ed0f375 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:24.346617 18505 tablet_bootstrap.cc:492] T 78f81fe4c6234ff0b5fcabd974779a66 P 5b8291b26a7f448b8636187d1caf86d0: Bootstrap starting.
I20240503 02:00:24.346633 18501 raft_consensus.cc:374] T 78f81fe4c6234ff0b5fcabd974779a66 P c843193b801b4f1a88f630ed3ed0f375 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:24.346668 18501 raft_consensus.cc:729] T 78f81fe4c6234ff0b5fcabd974779a66 P c843193b801b4f1a88f630ed3ed0f375 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c843193b801b4f1a88f630ed3ed0f375, State: Initialized, Role: FOLLOWER
I20240503 02:00:24.346830 18501 consensus_queue.cc:260] T 78f81fe4c6234ff0b5fcabd974779a66 P c843193b801b4f1a88f630ed3ed0f375 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:24.347028 18503 tablet_bootstrap.cc:492] T 78f81fe4c6234ff0b5fcabd974779a66 P d43ed6b8663947d99a72cd922cc323aa: No bootstrap required, opened a new log
I20240503 02:00:24.347115 18503 ts_tablet_manager.cc:1367] T 78f81fe4c6234ff0b5fcabd974779a66 P d43ed6b8663947d99a72cd922cc323aa: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:24.347193 18501 ts_tablet_manager.cc:1398] T 78f81fe4c6234ff0b5fcabd974779a66 P c843193b801b4f1a88f630ed3ed0f375: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:24.347352 18501 tablet_bootstrap.cc:492] T e245c9cf642043cfbb3b87114beeb002 P c843193b801b4f1a88f630ed3ed0f375: Bootstrap starting.
I20240503 02:00:24.347486 18505 tablet_bootstrap.cc:654] T 78f81fe4c6234ff0b5fcabd974779a66 P 5b8291b26a7f448b8636187d1caf86d0: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:24.347571 18503 raft_consensus.cc:348] T 78f81fe4c6234ff0b5fcabd974779a66 P d43ed6b8663947d99a72cd922cc323aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:24.347697 18503 raft_consensus.cc:374] T 78f81fe4c6234ff0b5fcabd974779a66 P d43ed6b8663947d99a72cd922cc323aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:24.347750 18503 raft_consensus.cc:729] T 78f81fe4c6234ff0b5fcabd974779a66 P d43ed6b8663947d99a72cd922cc323aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d43ed6b8663947d99a72cd922cc323aa, State: Initialized, Role: FOLLOWER
I20240503 02:00:24.347901 18503 consensus_queue.cc:260] T 78f81fe4c6234ff0b5fcabd974779a66 P d43ed6b8663947d99a72cd922cc323aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:24.348227 18501 tablet_bootstrap.cc:654] T e245c9cf642043cfbb3b87114beeb002 P c843193b801b4f1a88f630ed3ed0f375: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:24.348294 18503 ts_tablet_manager.cc:1398] T 78f81fe4c6234ff0b5fcabd974779a66 P d43ed6b8663947d99a72cd922cc323aa: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:24.348485 18503 tablet_bootstrap.cc:492] T e245c9cf642043cfbb3b87114beeb002 P d43ed6b8663947d99a72cd922cc323aa: Bootstrap starting.
I20240503 02:00:24.349387 18492 tablet_bootstrap.cc:492] T 5d2104698d5f452c8350cce10afd3898 P 16d437fe629643948c95d628926d5ffa: No bootstrap required, opened a new log
I20240503 02:00:24.349421 18503 tablet_bootstrap.cc:654] T e245c9cf642043cfbb3b87114beeb002 P d43ed6b8663947d99a72cd922cc323aa: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:24.349498 18492 ts_tablet_manager.cc:1367] T 5d2104698d5f452c8350cce10afd3898 P 16d437fe629643948c95d628926d5ffa: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240503 02:00:24.350204 18492 raft_consensus.cc:348] T 5d2104698d5f452c8350cce10afd3898 P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:24.350344 18492 raft_consensus.cc:374] T 5d2104698d5f452c8350cce10afd3898 P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:24.350379 18492 raft_consensus.cc:729] T 5d2104698d5f452c8350cce10afd3898 P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16d437fe629643948c95d628926d5ffa, State: Initialized, Role: FOLLOWER
I20240503 02:00:24.350400 18503 tablet_bootstrap.cc:492] T e245c9cf642043cfbb3b87114beeb002 P d43ed6b8663947d99a72cd922cc323aa: No bootstrap required, opened a new log
I20240503 02:00:24.350476 18503 ts_tablet_manager.cc:1367] T e245c9cf642043cfbb3b87114beeb002 P d43ed6b8663947d99a72cd922cc323aa: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:24.350523 18492 consensus_queue.cc:260] T 5d2104698d5f452c8350cce10afd3898 P 16d437fe629643948c95d628926d5ffa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:24.350805 18492 ts_tablet_manager.cc:1398] T 5d2104698d5f452c8350cce10afd3898 P 16d437fe629643948c95d628926d5ffa: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:24.350915 18503 raft_consensus.cc:348] T e245c9cf642043cfbb3b87114beeb002 P d43ed6b8663947d99a72cd922cc323aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:24.351022 18503 raft_consensus.cc:374] T e245c9cf642043cfbb3b87114beeb002 P d43ed6b8663947d99a72cd922cc323aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:24.351052 18503 raft_consensus.cc:729] T e245c9cf642043cfbb3b87114beeb002 P d43ed6b8663947d99a72cd922cc323aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d43ed6b8663947d99a72cd922cc323aa, State: Initialized, Role: FOLLOWER
I20240503 02:00:24.351154 18503 consensus_queue.cc:260] T e245c9cf642043cfbb3b87114beeb002 P d43ed6b8663947d99a72cd922cc323aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:24.351411 18503 ts_tablet_manager.cc:1398] T e245c9cf642043cfbb3b87114beeb002 P d43ed6b8663947d99a72cd922cc323aa: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:24.351503 18489 tablet_bootstrap.cc:492] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67: No bootstrap required, opened a new log
I20240503 02:00:24.351554 18503 tablet_bootstrap.cc:492] T 5d2104698d5f452c8350cce10afd3898 P d43ed6b8663947d99a72cd922cc323aa: Bootstrap starting.
I20240503 02:00:24.351627 18489 ts_tablet_manager.cc:1367] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240503 02:00:24.352090 18489 raft_consensus.cc:348] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:24.352247 18489 raft_consensus.cc:374] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:24.352289 18489 raft_consensus.cc:729] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89dd09f8aaee4b249ae3ce924a422a67, State: Initialized, Role: FOLLOWER
I20240503 02:00:24.352427 18489 consensus_queue.cc:260] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:24.352557 18503 tablet_bootstrap.cc:654] T 5d2104698d5f452c8350cce10afd3898 P d43ed6b8663947d99a72cd922cc323aa: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:24.352747 18489 ts_tablet_manager.cc:1398] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 02:00:24.352926 18489 tablet_bootstrap.cc:492] T 5d2104698d5f452c8350cce10afd3898 P 89dd09f8aaee4b249ae3ce924a422a67: Bootstrap starting.
I20240503 02:00:24.353843 18489 tablet_bootstrap.cc:654] T 5d2104698d5f452c8350cce10afd3898 P 89dd09f8aaee4b249ae3ce924a422a67: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:24.354683 18503 tablet_bootstrap.cc:492] T 5d2104698d5f452c8350cce10afd3898 P d43ed6b8663947d99a72cd922cc323aa: No bootstrap required, opened a new log
I20240503 02:00:24.354777 18503 ts_tablet_manager.cc:1367] T 5d2104698d5f452c8350cce10afd3898 P d43ed6b8663947d99a72cd922cc323aa: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:24.355176 18503 raft_consensus.cc:348] T 5d2104698d5f452c8350cce10afd3898 P d43ed6b8663947d99a72cd922cc323aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:24.355286 18503 raft_consensus.cc:374] T 5d2104698d5f452c8350cce10afd3898 P d43ed6b8663947d99a72cd922cc323aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:24.355319 18503 raft_consensus.cc:729] T 5d2104698d5f452c8350cce10afd3898 P d43ed6b8663947d99a72cd922cc323aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d43ed6b8663947d99a72cd922cc323aa, State: Initialized, Role: FOLLOWER
I20240503 02:00:24.355425 18503 consensus_queue.cc:260] T 5d2104698d5f452c8350cce10afd3898 P d43ed6b8663947d99a72cd922cc323aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:24.355696 18503 ts_tablet_manager.cc:1398] T 5d2104698d5f452c8350cce10afd3898 P d43ed6b8663947d99a72cd922cc323aa: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:24.355770 18501 tablet_bootstrap.cc:492] T e245c9cf642043cfbb3b87114beeb002 P c843193b801b4f1a88f630ed3ed0f375: No bootstrap required, opened a new log
I20240503 02:00:24.355868 18501 ts_tablet_manager.cc:1367] T e245c9cf642043cfbb3b87114beeb002 P c843193b801b4f1a88f630ed3ed0f375: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20240503 02:00:24.356274 18501 raft_consensus.cc:348] T e245c9cf642043cfbb3b87114beeb002 P c843193b801b4f1a88f630ed3ed0f375 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:24.356403 18501 raft_consensus.cc:374] T e245c9cf642043cfbb3b87114beeb002 P c843193b801b4f1a88f630ed3ed0f375 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:24.356469 18501 raft_consensus.cc:729] T e245c9cf642043cfbb3b87114beeb002 P c843193b801b4f1a88f630ed3ed0f375 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c843193b801b4f1a88f630ed3ed0f375, State: Initialized, Role: FOLLOWER
I20240503 02:00:24.356797 18505 tablet_bootstrap.cc:492] T 78f81fe4c6234ff0b5fcabd974779a66 P 5b8291b26a7f448b8636187d1caf86d0: No bootstrap required, opened a new log
I20240503 02:00:24.356638 18501 consensus_queue.cc:260] T e245c9cf642043cfbb3b87114beeb002 P c843193b801b4f1a88f630ed3ed0f375 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:24.357095 18505 ts_tablet_manager.cc:1367] T 78f81fe4c6234ff0b5fcabd974779a66 P 5b8291b26a7f448b8636187d1caf86d0: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240503 02:00:24.357488 18501 ts_tablet_manager.cc:1398] T e245c9cf642043cfbb3b87114beeb002 P c843193b801b4f1a88f630ed3ed0f375: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:24.357679 18505 raft_consensus.cc:348] T 78f81fe4c6234ff0b5fcabd974779a66 P 5b8291b26a7f448b8636187d1caf86d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:24.357896 18505 raft_consensus.cc:374] T 78f81fe4c6234ff0b5fcabd974779a66 P 5b8291b26a7f448b8636187d1caf86d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:24.357991 18505 raft_consensus.cc:729] T 78f81fe4c6234ff0b5fcabd974779a66 P 5b8291b26a7f448b8636187d1caf86d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b8291b26a7f448b8636187d1caf86d0, State: Initialized, Role: FOLLOWER
I20240503 02:00:24.358212 18489 tablet_bootstrap.cc:492] T 5d2104698d5f452c8350cce10afd3898 P 89dd09f8aaee4b249ae3ce924a422a67: No bootstrap required, opened a new log
I20240503 02:00:24.358321 18489 ts_tablet_manager.cc:1367] T 5d2104698d5f452c8350cce10afd3898 P 89dd09f8aaee4b249ae3ce924a422a67: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240503 02:00:24.358222 18505 consensus_queue.cc:260] T 78f81fe4c6234ff0b5fcabd974779a66 P 5b8291b26a7f448b8636187d1caf86d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:24.358783 18489 raft_consensus.cc:348] T 5d2104698d5f452c8350cce10afd3898 P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:24.358902 18489 raft_consensus.cc:374] T 5d2104698d5f452c8350cce10afd3898 P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:24.358942 18489 raft_consensus.cc:729] T 5d2104698d5f452c8350cce10afd3898 P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89dd09f8aaee4b249ae3ce924a422a67, State: Initialized, Role: FOLLOWER
I20240503 02:00:24.359074 18489 consensus_queue.cc:260] T 5d2104698d5f452c8350cce10afd3898 P 89dd09f8aaee4b249ae3ce924a422a67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:24.359433 18489 ts_tablet_manager.cc:1398] T 5d2104698d5f452c8350cce10afd3898 P 89dd09f8aaee4b249ae3ce924a422a67: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:24.359457 18505 ts_tablet_manager.cc:1398] T 78f81fe4c6234ff0b5fcabd974779a66 P 5b8291b26a7f448b8636187d1caf86d0: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:24.359624 18489 tablet_bootstrap.cc:492] T e245c9cf642043cfbb3b87114beeb002 P 89dd09f8aaee4b249ae3ce924a422a67: Bootstrap starting.
I20240503 02:00:24.359764 18505 tablet_bootstrap.cc:492] T 5d2104698d5f452c8350cce10afd3898 P 5b8291b26a7f448b8636187d1caf86d0: Bootstrap starting.
I20240503 02:00:24.360589 18505 tablet_bootstrap.cc:654] T 5d2104698d5f452c8350cce10afd3898 P 5b8291b26a7f448b8636187d1caf86d0: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:24.360613 18489 tablet_bootstrap.cc:654] T e245c9cf642043cfbb3b87114beeb002 P 89dd09f8aaee4b249ae3ce924a422a67: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:24.361639 18505 tablet_bootstrap.cc:492] T 5d2104698d5f452c8350cce10afd3898 P 5b8291b26a7f448b8636187d1caf86d0: No bootstrap required, opened a new log
I20240503 02:00:24.361732 18505 ts_tablet_manager.cc:1367] T 5d2104698d5f452c8350cce10afd3898 P 5b8291b26a7f448b8636187d1caf86d0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:24.362514 18489 tablet_bootstrap.cc:492] T e245c9cf642043cfbb3b87114beeb002 P 89dd09f8aaee4b249ae3ce924a422a67: No bootstrap required, opened a new log
I20240503 02:00:24.362578 18489 ts_tablet_manager.cc:1367] T e245c9cf642043cfbb3b87114beeb002 P 89dd09f8aaee4b249ae3ce924a422a67: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:24.362545 18505 raft_consensus.cc:348] T 5d2104698d5f452c8350cce10afd3898 P 5b8291b26a7f448b8636187d1caf86d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:24.362656 18505 raft_consensus.cc:374] T 5d2104698d5f452c8350cce10afd3898 P 5b8291b26a7f448b8636187d1caf86d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:24.362692 18505 raft_consensus.cc:729] T 5d2104698d5f452c8350cce10afd3898 P 5b8291b26a7f448b8636187d1caf86d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b8291b26a7f448b8636187d1caf86d0, State: Initialized, Role: FOLLOWER
I20240503 02:00:24.362809 18505 consensus_queue.cc:260] T 5d2104698d5f452c8350cce10afd3898 P 5b8291b26a7f448b8636187d1caf86d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:24.362905 18489 raft_consensus.cc:348] T e245c9cf642043cfbb3b87114beeb002 P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:24.363023 18489 raft_consensus.cc:374] T e245c9cf642043cfbb3b87114beeb002 P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:24.363065 18489 raft_consensus.cc:729] T e245c9cf642043cfbb3b87114beeb002 P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89dd09f8aaee4b249ae3ce924a422a67, State: Initialized, Role: FOLLOWER
I20240503 02:00:24.363118 18505 ts_tablet_manager.cc:1398] T 5d2104698d5f452c8350cce10afd3898 P 5b8291b26a7f448b8636187d1caf86d0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:24.363209 18489 consensus_queue.cc:260] T e245c9cf642043cfbb3b87114beeb002 P 89dd09f8aaee4b249ae3ce924a422a67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:24.363436 18505 tablet_bootstrap.cc:492] T e245c9cf642043cfbb3b87114beeb002 P 5b8291b26a7f448b8636187d1caf86d0: Bootstrap starting.
I20240503 02:00:24.363806 18489 ts_tablet_manager.cc:1398] T e245c9cf642043cfbb3b87114beeb002 P 89dd09f8aaee4b249ae3ce924a422a67: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:24.364606 18505 tablet_bootstrap.cc:654] T e245c9cf642043cfbb3b87114beeb002 P 5b8291b26a7f448b8636187d1caf86d0: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:24.365818 18505 tablet_bootstrap.cc:492] T e245c9cf642043cfbb3b87114beeb002 P 5b8291b26a7f448b8636187d1caf86d0: No bootstrap required, opened a new log
I20240503 02:00:24.365947 18505 ts_tablet_manager.cc:1367] T e245c9cf642043cfbb3b87114beeb002 P 5b8291b26a7f448b8636187d1caf86d0: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:24.366511 18505 raft_consensus.cc:348] T e245c9cf642043cfbb3b87114beeb002 P 5b8291b26a7f448b8636187d1caf86d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:24.366642 18505 raft_consensus.cc:374] T e245c9cf642043cfbb3b87114beeb002 P 5b8291b26a7f448b8636187d1caf86d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:24.366677 18505 raft_consensus.cc:729] T e245c9cf642043cfbb3b87114beeb002 P 5b8291b26a7f448b8636187d1caf86d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b8291b26a7f448b8636187d1caf86d0, State: Initialized, Role: FOLLOWER
I20240503 02:00:24.366797 18505 consensus_queue.cc:260] T e245c9cf642043cfbb3b87114beeb002 P 5b8291b26a7f448b8636187d1caf86d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:24.367003 18505 ts_tablet_manager.cc:1398] T e245c9cf642043cfbb3b87114beeb002 P 5b8291b26a7f448b8636187d1caf86d0: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:24.371429 18445 raft_consensus.cc:482] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:24.371515 18445 raft_consensus.cc:504] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:24.372208 18445 leader_election.cc:288] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6398c8993c3c45dcaab25b42142d5671 (127.0.99.129:36983), 5b8291b26a7f448b8636187d1caf86d0 (127.0.99.131:44479), d43ed6b8663947d99a72cd922cc323aa (127.0.99.132:40023), 16d437fe629643948c95d628926d5ffa (127.0.99.133:46581), c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134:44343), 002cc3411acb4fdfa0b67c6f57f687a6 (127.0.99.135:38989)
I20240503 02:00:24.372462 17354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78f81fe4c6234ff0b5fcabd974779a66" candidate_uuid: "89dd09f8aaee4b249ae3ce924a422a67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d43ed6b8663947d99a72cd922cc323aa" is_pre_election: true
I20240503 02:00:24.372471 16959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78f81fe4c6234ff0b5fcabd974779a66" candidate_uuid: "89dd09f8aaee4b249ae3ce924a422a67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6398c8993c3c45dcaab25b42142d5671" is_pre_election: true
I20240503 02:00:24.372566 17223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78f81fe4c6234ff0b5fcabd974779a66" candidate_uuid: "89dd09f8aaee4b249ae3ce924a422a67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b8291b26a7f448b8636187d1caf86d0" is_pre_election: true
I20240503 02:00:24.372622 17354 raft_consensus.cc:2454] T 78f81fe4c6234ff0b5fcabd974779a66 P d43ed6b8663947d99a72cd922cc323aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89dd09f8aaee4b249ae3ce924a422a67 in term 0.
I20240503 02:00:24.372696 17223 raft_consensus.cc:2454] T 78f81fe4c6234ff0b5fcabd974779a66 P 5b8291b26a7f448b8636187d1caf86d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89dd09f8aaee4b249ae3ce924a422a67 in term 0.
I20240503 02:00:24.372645 16959 raft_consensus.cc:2454] T 78f81fe4c6234ff0b5fcabd974779a66 P 6398c8993c3c45dcaab25b42142d5671 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89dd09f8aaee4b249ae3ce924a422a67 in term 0.
I20240503 02:00:24.373059 17750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78f81fe4c6234ff0b5fcabd974779a66" candidate_uuid: "89dd09f8aaee4b249ae3ce924a422a67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" is_pre_election: true
I20240503 02:00:24.373060 17486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78f81fe4c6234ff0b5fcabd974779a66" candidate_uuid: "89dd09f8aaee4b249ae3ce924a422a67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16d437fe629643948c95d628926d5ffa" is_pre_election: true
I20240503 02:00:24.373186 17486 raft_consensus.cc:2454] T 78f81fe4c6234ff0b5fcabd974779a66 P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89dd09f8aaee4b249ae3ce924a422a67 in term 0.
I20240503 02:00:24.373198 17750 raft_consensus.cc:2454] T 78f81fe4c6234ff0b5fcabd974779a66 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89dd09f8aaee4b249ae3ce924a422a67 in term 0.
I20240503 02:00:24.373420 17027 leader_election.cc:302] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 5b8291b26a7f448b8636187d1caf86d0, 6398c8993c3c45dcaab25b42142d5671, 89dd09f8aaee4b249ae3ce924a422a67, d43ed6b8663947d99a72cd922cc323aa; no voters:
I20240503 02:00:24.373433 17620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78f81fe4c6234ff0b5fcabd974779a66" candidate_uuid: "89dd09f8aaee4b249ae3ce924a422a67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c843193b801b4f1a88f630ed3ed0f375" is_pre_election: true
I20240503 02:00:24.373559 17620 raft_consensus.cc:2454] T 78f81fe4c6234ff0b5fcabd974779a66 P c843193b801b4f1a88f630ed3ed0f375 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89dd09f8aaee4b249ae3ce924a422a67 in term 0.
I20240503 02:00:24.373592 18445 raft_consensus.cc:2781] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:24.373652 18445 raft_consensus.cc:482] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:24.373687 18445 raft_consensus.cc:3037] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:24.374470 18445 raft_consensus.cc:504] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:24.374976 18445 leader_election.cc:288] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 [CANDIDATE]: Term 1 election: Requested vote from peers 6398c8993c3c45dcaab25b42142d5671 (127.0.99.129:36983), 5b8291b26a7f448b8636187d1caf86d0 (127.0.99.131:44479), d43ed6b8663947d99a72cd922cc323aa (127.0.99.132:40023), 16d437fe629643948c95d628926d5ffa (127.0.99.133:46581), c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134:44343), 002cc3411acb4fdfa0b67c6f57f687a6 (127.0.99.135:38989)
I20240503 02:00:24.375137 16959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78f81fe4c6234ff0b5fcabd974779a66" candidate_uuid: "89dd09f8aaee4b249ae3ce924a422a67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6398c8993c3c45dcaab25b42142d5671"
I20240503 02:00:24.375185 17354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78f81fe4c6234ff0b5fcabd974779a66" candidate_uuid: "89dd09f8aaee4b249ae3ce924a422a67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d43ed6b8663947d99a72cd922cc323aa"
I20240503 02:00:24.375241 16959 raft_consensus.cc:3037] T 78f81fe4c6234ff0b5fcabd974779a66 P 6398c8993c3c45dcaab25b42142d5671 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:24.375291 17354 raft_consensus.cc:3037] T 78f81fe4c6234ff0b5fcabd974779a66 P d43ed6b8663947d99a72cd922cc323aa [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:24.375295 17223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78f81fe4c6234ff0b5fcabd974779a66" candidate_uuid: "89dd09f8aaee4b249ae3ce924a422a67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b8291b26a7f448b8636187d1caf86d0"
I20240503 02:00:24.375380 17223 raft_consensus.cc:3037] T 78f81fe4c6234ff0b5fcabd974779a66 P 5b8291b26a7f448b8636187d1caf86d0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:24.375741 17620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78f81fe4c6234ff0b5fcabd974779a66" candidate_uuid: "89dd09f8aaee4b249ae3ce924a422a67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c843193b801b4f1a88f630ed3ed0f375"
I20240503 02:00:24.375840 17620 raft_consensus.cc:3037] T 78f81fe4c6234ff0b5fcabd974779a66 P c843193b801b4f1a88f630ed3ed0f375 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:24.376263 16959 raft_consensus.cc:2454] T 78f81fe4c6234ff0b5fcabd974779a66 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89dd09f8aaee4b249ae3ce924a422a67 in term 1.
I20240503 02:00:24.376267 17223 raft_consensus.cc:2454] T 78f81fe4c6234ff0b5fcabd974779a66 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89dd09f8aaee4b249ae3ce924a422a67 in term 1.
I20240503 02:00:24.376267 17354 raft_consensus.cc:2454] T 78f81fe4c6234ff0b5fcabd974779a66 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89dd09f8aaee4b249ae3ce924a422a67 in term 1.
I20240503 02:00:24.376638 17486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78f81fe4c6234ff0b5fcabd974779a66" candidate_uuid: "89dd09f8aaee4b249ae3ce924a422a67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16d437fe629643948c95d628926d5ffa"
I20240503 02:00:24.376792 17486 raft_consensus.cc:3037] T 78f81fe4c6234ff0b5fcabd974779a66 P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:24.376837 17620 raft_consensus.cc:2454] T 78f81fe4c6234ff0b5fcabd974779a66 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89dd09f8aaee4b249ae3ce924a422a67 in term 1.
I20240503 02:00:24.376848 17027 leader_election.cc:302] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 5b8291b26a7f448b8636187d1caf86d0, 6398c8993c3c45dcaab25b42142d5671, 89dd09f8aaee4b249ae3ce924a422a67, d43ed6b8663947d99a72cd922cc323aa; no voters:
I20240503 02:00:24.377003 18445 raft_consensus.cc:2781] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:24.377089 18445 raft_consensus.cc:686] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 LEADER]: Becoming Leader. State: Replica: 89dd09f8aaee4b249ae3ce924a422a67, State: Running, Role: LEADER
I20240503 02:00:24.377076 17750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78f81fe4c6234ff0b5fcabd974779a66" candidate_uuid: "89dd09f8aaee4b249ae3ce924a422a67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "002cc3411acb4fdfa0b67c6f57f687a6"
I20240503 02:00:24.377208 17750 raft_consensus.cc:3037] T 78f81fe4c6234ff0b5fcabd974779a66 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:24.377240 18445 consensus_queue.cc:237] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:24.377640 17486 raft_consensus.cc:2454] T 78f81fe4c6234ff0b5fcabd974779a66 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89dd09f8aaee4b249ae3ce924a422a67 in term 1.
I20240503 02:00:24.378206 17750 raft_consensus.cc:2454] T 78f81fe4c6234ff0b5fcabd974779a66 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89dd09f8aaee4b249ae3ce924a422a67 in term 1.
I20240503 02:00:24.379423 16813 catalog_manager.cc:5484] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 reported cstate change: term changed from 0 to 1, leader changed from <none> to 89dd09f8aaee4b249ae3ce924a422a67 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "89dd09f8aaee4b249ae3ce924a422a67" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:24.401290 18362 raft_consensus.cc:482] T e245c9cf642043cfbb3b87114beeb002 P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:24.401433 18362 raft_consensus.cc:504] T e245c9cf642043cfbb3b87114beeb002 P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:24.402104 18362 leader_election.cc:288] T e245c9cf642043cfbb3b87114beeb002 P 89dd09f8aaee4b249ae3ce924a422a67 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134:44343), d43ed6b8663947d99a72cd922cc323aa (127.0.99.132:40023), 16d437fe629643948c95d628926d5ffa (127.0.99.133:46581), 002cc3411acb4fdfa0b67c6f57f687a6 (127.0.99.135:38989), 6398c8993c3c45dcaab25b42142d5671 (127.0.99.129:36983), 5b8291b26a7f448b8636187d1caf86d0 (127.0.99.131:44479)
I20240503 02:00:24.402360 17354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e245c9cf642043cfbb3b87114beeb002" candidate_uuid: "89dd09f8aaee4b249ae3ce924a422a67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d43ed6b8663947d99a72cd922cc323aa" is_pre_election: true
I20240503 02:00:24.402510 17354 raft_consensus.cc:2454] T e245c9cf642043cfbb3b87114beeb002 P d43ed6b8663947d99a72cd922cc323aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89dd09f8aaee4b249ae3ce924a422a67 in term 0.
I20240503 02:00:24.402621 17620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e245c9cf642043cfbb3b87114beeb002" candidate_uuid: "89dd09f8aaee4b249ae3ce924a422a67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c843193b801b4f1a88f630ed3ed0f375" is_pre_election: true
I20240503 02:00:24.402704 17486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e245c9cf642043cfbb3b87114beeb002" candidate_uuid: "89dd09f8aaee4b249ae3ce924a422a67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16d437fe629643948c95d628926d5ffa" is_pre_election: true
I20240503 02:00:24.402768 17620 raft_consensus.cc:2454] T e245c9cf642043cfbb3b87114beeb002 P c843193b801b4f1a88f630ed3ed0f375 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89dd09f8aaee4b249ae3ce924a422a67 in term 0.
I20240503 02:00:24.402765 17223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e245c9cf642043cfbb3b87114beeb002" candidate_uuid: "89dd09f8aaee4b249ae3ce924a422a67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b8291b26a7f448b8636187d1caf86d0" is_pre_election: true
I20240503 02:00:24.402834 17486 raft_consensus.cc:2454] T e245c9cf642043cfbb3b87114beeb002 P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89dd09f8aaee4b249ae3ce924a422a67 in term 0.
I20240503 02:00:24.402845 16959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e245c9cf642043cfbb3b87114beeb002" candidate_uuid: "89dd09f8aaee4b249ae3ce924a422a67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6398c8993c3c45dcaab25b42142d5671" is_pre_election: true
I20240503 02:00:24.402890 17223 raft_consensus.cc:2454] T e245c9cf642043cfbb3b87114beeb002 P 5b8291b26a7f448b8636187d1caf86d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89dd09f8aaee4b249ae3ce924a422a67 in term 0.
I20240503 02:00:24.402978 16959 raft_consensus.cc:2454] T e245c9cf642043cfbb3b87114beeb002 P 6398c8993c3c45dcaab25b42142d5671 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89dd09f8aaee4b249ae3ce924a422a67 in term 0.
I20240503 02:00:24.403234 17026 leader_election.cc:302] T e245c9cf642043cfbb3b87114beeb002 P 89dd09f8aaee4b249ae3ce924a422a67 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 16d437fe629643948c95d628926d5ffa, 5b8291b26a7f448b8636187d1caf86d0, 89dd09f8aaee4b249ae3ce924a422a67, c843193b801b4f1a88f630ed3ed0f375; no voters:
I20240503 02:00:24.403348 17750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e245c9cf642043cfbb3b87114beeb002" candidate_uuid: "89dd09f8aaee4b249ae3ce924a422a67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" is_pre_election: true
I20240503 02:00:24.403421 18362 raft_consensus.cc:2781] T e245c9cf642043cfbb3b87114beeb002 P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:24.403499 18362 raft_consensus.cc:482] T e245c9cf642043cfbb3b87114beeb002 P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:24.403487 17750 raft_consensus.cc:2454] T e245c9cf642043cfbb3b87114beeb002 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89dd09f8aaee4b249ae3ce924a422a67 in term 0.
I20240503 02:00:24.403569 18362 raft_consensus.cc:3037] T e245c9cf642043cfbb3b87114beeb002 P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:24.404502 18362 raft_consensus.cc:504] T e245c9cf642043cfbb3b87114beeb002 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:24.405112 18362 leader_election.cc:288] T e245c9cf642043cfbb3b87114beeb002 P 89dd09f8aaee4b249ae3ce924a422a67 [CANDIDATE]: Term 1 election: Requested vote from peers c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134:44343), d43ed6b8663947d99a72cd922cc323aa (127.0.99.132:40023), 16d437fe629643948c95d628926d5ffa (127.0.99.133:46581), 002cc3411acb4fdfa0b67c6f57f687a6 (127.0.99.135:38989), 6398c8993c3c45dcaab25b42142d5671 (127.0.99.129:36983), 5b8291b26a7f448b8636187d1caf86d0 (127.0.99.131:44479)
I20240503 02:00:24.405300 17486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e245c9cf642043cfbb3b87114beeb002" candidate_uuid: "89dd09f8aaee4b249ae3ce924a422a67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16d437fe629643948c95d628926d5ffa"
I20240503 02:00:24.405285 17354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e245c9cf642043cfbb3b87114beeb002" candidate_uuid: "89dd09f8aaee4b249ae3ce924a422a67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d43ed6b8663947d99a72cd922cc323aa"
I20240503 02:00:24.405324 17750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e245c9cf642043cfbb3b87114beeb002" candidate_uuid: "89dd09f8aaee4b249ae3ce924a422a67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "002cc3411acb4fdfa0b67c6f57f687a6"
I20240503 02:00:24.405397 17354 raft_consensus.cc:3037] T e245c9cf642043cfbb3b87114beeb002 P d43ed6b8663947d99a72cd922cc323aa [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:24.405397 17486 raft_consensus.cc:3037] T e245c9cf642043cfbb3b87114beeb002 P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:24.405448 17750 raft_consensus.cc:3037] T e245c9cf642043cfbb3b87114beeb002 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:24.405721 17223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e245c9cf642043cfbb3b87114beeb002" candidate_uuid: "89dd09f8aaee4b249ae3ce924a422a67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b8291b26a7f448b8636187d1caf86d0"
I20240503 02:00:24.405818 17223 raft_consensus.cc:3037] T e245c9cf642043cfbb3b87114beeb002 P 5b8291b26a7f448b8636187d1caf86d0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:24.406338 17354 raft_consensus.cc:2454] T e245c9cf642043cfbb3b87114beeb002 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89dd09f8aaee4b249ae3ce924a422a67 in term 1.
I20240503 02:00:24.406338 17486 raft_consensus.cc:2454] T e245c9cf642043cfbb3b87114beeb002 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89dd09f8aaee4b249ae3ce924a422a67 in term 1.
I20240503 02:00:24.406417 17750 raft_consensus.cc:2454] T e245c9cf642043cfbb3b87114beeb002 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89dd09f8aaee4b249ae3ce924a422a67 in term 1.
I20240503 02:00:24.406723 17026 leader_election.cc:302] T e245c9cf642043cfbb3b87114beeb002 P 89dd09f8aaee4b249ae3ce924a422a67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 002cc3411acb4fdfa0b67c6f57f687a6, 16d437fe629643948c95d628926d5ffa, 89dd09f8aaee4b249ae3ce924a422a67, d43ed6b8663947d99a72cd922cc323aa; no voters:
I20240503 02:00:24.406769 16959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e245c9cf642043cfbb3b87114beeb002" candidate_uuid: "89dd09f8aaee4b249ae3ce924a422a67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6398c8993c3c45dcaab25b42142d5671"
I20240503 02:00:24.406769 17620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e245c9cf642043cfbb3b87114beeb002" candidate_uuid: "89dd09f8aaee4b249ae3ce924a422a67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c843193b801b4f1a88f630ed3ed0f375"
I20240503 02:00:24.406819 17223 raft_consensus.cc:2454] T e245c9cf642043cfbb3b87114beeb002 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89dd09f8aaee4b249ae3ce924a422a67 in term 1.
I20240503 02:00:24.406879 17620 raft_consensus.cc:3037] T e245c9cf642043cfbb3b87114beeb002 P c843193b801b4f1a88f630ed3ed0f375 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:24.406879 16959 raft_consensus.cc:3037] T e245c9cf642043cfbb3b87114beeb002 P 6398c8993c3c45dcaab25b42142d5671 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:24.406908 18362 raft_consensus.cc:2781] T e245c9cf642043cfbb3b87114beeb002 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:24.406982 18362 raft_consensus.cc:686] T e245c9cf642043cfbb3b87114beeb002 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 LEADER]: Becoming Leader. State: Replica: 89dd09f8aaee4b249ae3ce924a422a67, State: Running, Role: LEADER
I20240503 02:00:24.407166 18362 consensus_queue.cc:237] T e245c9cf642043cfbb3b87114beeb002 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:24.407773 16959 raft_consensus.cc:2454] T e245c9cf642043cfbb3b87114beeb002 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89dd09f8aaee4b249ae3ce924a422a67 in term 1.
I20240503 02:00:24.407773 17620 raft_consensus.cc:2454] T e245c9cf642043cfbb3b87114beeb002 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89dd09f8aaee4b249ae3ce924a422a67 in term 1.
I20240503 02:00:24.409364 16813 catalog_manager.cc:5484] T e245c9cf642043cfbb3b87114beeb002 P 89dd09f8aaee4b249ae3ce924a422a67 reported cstate change: term changed from 0 to 1, leader changed from <none> to 89dd09f8aaee4b249ae3ce924a422a67 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "89dd09f8aaee4b249ae3ce924a422a67" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:24.426851 18230 raft_consensus.cc:482] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:24.426975 18230 raft_consensus.cc:504] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:24.428365 18230 leader_election.cc:288] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 16d437fe629643948c95d628926d5ffa (127.0.99.133:46581), c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134:44343), d43ed6b8663947d99a72cd922cc323aa (127.0.99.132:40023), 89dd09f8aaee4b249ae3ce924a422a67 (127.0.99.130:38889), 6398c8993c3c45dcaab25b42142d5671 (127.0.99.129:36983), 5b8291b26a7f448b8636187d1caf86d0 (127.0.99.131:44479)
I20240503 02:00:24.433337 17354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d2104698d5f452c8350cce10afd3898" candidate_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d43ed6b8663947d99a72cd922cc323aa" is_pre_election: true
I20240503 02:00:24.433506 17354 raft_consensus.cc:2454] T 5d2104698d5f452c8350cce10afd3898 P d43ed6b8663947d99a72cd922cc323aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 002cc3411acb4fdfa0b67c6f57f687a6 in term 0.
I20240503 02:00:24.433955 17223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d2104698d5f452c8350cce10afd3898" candidate_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b8291b26a7f448b8636187d1caf86d0" is_pre_election: true
I20240503 02:00:24.434099 17223 raft_consensus.cc:2454] T 5d2104698d5f452c8350cce10afd3898 P 5b8291b26a7f448b8636187d1caf86d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 002cc3411acb4fdfa0b67c6f57f687a6 in term 0.
I20240503 02:00:24.434185 17486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d2104698d5f452c8350cce10afd3898" candidate_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16d437fe629643948c95d628926d5ffa" is_pre_election: true
I20240503 02:00:24.434321 17486 raft_consensus.cc:2454] T 5d2104698d5f452c8350cce10afd3898 P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 002cc3411acb4fdfa0b67c6f57f687a6 in term 0.
I20240503 02:00:24.435030 17686 leader_election.cc:302] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 002cc3411acb4fdfa0b67c6f57f687a6, 16d437fe629643948c95d628926d5ffa, 5b8291b26a7f448b8636187d1caf86d0, d43ed6b8663947d99a72cd922cc323aa; no voters:
I20240503 02:00:24.435249 18230 raft_consensus.cc:2781] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:24.435312 18230 raft_consensus.cc:482] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:24.435356 18230 raft_consensus.cc:3037] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:24.436281 18230 raft_consensus.cc:504] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:24.436977 18230 leader_election.cc:288] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6 [CANDIDATE]: Term 1 election: Requested vote from peers 16d437fe629643948c95d628926d5ffa (127.0.99.133:46581), c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134:44343), d43ed6b8663947d99a72cd922cc323aa (127.0.99.132:40023), 89dd09f8aaee4b249ae3ce924a422a67 (127.0.99.130:38889), 6398c8993c3c45dcaab25b42142d5671 (127.0.99.129:36983), 5b8291b26a7f448b8636187d1caf86d0 (127.0.99.131:44479)
I20240503 02:00:24.437209 17486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d2104698d5f452c8350cce10afd3898" candidate_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16d437fe629643948c95d628926d5ffa"
I20240503 02:00:24.437325 17486 raft_consensus.cc:3037] T 5d2104698d5f452c8350cce10afd3898 P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:24.437393 16959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d2104698d5f452c8350cce10afd3898" candidate_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6398c8993c3c45dcaab25b42142d5671" is_pre_election: true
I20240503 02:00:24.437526 16959 raft_consensus.cc:2454] T 5d2104698d5f452c8350cce10afd3898 P 6398c8993c3c45dcaab25b42142d5671 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 002cc3411acb4fdfa0b67c6f57f687a6 in term 0.
I20240503 02:00:24.437925 16959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d2104698d5f452c8350cce10afd3898" candidate_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6398c8993c3c45dcaab25b42142d5671"
I20240503 02:00:24.437893 17090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d2104698d5f452c8350cce10afd3898" candidate_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89dd09f8aaee4b249ae3ce924a422a67" is_pre_election: true
I20240503 02:00:24.438021 16959 raft_consensus.cc:3037] T 5d2104698d5f452c8350cce10afd3898 P 6398c8993c3c45dcaab25b42142d5671 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:24.438042 17090 raft_consensus.cc:2454] T 5d2104698d5f452c8350cce10afd3898 P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 002cc3411acb4fdfa0b67c6f57f687a6 in term 0.
I20240503 02:00:24.437918 17089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d2104698d5f452c8350cce10afd3898" candidate_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89dd09f8aaee4b249ae3ce924a422a67"
I20240503 02:00:24.438330 17089 raft_consensus.cc:3037] T 5d2104698d5f452c8350cce10afd3898 P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:24.438318 17354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d2104698d5f452c8350cce10afd3898" candidate_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d43ed6b8663947d99a72cd922cc323aa"
I20240503 02:00:24.438417 17354 raft_consensus.cc:3037] T 5d2104698d5f452c8350cce10afd3898 P d43ed6b8663947d99a72cd922cc323aa [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:24.438473 17486 raft_consensus.cc:2454] T 5d2104698d5f452c8350cce10afd3898 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 002cc3411acb4fdfa0b67c6f57f687a6 in term 1.
I20240503 02:00:24.438966 16959 raft_consensus.cc:2454] T 5d2104698d5f452c8350cce10afd3898 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 002cc3411acb4fdfa0b67c6f57f687a6 in term 1.
I20240503 02:00:24.439329 17223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d2104698d5f452c8350cce10afd3898" candidate_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b8291b26a7f448b8636187d1caf86d0"
I20240503 02:00:24.439378 17354 raft_consensus.cc:2454] T 5d2104698d5f452c8350cce10afd3898 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 002cc3411acb4fdfa0b67c6f57f687a6 in term 1.
I20240503 02:00:24.439435 17089 raft_consensus.cc:2454] T 5d2104698d5f452c8350cce10afd3898 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 002cc3411acb4fdfa0b67c6f57f687a6 in term 1.
I20240503 02:00:24.439448 17223 raft_consensus.cc:3037] T 5d2104698d5f452c8350cce10afd3898 P 5b8291b26a7f448b8636187d1caf86d0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:24.439999 17687 leader_election.cc:302] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 002cc3411acb4fdfa0b67c6f57f687a6, 16d437fe629643948c95d628926d5ffa, 6398c8993c3c45dcaab25b42142d5671, d43ed6b8663947d99a72cd922cc323aa; no voters:
I20240503 02:00:24.440243 17618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d2104698d5f452c8350cce10afd3898" candidate_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c843193b801b4f1a88f630ed3ed0f375"
I20240503 02:00:24.440268 17620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d2104698d5f452c8350cce10afd3898" candidate_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c843193b801b4f1a88f630ed3ed0f375" is_pre_election: true
I20240503 02:00:24.440382 17618 raft_consensus.cc:3037] T 5d2104698d5f452c8350cce10afd3898 P c843193b801b4f1a88f630ed3ed0f375 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:24.440433 17223 raft_consensus.cc:2454] T 5d2104698d5f452c8350cce10afd3898 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 002cc3411acb4fdfa0b67c6f57f687a6 in term 1.
I20240503 02:00:24.440748 18230 raft_consensus.cc:2781] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:24.440940 18230 raft_consensus.cc:686] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 LEADER]: Becoming Leader. State: Replica: 002cc3411acb4fdfa0b67c6f57f687a6, State: Running, Role: LEADER
I20240503 02:00:24.441255 18230 consensus_queue.cc:237] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:24.441363 17618 raft_consensus.cc:2454] T 5d2104698d5f452c8350cce10afd3898 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 002cc3411acb4fdfa0b67c6f57f687a6 in term 1.
I20240503 02:00:24.443814 16813 catalog_manager.cc:5484] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 002cc3411acb4fdfa0b67c6f57f687a6 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } health_report { overall_health: UNKNOWN } } }
W20240503 02:00:24.571700 18531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:00:24.571933 18531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:00:24.574822 18531 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240503 02:00:24.670121 17750 raft_consensus.cc:1261] T 78f81fe4c6234ff0b5fcabd974779a66 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:24.670727 17618 raft_consensus.cc:1261] T 78f81fe4c6234ff0b5fcabd974779a66 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:24.671237 17486 raft_consensus.cc:1261] T 78f81fe4c6234ff0b5fcabd974779a66 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:24.671226 17223 raft_consensus.cc:1261] T 78f81fe4c6234ff0b5fcabd974779a66 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:24.671667 17354 raft_consensus.cc:1261] T 78f81fe4c6234ff0b5fcabd974779a66 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:24.672163 16959 raft_consensus.cc:1261] T 78f81fe4c6234ff0b5fcabd974779a66 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:24.673080 18346 consensus_queue.cc:1035] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:24.673321 18563 consensus_queue.cc:1035] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:24.673691 17750 raft_consensus.cc:1261] T e245c9cf642043cfbb3b87114beeb002 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:24.673866 18563 consensus_queue.cc:1035] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:24.674134 17486 raft_consensus.cc:1261] T e245c9cf642043cfbb3b87114beeb002 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:24.674175 18563 consensus_queue.cc:1035] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:24.674536 17618 raft_consensus.cc:1261] T e245c9cf642043cfbb3b87114beeb002 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:24.674517 18563 consensus_queue.cc:1035] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:24.674823 18445 consensus_queue.cc:1035] T e245c9cf642043cfbb3b87114beeb002 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:24.676156 17354 raft_consensus.cc:1261] T e245c9cf642043cfbb3b87114beeb002 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:24.679188 18365 consensus_queue.cc:1035] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:24.679430 18365 consensus_queue.cc:1035] T e245c9cf642043cfbb3b87114beeb002 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:24.679649 18365 consensus_queue.cc:1035] T e245c9cf642043cfbb3b87114beeb002 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:24.683485 16959 raft_consensus.cc:1261] T e245c9cf642043cfbb3b87114beeb002 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:24.684001 17223 raft_consensus.cc:1261] T e245c9cf642043cfbb3b87114beeb002 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:24.685027 18445 consensus_queue.cc:1035] T e245c9cf642043cfbb3b87114beeb002 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:24.691304 18563 consensus_queue.cc:1035] T e245c9cf642043cfbb3b87114beeb002 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:24.691895 18563 consensus_queue.cc:1035] T e245c9cf642043cfbb3b87114beeb002 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:24.694608 17089 raft_consensus.cc:1261] T 5d2104698d5f452c8350cce10afd3898 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:24.694972 17354 raft_consensus.cc:1261] T 5d2104698d5f452c8350cce10afd3898 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:24.695405 18230 consensus_queue.cc:1035] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:24.695622 18529 consensus_queue.cc:1035] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:24.696743 16959 raft_consensus.cc:1261] T 5d2104698d5f452c8350cce10afd3898 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:24.697295 17223 raft_consensus.cc:1261] T 5d2104698d5f452c8350cce10afd3898 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:24.697906 18529 consensus_queue.cc:1035] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:24.698228 18529 consensus_queue.cc:1035] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:24.701581 17620 raft_consensus.cc:1261] T 5d2104698d5f452c8350cce10afd3898 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:24.701766 17488 raft_consensus.cc:1261] T 5d2104698d5f452c8350cce10afd3898 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:24.720386 18230 consensus_queue.cc:1035] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:24.720836 18230 consensus_queue.cc:1035] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 199.242 ms
time per row: 0.099621 ms
I20240503 02:00:24.846623 16814 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39916:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 7
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240503 02:00:24.847355 16814 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_3 in case of a server failure: 8 tablet servers would be needed, 7 are available. Consider bringing up more tablet servers.
I20240503 02:00:24.853448 17468 tablet_service.cc:1467] Processing CreateTablet for tablet fdf0a4b9e17642849a4094d2f0b9a255 (DEFAULT_TABLE table=rebalance_test_table_3 [id=78777800a49e46109c1b3bb45ab99e28]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:24.853448 17334 tablet_service.cc:1467] Processing CreateTablet for tablet fdf0a4b9e17642849a4094d2f0b9a255 (DEFAULT_TABLE table=rebalance_test_table_3 [id=78777800a49e46109c1b3bb45ab99e28]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:24.853775 17334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdf0a4b9e17642849a4094d2f0b9a255. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:24.853773 17468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdf0a4b9e17642849a4094d2f0b9a255. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:24.854291 17072 tablet_service.cc:1467] Processing CreateTablet for tablet fdf0a4b9e17642849a4094d2f0b9a255 (DEFAULT_TABLE table=rebalance_test_table_3 [id=78777800a49e46109c1b3bb45ab99e28]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:24.854643 17072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdf0a4b9e17642849a4094d2f0b9a255. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:24.856230 18503 tablet_bootstrap.cc:492] T fdf0a4b9e17642849a4094d2f0b9a255 P d43ed6b8663947d99a72cd922cc323aa: Bootstrap starting.
I20240503 02:00:24.856381 17600 tablet_service.cc:1467] Processing CreateTablet for tablet fdf0a4b9e17642849a4094d2f0b9a255 (DEFAULT_TABLE table=rebalance_test_table_3 [id=78777800a49e46109c1b3bb45ab99e28]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:24.856688 17600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdf0a4b9e17642849a4094d2f0b9a255. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:24.857199 18503 tablet_bootstrap.cc:654] T fdf0a4b9e17642849a4094d2f0b9a255 P d43ed6b8663947d99a72cd922cc323aa: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:24.857388 16940 tablet_service.cc:1467] Processing CreateTablet for tablet d52afe1281064d6480427554ca4bb2f6 (DEFAULT_TABLE table=rebalance_test_table_3 [id=78777800a49e46109c1b3bb45ab99e28]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:24.857690 16940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d52afe1281064d6480427554ca4bb2f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:24.858075 18489 tablet_bootstrap.cc:492] T fdf0a4b9e17642849a4094d2f0b9a255 P 89dd09f8aaee4b249ae3ce924a422a67: Bootstrap starting.
I20240503 02:00:24.859035 18489 tablet_bootstrap.cc:654] T fdf0a4b9e17642849a4094d2f0b9a255 P 89dd09f8aaee4b249ae3ce924a422a67: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:24.859203 17730 tablet_service.cc:1467] Processing CreateTablet for tablet fdf0a4b9e17642849a4094d2f0b9a255 (DEFAULT_TABLE table=rebalance_test_table_3 [id=78777800a49e46109c1b3bb45ab99e28]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:24.859498 17730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdf0a4b9e17642849a4094d2f0b9a255. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:24.859606 18501 tablet_bootstrap.cc:492] T fdf0a4b9e17642849a4094d2f0b9a255 P c843193b801b4f1a88f630ed3ed0f375: Bootstrap starting.
I20240503 02:00:24.859911 18595 tablet_bootstrap.cc:492] T d52afe1281064d6480427554ca4bb2f6 P 6398c8993c3c45dcaab25b42142d5671: Bootstrap starting.
I20240503 02:00:24.860417 18501 tablet_bootstrap.cc:654] T fdf0a4b9e17642849a4094d2f0b9a255 P c843193b801b4f1a88f630ed3ed0f375: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:24.860697 17071 tablet_service.cc:1467] Processing CreateTablet for tablet d52afe1281064d6480427554ca4bb2f6 (DEFAULT_TABLE table=rebalance_test_table_3 [id=78777800a49e46109c1b3bb45ab99e28]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:24.860942 17071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d52afe1281064d6480427554ca4bb2f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:24.861002 18595 tablet_bootstrap.cc:654] T d52afe1281064d6480427554ca4bb2f6 P 6398c8993c3c45dcaab25b42142d5671: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:24.861291 17599 tablet_service.cc:1467] Processing CreateTablet for tablet d52afe1281064d6480427554ca4bb2f6 (DEFAULT_TABLE table=rebalance_test_table_3 [id=78777800a49e46109c1b3bb45ab99e28]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:24.861557 17599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d52afe1281064d6480427554ca4bb2f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:24.862236 16938 tablet_service.cc:1467] Processing CreateTablet for tablet fdf0a4b9e17642849a4094d2f0b9a255 (DEFAULT_TABLE table=rebalance_test_table_3 [id=78777800a49e46109c1b3bb45ab99e28]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:24.862377 17600 tablet_service.cc:1467] Processing CreateTablet for tablet 63962b169caf4f14888871e246c00075 (DEFAULT_TABLE table=rebalance_test_table_3 [id=78777800a49e46109c1b3bb45ab99e28]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:24.862568 16938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdf0a4b9e17642849a4094d2f0b9a255. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:24.862951 17072 tablet_service.cc:1467] Processing CreateTablet for tablet 63962b169caf4f14888871e246c00075 (DEFAULT_TABLE table=rebalance_test_table_3 [id=78777800a49e46109c1b3bb45ab99e28]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:24.863168 17072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63962b169caf4f14888871e246c00075. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:24.864785 17732 tablet_service.cc:1467] Processing CreateTablet for tablet d52afe1281064d6480427554ca4bb2f6 (DEFAULT_TABLE table=rebalance_test_table_3 [id=78777800a49e46109c1b3bb45ab99e28]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:24.865052 17732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d52afe1281064d6480427554ca4bb2f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:24.865119 17600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63962b169caf4f14888871e246c00075. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:24.865577 18489 tablet_bootstrap.cc:492] T fdf0a4b9e17642849a4094d2f0b9a255 P 89dd09f8aaee4b249ae3ce924a422a67: No bootstrap required, opened a new log
I20240503 02:00:24.865692 18489 ts_tablet_manager.cc:1367] T fdf0a4b9e17642849a4094d2f0b9a255 P 89dd09f8aaee4b249ae3ce924a422a67: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 02:00:24.866261 18489 raft_consensus.cc:348] T fdf0a4b9e17642849a4094d2f0b9a255 P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:24.866400 18489 raft_consensus.cc:374] T fdf0a4b9e17642849a4094d2f0b9a255 P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:24.866431 18489 raft_consensus.cc:729] T fdf0a4b9e17642849a4094d2f0b9a255 P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89dd09f8aaee4b249ae3ce924a422a67, State: Initialized, Role: FOLLOWER
I20240503 02:00:24.866573 18489 consensus_queue.cc:260] T fdf0a4b9e17642849a4094d2f0b9a255 P 89dd09f8aaee4b249ae3ce924a422a67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:24.866886 18489 ts_tablet_manager.cc:1398] T fdf0a4b9e17642849a4094d2f0b9a255 P 89dd09f8aaee4b249ae3ce924a422a67: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 02:00:24.867447 16939 tablet_service.cc:1467] Processing CreateTablet for tablet 63962b169caf4f14888871e246c00075 (DEFAULT_TABLE table=rebalance_test_table_3 [id=78777800a49e46109c1b3bb45ab99e28]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:24.867697 16939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63962b169caf4f14888871e246c00075. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:24.867801 18503 tablet_bootstrap.cc:492] T fdf0a4b9e17642849a4094d2f0b9a255 P d43ed6b8663947d99a72cd922cc323aa: No bootstrap required, opened a new log
I20240503 02:00:24.867920 18503 ts_tablet_manager.cc:1367] T fdf0a4b9e17642849a4094d2f0b9a255 P d43ed6b8663947d99a72cd922cc323aa: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.002s
I20240503 02:00:24.867966 18489 tablet_bootstrap.cc:492] T d52afe1281064d6480427554ca4bb2f6 P 89dd09f8aaee4b249ae3ce924a422a67: Bootstrap starting.
I20240503 02:00:24.868474 18503 raft_consensus.cc:348] T fdf0a4b9e17642849a4094d2f0b9a255 P d43ed6b8663947d99a72cd922cc323aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:24.868629 18503 raft_consensus.cc:374] T fdf0a4b9e17642849a4094d2f0b9a255 P d43ed6b8663947d99a72cd922cc323aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:24.868804 18503 raft_consensus.cc:729] T fdf0a4b9e17642849a4094d2f0b9a255 P d43ed6b8663947d99a72cd922cc323aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d43ed6b8663947d99a72cd922cc323aa, State: Initialized, Role: FOLLOWER
I20240503 02:00:24.868916 18489 tablet_bootstrap.cc:654] T d52afe1281064d6480427554ca4bb2f6 P 89dd09f8aaee4b249ae3ce924a422a67: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:24.868978 18503 consensus_queue.cc:260] T fdf0a4b9e17642849a4094d2f0b9a255 P d43ed6b8663947d99a72cd922cc323aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:24.869375 18503 ts_tablet_manager.cc:1398] T fdf0a4b9e17642849a4094d2f0b9a255 P d43ed6b8663947d99a72cd922cc323aa: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20240503 02:00:24.869972 18599 tablet_bootstrap.cc:492] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6: Bootstrap starting.
I20240503 02:00:24.870445 17467 tablet_service.cc:1467] Processing CreateTablet for tablet d52afe1281064d6480427554ca4bb2f6 (DEFAULT_TABLE table=rebalance_test_table_3 [id=78777800a49e46109c1b3bb45ab99e28]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:24.870733 17467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d52afe1281064d6480427554ca4bb2f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:24.871018 18599 tablet_bootstrap.cc:654] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:24.871213 18489 tablet_bootstrap.cc:492] T d52afe1281064d6480427554ca4bb2f6 P 89dd09f8aaee4b249ae3ce924a422a67: No bootstrap required, opened a new log
I20240503 02:00:24.871285 18489 ts_tablet_manager.cc:1367] T d52afe1281064d6480427554ca4bb2f6 P 89dd09f8aaee4b249ae3ce924a422a67: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:24.871681 18489 raft_consensus.cc:348] T d52afe1281064d6480427554ca4bb2f6 P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:24.871829 18489 raft_consensus.cc:374] T d52afe1281064d6480427554ca4bb2f6 P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:24.871865 18489 raft_consensus.cc:729] T d52afe1281064d6480427554ca4bb2f6 P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89dd09f8aaee4b249ae3ce924a422a67, State: Initialized, Role: FOLLOWER
I20240503 02:00:24.871994 18489 consensus_queue.cc:260] T d52afe1281064d6480427554ca4bb2f6 P 89dd09f8aaee4b249ae3ce924a422a67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:24.872273 18595 tablet_bootstrap.cc:492] T d52afe1281064d6480427554ca4bb2f6 P 6398c8993c3c45dcaab25b42142d5671: No bootstrap required, opened a new log
I20240503 02:00:24.872380 18595 ts_tablet_manager.cc:1367] T d52afe1281064d6480427554ca4bb2f6 P 6398c8993c3c45dcaab25b42142d5671: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240503 02:00:24.872541 18489 ts_tablet_manager.cc:1398] T d52afe1281064d6480427554ca4bb2f6 P 89dd09f8aaee4b249ae3ce924a422a67: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:24.872871 18489 tablet_bootstrap.cc:492] T 63962b169caf4f14888871e246c00075 P 89dd09f8aaee4b249ae3ce924a422a67: Bootstrap starting.
I20240503 02:00:24.872967 18595 raft_consensus.cc:348] T d52afe1281064d6480427554ca4bb2f6 P 6398c8993c3c45dcaab25b42142d5671 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:24.873118 18595 raft_consensus.cc:374] T d52afe1281064d6480427554ca4bb2f6 P 6398c8993c3c45dcaab25b42142d5671 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:24.873157 18595 raft_consensus.cc:729] T d52afe1281064d6480427554ca4bb2f6 P 6398c8993c3c45dcaab25b42142d5671 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6398c8993c3c45dcaab25b42142d5671, State: Initialized, Role: FOLLOWER
I20240503 02:00:24.873287 18595 consensus_queue.cc:260] T d52afe1281064d6480427554ca4bb2f6 P 6398c8993c3c45dcaab25b42142d5671 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:24.873601 18595 ts_tablet_manager.cc:1398] T d52afe1281064d6480427554ca4bb2f6 P 6398c8993c3c45dcaab25b42142d5671: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:24.873776 18489 tablet_bootstrap.cc:654] T 63962b169caf4f14888871e246c00075 P 89dd09f8aaee4b249ae3ce924a422a67: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:24.873793 18595 tablet_bootstrap.cc:492] T 63962b169caf4f14888871e246c00075 P 6398c8993c3c45dcaab25b42142d5671: Bootstrap starting.
I20240503 02:00:24.874235 17202 tablet_service.cc:1467] Processing CreateTablet for tablet fdf0a4b9e17642849a4094d2f0b9a255 (DEFAULT_TABLE table=rebalance_test_table_3 [id=78777800a49e46109c1b3bb45ab99e28]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:24.874526 17202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdf0a4b9e17642849a4094d2f0b9a255. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:24.874791 18595 tablet_bootstrap.cc:654] T 63962b169caf4f14888871e246c00075 P 6398c8993c3c45dcaab25b42142d5671: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:24.875263 18599 tablet_bootstrap.cc:492] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6: No bootstrap required, opened a new log
I20240503 02:00:24.875401 18599 ts_tablet_manager.cc:1367] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 02:00:24.875928 18599 raft_consensus.cc:348] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:24.876036 17336 tablet_service.cc:1467] Processing CreateTablet for tablet d52afe1281064d6480427554ca4bb2f6 (DEFAULT_TABLE table=rebalance_test_table_3 [id=78777800a49e46109c1b3bb45ab99e28]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:24.876118 18599 raft_consensus.cc:374] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:24.876152 18599 raft_consensus.cc:729] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 002cc3411acb4fdfa0b67c6f57f687a6, State: Initialized, Role: FOLLOWER
I20240503 02:00:24.876318 17336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d52afe1281064d6480427554ca4bb2f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:24.876332 18599 consensus_queue.cc:260] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:24.876770 18599 ts_tablet_manager.cc:1398] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 02:00:24.877000 18599 tablet_bootstrap.cc:492] T d52afe1281064d6480427554ca4bb2f6 P 002cc3411acb4fdfa0b67c6f57f687a6: Bootstrap starting.
I20240503 02:00:24.877018 18604 tablet_bootstrap.cc:492] T fdf0a4b9e17642849a4094d2f0b9a255 P 5b8291b26a7f448b8636187d1caf86d0: Bootstrap starting.
I20240503 02:00:24.876519 18595 tablet_bootstrap.cc:492] T 63962b169caf4f14888871e246c00075 P 6398c8993c3c45dcaab25b42142d5671: No bootstrap required, opened a new log
I20240503 02:00:24.877249 18595 ts_tablet_manager.cc:1367] T 63962b169caf4f14888871e246c00075 P 6398c8993c3c45dcaab25b42142d5671: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:24.877912 18604 tablet_bootstrap.cc:654] T fdf0a4b9e17642849a4094d2f0b9a255 P 5b8291b26a7f448b8636187d1caf86d0: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:24.877827 18595 raft_consensus.cc:348] T 63962b169caf4f14888871e246c00075 P 6398c8993c3c45dcaab25b42142d5671 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:24.878101 18595 raft_consensus.cc:374] T 63962b169caf4f14888871e246c00075 P 6398c8993c3c45dcaab25b42142d5671 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:24.878098 17334 tablet_service.cc:1467] Processing CreateTablet for tablet 63962b169caf4f14888871e246c00075 (DEFAULT_TABLE table=rebalance_test_table_3 [id=78777800a49e46109c1b3bb45ab99e28]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:24.878190 18595 raft_consensus.cc:729] T 63962b169caf4f14888871e246c00075 P 6398c8993c3c45dcaab25b42142d5671 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6398c8993c3c45dcaab25b42142d5671, State: Initialized, Role: FOLLOWER
I20240503 02:00:24.878382 17334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63962b169caf4f14888871e246c00075. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:24.879007 18604 tablet_bootstrap.cc:492] T fdf0a4b9e17642849a4094d2f0b9a255 P 5b8291b26a7f448b8636187d1caf86d0: No bootstrap required, opened a new log
I20240503 02:00:24.878962 18595 consensus_queue.cc:260] T 63962b169caf4f14888871e246c00075 P 6398c8993c3c45dcaab25b42142d5671 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:24.879099 18604 ts_tablet_manager.cc:1367] T fdf0a4b9e17642849a4094d2f0b9a255 P 5b8291b26a7f448b8636187d1caf86d0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:24.879482 18595 ts_tablet_manager.cc:1398] T 63962b169caf4f14888871e246c00075 P 6398c8993c3c45dcaab25b42142d5671: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:24.879531 18604 raft_consensus.cc:348] T fdf0a4b9e17642849a4094d2f0b9a255 P 5b8291b26a7f448b8636187d1caf86d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:24.879657 18604 raft_consensus.cc:374] T fdf0a4b9e17642849a4094d2f0b9a255 P 5b8291b26a7f448b8636187d1caf86d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:24.879696 18604 raft_consensus.cc:729] T fdf0a4b9e17642849a4094d2f0b9a255 P 5b8291b26a7f448b8636187d1caf86d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b8291b26a7f448b8636187d1caf86d0, State: Initialized, Role: FOLLOWER
I20240503 02:00:24.879848 18604 consensus_queue.cc:260] T fdf0a4b9e17642849a4094d2f0b9a255 P 5b8291b26a7f448b8636187d1caf86d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:24.880121 18604 ts_tablet_manager.cc:1398] T fdf0a4b9e17642849a4094d2f0b9a255 P 5b8291b26a7f448b8636187d1caf86d0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:24.880774 18503 tablet_bootstrap.cc:492] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa: Bootstrap starting.
I20240503 02:00:24.880911 18599 tablet_bootstrap.cc:654] T d52afe1281064d6480427554ca4bb2f6 P 002cc3411acb4fdfa0b67c6f57f687a6: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:24.881076 17203 tablet_service.cc:1467] Processing CreateTablet for tablet d52afe1281064d6480427554ca4bb2f6 (DEFAULT_TABLE table=rebalance_test_table_3 [id=78777800a49e46109c1b3bb45ab99e28]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:24.881386 17203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d52afe1281064d6480427554ca4bb2f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:24.881708 18503 tablet_bootstrap.cc:654] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:24.882306 18599 tablet_bootstrap.cc:492] T d52afe1281064d6480427554ca4bb2f6 P 002cc3411acb4fdfa0b67c6f57f687a6: No bootstrap required, opened a new log
I20240503 02:00:24.882413 18599 ts_tablet_manager.cc:1367] T d52afe1281064d6480427554ca4bb2f6 P 002cc3411acb4fdfa0b67c6f57f687a6: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 02:00:24.882866 18503 tablet_bootstrap.cc:492] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa: No bootstrap required, opened a new log
I20240503 02:00:24.882974 18503 ts_tablet_manager.cc:1367] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:24.882978 18599 raft_consensus.cc:348] T d52afe1281064d6480427554ca4bb2f6 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:24.883087 18599 raft_consensus.cc:374] T d52afe1281064d6480427554ca4bb2f6 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:24.883116 18599 raft_consensus.cc:729] T d52afe1281064d6480427554ca4bb2f6 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 002cc3411acb4fdfa0b67c6f57f687a6, State: Initialized, Role: FOLLOWER
I20240503 02:00:24.883258 18599 consensus_queue.cc:260] T d52afe1281064d6480427554ca4bb2f6 P 002cc3411acb4fdfa0b67c6f57f687a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:24.883796 18604 tablet_bootstrap.cc:492] T d52afe1281064d6480427554ca4bb2f6 P 5b8291b26a7f448b8636187d1caf86d0: Bootstrap starting.
I20240503 02:00:24.883728 18503 raft_consensus.cc:348] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:24.883868 18503 raft_consensus.cc:374] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:24.883899 18503 raft_consensus.cc:729] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d43ed6b8663947d99a72cd922cc323aa, State: Initialized, Role: FOLLOWER
I20240503 02:00:24.883913 18599 ts_tablet_manager.cc:1398] T d52afe1281064d6480427554ca4bb2f6 P 002cc3411acb4fdfa0b67c6f57f687a6: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 02:00:24.883945 17204 tablet_service.cc:1467] Processing CreateTablet for tablet 63962b169caf4f14888871e246c00075 (DEFAULT_TABLE table=rebalance_test_table_3 [id=78777800a49e46109c1b3bb45ab99e28]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:24.884020 18503 consensus_queue.cc:260] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:24.884145 17204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63962b169caf4f14888871e246c00075. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:24.884371 18503 ts_tablet_manager.cc:1398] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:24.884742 18604 tablet_bootstrap.cc:654] T d52afe1281064d6480427554ca4bb2f6 P 5b8291b26a7f448b8636187d1caf86d0: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:24.886713 18604 tablet_bootstrap.cc:492] T d52afe1281064d6480427554ca4bb2f6 P 5b8291b26a7f448b8636187d1caf86d0: No bootstrap required, opened a new log
I20240503 02:00:24.886801 18604 ts_tablet_manager.cc:1367] T d52afe1281064d6480427554ca4bb2f6 P 5b8291b26a7f448b8636187d1caf86d0: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 02:00:24.887158 18604 raft_consensus.cc:348] T d52afe1281064d6480427554ca4bb2f6 P 5b8291b26a7f448b8636187d1caf86d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:24.887277 18604 raft_consensus.cc:374] T d52afe1281064d6480427554ca4bb2f6 P 5b8291b26a7f448b8636187d1caf86d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:24.887312 18604 raft_consensus.cc:729] T d52afe1281064d6480427554ca4bb2f6 P 5b8291b26a7f448b8636187d1caf86d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b8291b26a7f448b8636187d1caf86d0, State: Initialized, Role: FOLLOWER
I20240503 02:00:24.887439 18604 consensus_queue.cc:260] T d52afe1281064d6480427554ca4bb2f6 P 5b8291b26a7f448b8636187d1caf86d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:24.887714 18604 ts_tablet_manager.cc:1398] T d52afe1281064d6480427554ca4bb2f6 P 5b8291b26a7f448b8636187d1caf86d0: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:24.887884 18604 tablet_bootstrap.cc:492] T 63962b169caf4f14888871e246c00075 P 5b8291b26a7f448b8636187d1caf86d0: Bootstrap starting.
I20240503 02:00:24.888427 18503 tablet_bootstrap.cc:492] T d52afe1281064d6480427554ca4bb2f6 P d43ed6b8663947d99a72cd922cc323aa: Bootstrap starting.
I20240503 02:00:24.889011 18604 tablet_bootstrap.cc:654] T 63962b169caf4f14888871e246c00075 P 5b8291b26a7f448b8636187d1caf86d0: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:24.889523 18503 tablet_bootstrap.cc:654] T d52afe1281064d6480427554ca4bb2f6 P d43ed6b8663947d99a72cd922cc323aa: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:24.890136 17468 tablet_service.cc:1467] Processing CreateTablet for tablet 63962b169caf4f14888871e246c00075 (DEFAULT_TABLE table=rebalance_test_table_3 [id=78777800a49e46109c1b3bb45ab99e28]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:24.890380 18595 tablet_bootstrap.cc:492] T fdf0a4b9e17642849a4094d2f0b9a255 P 6398c8993c3c45dcaab25b42142d5671: Bootstrap starting.
I20240503 02:00:24.890445 17468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63962b169caf4f14888871e246c00075. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:24.891990 18595 tablet_bootstrap.cc:654] T fdf0a4b9e17642849a4094d2f0b9a255 P 6398c8993c3c45dcaab25b42142d5671: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:24.892005 18604 tablet_bootstrap.cc:492] T 63962b169caf4f14888871e246c00075 P 5b8291b26a7f448b8636187d1caf86d0: No bootstrap required, opened a new log
I20240503 02:00:24.892079 18604 ts_tablet_manager.cc:1367] T 63962b169caf4f14888871e246c00075 P 5b8291b26a7f448b8636187d1caf86d0: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20240503 02:00:24.892558 18604 raft_consensus.cc:348] T 63962b169caf4f14888871e246c00075 P 5b8291b26a7f448b8636187d1caf86d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:24.892732 18604 raft_consensus.cc:374] T 63962b169caf4f14888871e246c00075 P 5b8291b26a7f448b8636187d1caf86d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:24.892776 18604 raft_consensus.cc:729] T 63962b169caf4f14888871e246c00075 P 5b8291b26a7f448b8636187d1caf86d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b8291b26a7f448b8636187d1caf86d0, State: Initialized, Role: FOLLOWER
I20240503 02:00:24.892920 18604 consensus_queue.cc:260] T 63962b169caf4f14888871e246c00075 P 5b8291b26a7f448b8636187d1caf86d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:24.893205 17730 tablet_service.cc:1467] Processing CreateTablet for tablet 63962b169caf4f14888871e246c00075 (DEFAULT_TABLE table=rebalance_test_table_3 [id=78777800a49e46109c1b3bb45ab99e28]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:24.893301 18604 ts_tablet_manager.cc:1398] T 63962b169caf4f14888871e246c00075 P 5b8291b26a7f448b8636187d1caf86d0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:24.893497 17730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63962b169caf4f14888871e246c00075. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:24.893574 18503 tablet_bootstrap.cc:492] T d52afe1281064d6480427554ca4bb2f6 P d43ed6b8663947d99a72cd922cc323aa: No bootstrap required, opened a new log
I20240503 02:00:24.893664 18503 ts_tablet_manager.cc:1367] T d52afe1281064d6480427554ca4bb2f6 P d43ed6b8663947d99a72cd922cc323aa: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 02:00:24.894169 18503 raft_consensus.cc:348] T d52afe1281064d6480427554ca4bb2f6 P d43ed6b8663947d99a72cd922cc323aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:24.894309 18503 raft_consensus.cc:374] T d52afe1281064d6480427554ca4bb2f6 P d43ed6b8663947d99a72cd922cc323aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:24.894348 18503 raft_consensus.cc:729] T d52afe1281064d6480427554ca4bb2f6 P d43ed6b8663947d99a72cd922cc323aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d43ed6b8663947d99a72cd922cc323aa, State: Initialized, Role: FOLLOWER
I20240503 02:00:24.894496 18503 consensus_queue.cc:260] T d52afe1281064d6480427554ca4bb2f6 P d43ed6b8663947d99a72cd922cc323aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:24.895011 18503 ts_tablet_manager.cc:1398] T d52afe1281064d6480427554ca4bb2f6 P d43ed6b8663947d99a72cd922cc323aa: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:24.895783 18595 tablet_bootstrap.cc:492] T fdf0a4b9e17642849a4094d2f0b9a255 P 6398c8993c3c45dcaab25b42142d5671: No bootstrap required, opened a new log
I20240503 02:00:24.895850 18599 tablet_bootstrap.cc:492] T 63962b169caf4f14888871e246c00075 P 002cc3411acb4fdfa0b67c6f57f687a6: Bootstrap starting.
I20240503 02:00:24.895881 18595 ts_tablet_manager.cc:1367] T fdf0a4b9e17642849a4094d2f0b9a255 P 6398c8993c3c45dcaab25b42142d5671: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 02:00:24.896427 18593 tablet_bootstrap.cc:492] T fdf0a4b9e17642849a4094d2f0b9a255 P 16d437fe629643948c95d628926d5ffa: Bootstrap starting.
I20240503 02:00:24.896365 18595 raft_consensus.cc:348] T fdf0a4b9e17642849a4094d2f0b9a255 P 6398c8993c3c45dcaab25b42142d5671 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:24.896494 18595 raft_consensus.cc:374] T fdf0a4b9e17642849a4094d2f0b9a255 P 6398c8993c3c45dcaab25b42142d5671 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:24.896531 18595 raft_consensus.cc:729] T fdf0a4b9e17642849a4094d2f0b9a255 P 6398c8993c3c45dcaab25b42142d5671 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6398c8993c3c45dcaab25b42142d5671, State: Initialized, Role: FOLLOWER
I20240503 02:00:24.896696 18595 consensus_queue.cc:260] T fdf0a4b9e17642849a4094d2f0b9a255 P 6398c8993c3c45dcaab25b42142d5671 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:24.896975 18595 ts_tablet_manager.cc:1398] T fdf0a4b9e17642849a4094d2f0b9a255 P 6398c8993c3c45dcaab25b42142d5671: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:24.897374 18593 tablet_bootstrap.cc:654] T fdf0a4b9e17642849a4094d2f0b9a255 P 16d437fe629643948c95d628926d5ffa: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:24.897644 18501 tablet_bootstrap.cc:492] T fdf0a4b9e17642849a4094d2f0b9a255 P c843193b801b4f1a88f630ed3ed0f375: No bootstrap required, opened a new log
I20240503 02:00:24.897748 18501 ts_tablet_manager.cc:1367] T fdf0a4b9e17642849a4094d2f0b9a255 P c843193b801b4f1a88f630ed3ed0f375: Time spent bootstrapping tablet: real 0.038s user 0.001s sys 0.001s
I20240503 02:00:24.898262 18501 raft_consensus.cc:348] T fdf0a4b9e17642849a4094d2f0b9a255 P c843193b801b4f1a88f630ed3ed0f375 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:24.898413 18501 raft_consensus.cc:374] T fdf0a4b9e17642849a4094d2f0b9a255 P c843193b801b4f1a88f630ed3ed0f375 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:24.898444 18501 raft_consensus.cc:729] T fdf0a4b9e17642849a4094d2f0b9a255 P c843193b801b4f1a88f630ed3ed0f375 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c843193b801b4f1a88f630ed3ed0f375, State: Initialized, Role: FOLLOWER
I20240503 02:00:24.898612 18501 consensus_queue.cc:260] T fdf0a4b9e17642849a4094d2f0b9a255 P c843193b801b4f1a88f630ed3ed0f375 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:24.898695 18489 tablet_bootstrap.cc:492] T 63962b169caf4f14888871e246c00075 P 89dd09f8aaee4b249ae3ce924a422a67: No bootstrap required, opened a new log
I20240503 02:00:24.898783 18489 ts_tablet_manager.cc:1367] T 63962b169caf4f14888871e246c00075 P 89dd09f8aaee4b249ae3ce924a422a67: Time spent bootstrapping tablet: real 0.026s user 0.001s sys 0.000s
I20240503 02:00:24.898914 18501 ts_tablet_manager.cc:1398] T fdf0a4b9e17642849a4094d2f0b9a255 P c843193b801b4f1a88f630ed3ed0f375: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:24.899070 18501 tablet_bootstrap.cc:492] T d52afe1281064d6480427554ca4bb2f6 P c843193b801b4f1a88f630ed3ed0f375: Bootstrap starting.
I20240503 02:00:24.899264 18489 raft_consensus.cc:348] T 63962b169caf4f14888871e246c00075 P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:24.899381 18489 raft_consensus.cc:374] T 63962b169caf4f14888871e246c00075 P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:24.899412 18489 raft_consensus.cc:729] T 63962b169caf4f14888871e246c00075 P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89dd09f8aaee4b249ae3ce924a422a67, State: Initialized, Role: FOLLOWER
I20240503 02:00:24.899557 18489 consensus_queue.cc:260] T 63962b169caf4f14888871e246c00075 P 89dd09f8aaee4b249ae3ce924a422a67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:24.899859 18489 ts_tablet_manager.cc:1398] T 63962b169caf4f14888871e246c00075 P 89dd09f8aaee4b249ae3ce924a422a67: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:24.900015 18501 tablet_bootstrap.cc:654] T d52afe1281064d6480427554ca4bb2f6 P c843193b801b4f1a88f630ed3ed0f375: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:24.901166 18599 tablet_bootstrap.cc:654] T 63962b169caf4f14888871e246c00075 P 002cc3411acb4fdfa0b67c6f57f687a6: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:24.901261 18501 tablet_bootstrap.cc:492] T d52afe1281064d6480427554ca4bb2f6 P c843193b801b4f1a88f630ed3ed0f375: No bootstrap required, opened a new log
I20240503 02:00:24.901346 18501 ts_tablet_manager.cc:1367] T d52afe1281064d6480427554ca4bb2f6 P c843193b801b4f1a88f630ed3ed0f375: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 02:00:24.901788 18501 raft_consensus.cc:348] T d52afe1281064d6480427554ca4bb2f6 P c843193b801b4f1a88f630ed3ed0f375 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:24.901914 18501 raft_consensus.cc:374] T d52afe1281064d6480427554ca4bb2f6 P c843193b801b4f1a88f630ed3ed0f375 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:24.901953 18501 raft_consensus.cc:729] T d52afe1281064d6480427554ca4bb2f6 P c843193b801b4f1a88f630ed3ed0f375 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c843193b801b4f1a88f630ed3ed0f375, State: Initialized, Role: FOLLOWER
I20240503 02:00:24.902076 18501 consensus_queue.cc:260] T d52afe1281064d6480427554ca4bb2f6 P c843193b801b4f1a88f630ed3ed0f375 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:24.902441 18501 ts_tablet_manager.cc:1398] T d52afe1281064d6480427554ca4bb2f6 P c843193b801b4f1a88f630ed3ed0f375: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:24.902642 18501 tablet_bootstrap.cc:492] T 63962b169caf4f14888871e246c00075 P c843193b801b4f1a88f630ed3ed0f375: Bootstrap starting.
I20240503 02:00:24.903069 18593 tablet_bootstrap.cc:492] T fdf0a4b9e17642849a4094d2f0b9a255 P 16d437fe629643948c95d628926d5ffa: No bootstrap required, opened a new log
I20240503 02:00:24.903120 18599 tablet_bootstrap.cc:492] T 63962b169caf4f14888871e246c00075 P 002cc3411acb4fdfa0b67c6f57f687a6: No bootstrap required, opened a new log
I20240503 02:00:24.903229 18599 ts_tablet_manager.cc:1367] T 63962b169caf4f14888871e246c00075 P 002cc3411acb4fdfa0b67c6f57f687a6: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 02:00:24.903332 18593 ts_tablet_manager.cc:1367] T fdf0a4b9e17642849a4094d2f0b9a255 P 16d437fe629643948c95d628926d5ffa: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 02:00:24.903581 18501 tablet_bootstrap.cc:654] T 63962b169caf4f14888871e246c00075 P c843193b801b4f1a88f630ed3ed0f375: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:24.903760 18599 raft_consensus.cc:348] T 63962b169caf4f14888871e246c00075 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:24.903923 18599 raft_consensus.cc:374] T 63962b169caf4f14888871e246c00075 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:24.903964 18599 raft_consensus.cc:729] T 63962b169caf4f14888871e246c00075 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 002cc3411acb4fdfa0b67c6f57f687a6, State: Initialized, Role: FOLLOWER
I20240503 02:00:24.904098 18599 consensus_queue.cc:260] T 63962b169caf4f14888871e246c00075 P 002cc3411acb4fdfa0b67c6f57f687a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:24.904412 18599 ts_tablet_manager.cc:1398] T 63962b169caf4f14888871e246c00075 P 002cc3411acb4fdfa0b67c6f57f687a6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:24.904356 18593 raft_consensus.cc:348] T fdf0a4b9e17642849a4094d2f0b9a255 P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:24.904603 18501 tablet_bootstrap.cc:492] T 63962b169caf4f14888871e246c00075 P c843193b801b4f1a88f630ed3ed0f375: No bootstrap required, opened a new log
I20240503 02:00:24.904620 18593 raft_consensus.cc:374] T fdf0a4b9e17642849a4094d2f0b9a255 P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:24.904727 18501 ts_tablet_manager.cc:1367] T 63962b169caf4f14888871e246c00075 P c843193b801b4f1a88f630ed3ed0f375: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 02:00:24.904765 18593 raft_consensus.cc:729] T fdf0a4b9e17642849a4094d2f0b9a255 P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16d437fe629643948c95d628926d5ffa, State: Initialized, Role: FOLLOWER
I20240503 02:00:24.905107 18593 consensus_queue.cc:260] T fdf0a4b9e17642849a4094d2f0b9a255 P 16d437fe629643948c95d628926d5ffa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:24.905156 18501 raft_consensus.cc:348] T 63962b169caf4f14888871e246c00075 P c843193b801b4f1a88f630ed3ed0f375 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:24.905932 18501 raft_consensus.cc:374] T 63962b169caf4f14888871e246c00075 P c843193b801b4f1a88f630ed3ed0f375 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:24.905985 18501 raft_consensus.cc:729] T 63962b169caf4f14888871e246c00075 P c843193b801b4f1a88f630ed3ed0f375 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c843193b801b4f1a88f630ed3ed0f375, State: Initialized, Role: FOLLOWER
I20240503 02:00:24.906177 18593 ts_tablet_manager.cc:1398] T fdf0a4b9e17642849a4094d2f0b9a255 P 16d437fe629643948c95d628926d5ffa: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 02:00:24.906147 18501 consensus_queue.cc:260] T 63962b169caf4f14888871e246c00075 P c843193b801b4f1a88f630ed3ed0f375 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:24.906428 18593 tablet_bootstrap.cc:492] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa: Bootstrap starting.
I20240503 02:00:24.906466 18501 ts_tablet_manager.cc:1398] T 63962b169caf4f14888871e246c00075 P c843193b801b4f1a88f630ed3ed0f375: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 02:00:24.907543 18593 tablet_bootstrap.cc:654] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:24.909093 18593 tablet_bootstrap.cc:492] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa: No bootstrap required, opened a new log
I20240503 02:00:24.909178 18593 ts_tablet_manager.cc:1367] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:24.909531 18593 raft_consensus.cc:348] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:24.909660 18593 raft_consensus.cc:374] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:24.909682 18593 raft_consensus.cc:729] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16d437fe629643948c95d628926d5ffa, State: Initialized, Role: FOLLOWER
I20240503 02:00:24.909832 18593 consensus_queue.cc:260] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:24.910106 18593 ts_tablet_manager.cc:1398] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:24.910344 18593 tablet_bootstrap.cc:492] T 63962b169caf4f14888871e246c00075 P 16d437fe629643948c95d628926d5ffa: Bootstrap starting.
I20240503 02:00:24.911293 18593 tablet_bootstrap.cc:654] T 63962b169caf4f14888871e246c00075 P 16d437fe629643948c95d628926d5ffa: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:24.912360 18593 tablet_bootstrap.cc:492] T 63962b169caf4f14888871e246c00075 P 16d437fe629643948c95d628926d5ffa: No bootstrap required, opened a new log
I20240503 02:00:24.912456 18593 ts_tablet_manager.cc:1367] T 63962b169caf4f14888871e246c00075 P 16d437fe629643948c95d628926d5ffa: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:24.912869 18593 raft_consensus.cc:348] T 63962b169caf4f14888871e246c00075 P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:24.912977 18593 raft_consensus.cc:374] T 63962b169caf4f14888871e246c00075 P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:24.912998 18593 raft_consensus.cc:729] T 63962b169caf4f14888871e246c00075 P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16d437fe629643948c95d628926d5ffa, State: Initialized, Role: FOLLOWER
I20240503 02:00:24.913120 18593 consensus_queue.cc:260] T 63962b169caf4f14888871e246c00075 P 16d437fe629643948c95d628926d5ffa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:24.913408 18593 ts_tablet_manager.cc:1398] T 63962b169caf4f14888871e246c00075 P 16d437fe629643948c95d628926d5ffa: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:24.922374 18589 raft_consensus.cc:482] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:24.922509 18589 raft_consensus.cc:504] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:24.923239 18589 leader_election.cc:288] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d43ed6b8663947d99a72cd922cc323aa (127.0.99.132:40023), 16d437fe629643948c95d628926d5ffa (127.0.99.133:46581), c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134:44343), 6398c8993c3c45dcaab25b42142d5671 (127.0.99.129:36983), 89dd09f8aaee4b249ae3ce924a422a67 (127.0.99.130:38889), 5b8291b26a7f448b8636187d1caf86d0 (127.0.99.131:44479)
I20240503 02:00:24.923485 17486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdf0a4b9e17642849a4094d2f0b9a255" candidate_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16d437fe629643948c95d628926d5ffa" is_pre_election: true
I20240503 02:00:24.923487 17354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdf0a4b9e17642849a4094d2f0b9a255" candidate_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d43ed6b8663947d99a72cd922cc323aa" is_pre_election: true
I20240503 02:00:24.923614 17486 raft_consensus.cc:2454] T fdf0a4b9e17642849a4094d2f0b9a255 P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 002cc3411acb4fdfa0b67c6f57f687a6 in term 0.
I20240503 02:00:24.923633 17354 raft_consensus.cc:2454] T fdf0a4b9e17642849a4094d2f0b9a255 P d43ed6b8663947d99a72cd922cc323aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 002cc3411acb4fdfa0b67c6f57f687a6 in term 0.
I20240503 02:00:24.923730 16958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdf0a4b9e17642849a4094d2f0b9a255" candidate_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6398c8993c3c45dcaab25b42142d5671" is_pre_election: true
I20240503 02:00:24.923725 17089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdf0a4b9e17642849a4094d2f0b9a255" candidate_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89dd09f8aaee4b249ae3ce924a422a67" is_pre_election: true
I20240503 02:00:24.923866 17089 raft_consensus.cc:2454] T fdf0a4b9e17642849a4094d2f0b9a255 P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 002cc3411acb4fdfa0b67c6f57f687a6 in term 0.
I20240503 02:00:24.923864 16958 raft_consensus.cc:2454] T fdf0a4b9e17642849a4094d2f0b9a255 P 6398c8993c3c45dcaab25b42142d5671 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 002cc3411acb4fdfa0b67c6f57f687a6 in term 0.
I20240503 02:00:24.923892 17620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdf0a4b9e17642849a4094d2f0b9a255" candidate_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c843193b801b4f1a88f630ed3ed0f375" is_pre_election: true
I20240503 02:00:24.924044 17620 raft_consensus.cc:2454] T fdf0a4b9e17642849a4094d2f0b9a255 P c843193b801b4f1a88f630ed3ed0f375 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 002cc3411acb4fdfa0b67c6f57f687a6 in term 0.
I20240503 02:00:24.924275 17224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdf0a4b9e17642849a4094d2f0b9a255" candidate_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b8291b26a7f448b8636187d1caf86d0" is_pre_election: true
I20240503 02:00:24.924348 17685 leader_election.cc:302] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 002cc3411acb4fdfa0b67c6f57f687a6, 89dd09f8aaee4b249ae3ce924a422a67, c843193b801b4f1a88f630ed3ed0f375, d43ed6b8663947d99a72cd922cc323aa; no voters:
I20240503 02:00:24.924418 17224 raft_consensus.cc:2454] T fdf0a4b9e17642849a4094d2f0b9a255 P 5b8291b26a7f448b8636187d1caf86d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 002cc3411acb4fdfa0b67c6f57f687a6 in term 0.
I20240503 02:00:24.924533 18589 raft_consensus.cc:2781] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:24.924600 18589 raft_consensus.cc:482] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:24.924664 18589 raft_consensus.cc:3037] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:24.925650 18589 raft_consensus.cc:504] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:24.926281 18589 leader_election.cc:288] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 [CANDIDATE]: Term 1 election: Requested vote from peers d43ed6b8663947d99a72cd922cc323aa (127.0.99.132:40023), 16d437fe629643948c95d628926d5ffa (127.0.99.133:46581), c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134:44343), 6398c8993c3c45dcaab25b42142d5671 (127.0.99.129:36983), 89dd09f8aaee4b249ae3ce924a422a67 (127.0.99.130:38889), 5b8291b26a7f448b8636187d1caf86d0 (127.0.99.131:44479)
I20240503 02:00:24.926461 17486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdf0a4b9e17642849a4094d2f0b9a255" candidate_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16d437fe629643948c95d628926d5ffa"
I20240503 02:00:24.926576 17620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdf0a4b9e17642849a4094d2f0b9a255" candidate_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c843193b801b4f1a88f630ed3ed0f375"
I20240503 02:00:24.926589 17354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdf0a4b9e17642849a4094d2f0b9a255" candidate_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d43ed6b8663947d99a72cd922cc323aa"
I20240503 02:00:24.926605 17486 raft_consensus.cc:3037] T fdf0a4b9e17642849a4094d2f0b9a255 P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:24.926689 17354 raft_consensus.cc:3037] T fdf0a4b9e17642849a4094d2f0b9a255 P d43ed6b8663947d99a72cd922cc323aa [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:24.926689 17620 raft_consensus.cc:3037] T fdf0a4b9e17642849a4094d2f0b9a255 P c843193b801b4f1a88f630ed3ed0f375 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:24.926784 17089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdf0a4b9e17642849a4094d2f0b9a255" candidate_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89dd09f8aaee4b249ae3ce924a422a67"
I20240503 02:00:24.926896 17089 raft_consensus.cc:3037] T fdf0a4b9e17642849a4094d2f0b9a255 P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:24.927750 17620 raft_consensus.cc:2454] T fdf0a4b9e17642849a4094d2f0b9a255 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 002cc3411acb4fdfa0b67c6f57f687a6 in term 1.
I20240503 02:00:24.927747 17354 raft_consensus.cc:2454] T fdf0a4b9e17642849a4094d2f0b9a255 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 002cc3411acb4fdfa0b67c6f57f687a6 in term 1.
I20240503 02:00:24.927789 17486 raft_consensus.cc:2454] T fdf0a4b9e17642849a4094d2f0b9a255 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 002cc3411acb4fdfa0b67c6f57f687a6 in term 1.
I20240503 02:00:24.927947 17089 raft_consensus.cc:2454] T fdf0a4b9e17642849a4094d2f0b9a255 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 002cc3411acb4fdfa0b67c6f57f687a6 in term 1.
I20240503 02:00:24.928046 16958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdf0a4b9e17642849a4094d2f0b9a255" candidate_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6398c8993c3c45dcaab25b42142d5671"
I20240503 02:00:24.928157 16958 raft_consensus.cc:3037] T fdf0a4b9e17642849a4094d2f0b9a255 P 6398c8993c3c45dcaab25b42142d5671 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:24.928278 17688 leader_election.cc:302] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 002cc3411acb4fdfa0b67c6f57f687a6, 16d437fe629643948c95d628926d5ffa, 89dd09f8aaee4b249ae3ce924a422a67, c843193b801b4f1a88f630ed3ed0f375; no voters:
I20240503 02:00:24.928373 17224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdf0a4b9e17642849a4094d2f0b9a255" candidate_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b8291b26a7f448b8636187d1caf86d0"
I20240503 02:00:24.928511 18589 raft_consensus.cc:2781] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:24.928530 17224 raft_consensus.cc:3037] T fdf0a4b9e17642849a4094d2f0b9a255 P 5b8291b26a7f448b8636187d1caf86d0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:24.928890 18589 raft_consensus.cc:686] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 LEADER]: Becoming Leader. State: Replica: 002cc3411acb4fdfa0b67c6f57f687a6, State: Running, Role: LEADER
I20240503 02:00:24.929090 18589 consensus_queue.cc:237] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:24.929251 16958 raft_consensus.cc:2454] T fdf0a4b9e17642849a4094d2f0b9a255 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 002cc3411acb4fdfa0b67c6f57f687a6 in term 1.
I20240503 02:00:24.929579 17224 raft_consensus.cc:2454] T fdf0a4b9e17642849a4094d2f0b9a255 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 002cc3411acb4fdfa0b67c6f57f687a6 in term 1.
I20240503 02:00:24.931738 16815 catalog_manager.cc:5484] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 002cc3411acb4fdfa0b67c6f57f687a6 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:24.939388 18276 raft_consensus.cc:482] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:24.939527 18276 raft_consensus.cc:504] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:24.940225 18276 leader_election.cc:288] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6398c8993c3c45dcaab25b42142d5671 (127.0.99.129:36983), 89dd09f8aaee4b249ae3ce924a422a67 (127.0.99.130:38889), 5b8291b26a7f448b8636187d1caf86d0 (127.0.99.131:44479), d43ed6b8663947d99a72cd922cc323aa (127.0.99.132:40023), c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134:44343), 002cc3411acb4fdfa0b67c6f57f687a6 (127.0.99.135:38989)
I20240503 02:00:24.940783 17224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d52afe1281064d6480427554ca4bb2f6" candidate_uuid: "16d437fe629643948c95d628926d5ffa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b8291b26a7f448b8636187d1caf86d0" is_pre_election: true
I20240503 02:00:24.940799 17354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d52afe1281064d6480427554ca4bb2f6" candidate_uuid: "16d437fe629643948c95d628926d5ffa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d43ed6b8663947d99a72cd922cc323aa" is_pre_election: true
I20240503 02:00:24.940929 17224 raft_consensus.cc:2454] T d52afe1281064d6480427554ca4bb2f6 P 5b8291b26a7f448b8636187d1caf86d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16d437fe629643948c95d628926d5ffa in term 0.
I20240503 02:00:24.940999 17354 raft_consensus.cc:2454] T d52afe1281064d6480427554ca4bb2f6 P d43ed6b8663947d99a72cd922cc323aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16d437fe629643948c95d628926d5ffa in term 0.
I20240503 02:00:24.941332 17751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d52afe1281064d6480427554ca4bb2f6" candidate_uuid: "16d437fe629643948c95d628926d5ffa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" is_pre_election: true
I20240503 02:00:24.941457 17751 raft_consensus.cc:2454] T d52afe1281064d6480427554ca4bb2f6 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16d437fe629643948c95d628926d5ffa in term 0.
I20240503 02:00:24.941759 17620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d52afe1281064d6480427554ca4bb2f6" candidate_uuid: "16d437fe629643948c95d628926d5ffa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c843193b801b4f1a88f630ed3ed0f375" is_pre_election: true
I20240503 02:00:24.941854 16958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d52afe1281064d6480427554ca4bb2f6" candidate_uuid: "16d437fe629643948c95d628926d5ffa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6398c8993c3c45dcaab25b42142d5671" is_pre_election: true
I20240503 02:00:24.941918 17620 raft_consensus.cc:2454] T d52afe1281064d6480427554ca4bb2f6 P c843193b801b4f1a88f630ed3ed0f375 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16d437fe629643948c95d628926d5ffa in term 0.
I20240503 02:00:24.941993 16958 raft_consensus.cc:2454] T d52afe1281064d6480427554ca4bb2f6 P 6398c8993c3c45dcaab25b42142d5671 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16d437fe629643948c95d628926d5ffa in term 0.
I20240503 02:00:24.942250 17421 leader_election.cc:302] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 002cc3411acb4fdfa0b67c6f57f687a6, 16d437fe629643948c95d628926d5ffa, c843193b801b4f1a88f630ed3ed0f375, d43ed6b8663947d99a72cd922cc323aa; no voters:
I20240503 02:00:24.940573 17089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d52afe1281064d6480427554ca4bb2f6" candidate_uuid: "16d437fe629643948c95d628926d5ffa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89dd09f8aaee4b249ae3ce924a422a67" is_pre_election: true
I20240503 02:00:24.942453 18276 raft_consensus.cc:2781] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:24.942515 18276 raft_consensus.cc:482] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:24.942497 17089 raft_consensus.cc:2454] T d52afe1281064d6480427554ca4bb2f6 P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16d437fe629643948c95d628926d5ffa in term 0.
I20240503 02:00:24.942577 18276 raft_consensus.cc:3037] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:24.943462 18276 raft_consensus.cc:504] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:24.944032 18276 leader_election.cc:288] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa [CANDIDATE]: Term 1 election: Requested vote from peers 6398c8993c3c45dcaab25b42142d5671 (127.0.99.129:36983), 89dd09f8aaee4b249ae3ce924a422a67 (127.0.99.130:38889), 5b8291b26a7f448b8636187d1caf86d0 (127.0.99.131:44479), d43ed6b8663947d99a72cd922cc323aa (127.0.99.132:40023), c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134:44343), 002cc3411acb4fdfa0b67c6f57f687a6 (127.0.99.135:38989)
I20240503 02:00:24.944108 16958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d52afe1281064d6480427554ca4bb2f6" candidate_uuid: "16d437fe629643948c95d628926d5ffa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6398c8993c3c45dcaab25b42142d5671"
I20240503 02:00:24.944203 16958 raft_consensus.cc:3037] T d52afe1281064d6480427554ca4bb2f6 P 6398c8993c3c45dcaab25b42142d5671 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:24.944487 17224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d52afe1281064d6480427554ca4bb2f6" candidate_uuid: "16d437fe629643948c95d628926d5ffa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b8291b26a7f448b8636187d1caf86d0"
I20240503 02:00:24.944586 17224 raft_consensus.cc:3037] T d52afe1281064d6480427554ca4bb2f6 P 5b8291b26a7f448b8636187d1caf86d0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:24.944602 17751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d52afe1281064d6480427554ca4bb2f6" candidate_uuid: "16d437fe629643948c95d628926d5ffa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "002cc3411acb4fdfa0b67c6f57f687a6"
I20240503 02:00:24.944736 17751 raft_consensus.cc:3037] T d52afe1281064d6480427554ca4bb2f6 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:24.945173 16958 raft_consensus.cc:2454] T d52afe1281064d6480427554ca4bb2f6 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16d437fe629643948c95d628926d5ffa in term 1.
I20240503 02:00:24.945683 17224 raft_consensus.cc:2454] T d52afe1281064d6480427554ca4bb2f6 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16d437fe629643948c95d628926d5ffa in term 1.
I20240503 02:00:24.945933 17620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d52afe1281064d6480427554ca4bb2f6" candidate_uuid: "16d437fe629643948c95d628926d5ffa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c843193b801b4f1a88f630ed3ed0f375"
I20240503 02:00:24.946045 17620 raft_consensus.cc:3037] T d52afe1281064d6480427554ca4bb2f6 P c843193b801b4f1a88f630ed3ed0f375 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:24.946970 17620 raft_consensus.cc:2454] T d52afe1281064d6480427554ca4bb2f6 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16d437fe629643948c95d628926d5ffa in term 1.
I20240503 02:00:24.947305 17421 leader_election.cc:302] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 16d437fe629643948c95d628926d5ffa, 5b8291b26a7f448b8636187d1caf86d0, 6398c8993c3c45dcaab25b42142d5671, c843193b801b4f1a88f630ed3ed0f375; no voters:
I20240503 02:00:24.947333 17354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d52afe1281064d6480427554ca4bb2f6" candidate_uuid: "16d437fe629643948c95d628926d5ffa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d43ed6b8663947d99a72cd922cc323aa"
I20240503 02:00:24.947464 17354 raft_consensus.cc:3037] T d52afe1281064d6480427554ca4bb2f6 P d43ed6b8663947d99a72cd922cc323aa [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:24.947500 18276 raft_consensus.cc:2781] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:24.947669 18276 raft_consensus.cc:686] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa [term 1 LEADER]: Becoming Leader. State: Replica: 16d437fe629643948c95d628926d5ffa, State: Running, Role: LEADER
I20240503 02:00:24.947842 18276 consensus_queue.cc:237] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:24.948455 17354 raft_consensus.cc:2454] T d52afe1281064d6480427554ca4bb2f6 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16d437fe629643948c95d628926d5ffa in term 1.
I20240503 02:00:24.948879 17089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d52afe1281064d6480427554ca4bb2f6" candidate_uuid: "16d437fe629643948c95d628926d5ffa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89dd09f8aaee4b249ae3ce924a422a67"
I20240503 02:00:24.949007 17089 raft_consensus.cc:3037] T d52afe1281064d6480427554ca4bb2f6 P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:24.950026 17089 raft_consensus.cc:2454] T d52afe1281064d6480427554ca4bb2f6 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16d437fe629643948c95d628926d5ffa in term 1.
I20240503 02:00:24.950363 16815 catalog_manager.cc:5484] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa reported cstate change: term changed from 0 to 1, leader changed from <none> to 16d437fe629643948c95d628926d5ffa (127.0.99.133). New cstate: current_term: 1 leader_uuid: "16d437fe629643948c95d628926d5ffa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:24.965602 17751 raft_consensus.cc:2454] T d52afe1281064d6480427554ca4bb2f6 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16d437fe629643948c95d628926d5ffa in term 1.
I20240503 02:00:24.969002 18238 raft_consensus.cc:482] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:24.969163 18238 raft_consensus.cc:504] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:24.970664 18238 leader_election.cc:288] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134:44343), 16d437fe629643948c95d628926d5ffa (127.0.99.133:46581), 002cc3411acb4fdfa0b67c6f57f687a6 (127.0.99.135:38989), 6398c8993c3c45dcaab25b42142d5671 (127.0.99.129:36983), 89dd09f8aaee4b249ae3ce924a422a67 (127.0.99.130:38889), 5b8291b26a7f448b8636187d1caf86d0 (127.0.99.131:44479)
I20240503 02:00:24.975737 17620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63962b169caf4f14888871e246c00075" candidate_uuid: "d43ed6b8663947d99a72cd922cc323aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c843193b801b4f1a88f630ed3ed0f375" is_pre_election: true
I20240503 02:00:24.975939 17620 raft_consensus.cc:2454] T 63962b169caf4f14888871e246c00075 P c843193b801b4f1a88f630ed3ed0f375 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d43ed6b8663947d99a72cd922cc323aa in term 0.
I20240503 02:00:24.976922 17224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63962b169caf4f14888871e246c00075" candidate_uuid: "d43ed6b8663947d99a72cd922cc323aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b8291b26a7f448b8636187d1caf86d0" is_pre_election: true
I20240503 02:00:24.977074 17224 raft_consensus.cc:2454] T 63962b169caf4f14888871e246c00075 P 5b8291b26a7f448b8636187d1caf86d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d43ed6b8663947d99a72cd922cc323aa in term 0.
I20240503 02:00:24.977170 17089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63962b169caf4f14888871e246c00075" candidate_uuid: "d43ed6b8663947d99a72cd922cc323aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89dd09f8aaee4b249ae3ce924a422a67" is_pre_election: true
I20240503 02:00:24.977342 17089 raft_consensus.cc:2454] T 63962b169caf4f14888871e246c00075 P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d43ed6b8663947d99a72cd922cc323aa in term 0.
I20240503 02:00:24.977654 17292 leader_election.cc:302] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 5b8291b26a7f448b8636187d1caf86d0, 89dd09f8aaee4b249ae3ce924a422a67, c843193b801b4f1a88f630ed3ed0f375, d43ed6b8663947d99a72cd922cc323aa; no voters:
I20240503 02:00:24.977994 18238 raft_consensus.cc:2781] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:24.977980 16958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63962b169caf4f14888871e246c00075" candidate_uuid: "d43ed6b8663947d99a72cd922cc323aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6398c8993c3c45dcaab25b42142d5671" is_pre_election: true
I20240503 02:00:24.978062 18238 raft_consensus.cc:482] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:24.978107 18238 raft_consensus.cc:3037] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:24.978147 16958 raft_consensus.cc:2454] T 63962b169caf4f14888871e246c00075 P 6398c8993c3c45dcaab25b42142d5671 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d43ed6b8663947d99a72cd922cc323aa in term 0.
I20240503 02:00:24.980021 17751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63962b169caf4f14888871e246c00075" candidate_uuid: "d43ed6b8663947d99a72cd922cc323aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" is_pre_election: true
I20240503 02:00:24.980165 17751 raft_consensus.cc:2454] T 63962b169caf4f14888871e246c00075 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d43ed6b8663947d99a72cd922cc323aa in term 0.
I20240503 02:00:24.981006 17486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63962b169caf4f14888871e246c00075" candidate_uuid: "d43ed6b8663947d99a72cd922cc323aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16d437fe629643948c95d628926d5ffa" is_pre_election: true
I20240503 02:00:24.981163 17486 raft_consensus.cc:2454] T 63962b169caf4f14888871e246c00075 P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d43ed6b8663947d99a72cd922cc323aa in term 0.
I20240503 02:00:25.001907 18238 raft_consensus.cc:504] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:25.002653 18238 leader_election.cc:288] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa [CANDIDATE]: Term 1 election: Requested vote from peers c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134:44343), 16d437fe629643948c95d628926d5ffa (127.0.99.133:46581), 002cc3411acb4fdfa0b67c6f57f687a6 (127.0.99.135:38989), 6398c8993c3c45dcaab25b42142d5671 (127.0.99.129:36983), 89dd09f8aaee4b249ae3ce924a422a67 (127.0.99.130:38889), 5b8291b26a7f448b8636187d1caf86d0 (127.0.99.131:44479)
I20240503 02:00:25.003355 16958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63962b169caf4f14888871e246c00075" candidate_uuid: "d43ed6b8663947d99a72cd922cc323aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6398c8993c3c45dcaab25b42142d5671"
I20240503 02:00:25.003376 17224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63962b169caf4f14888871e246c00075" candidate_uuid: "d43ed6b8663947d99a72cd922cc323aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b8291b26a7f448b8636187d1caf86d0"
I20240503 02:00:25.003512 17224 raft_consensus.cc:3037] T 63962b169caf4f14888871e246c00075 P 5b8291b26a7f448b8636187d1caf86d0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:25.003530 16958 raft_consensus.cc:3037] T 63962b169caf4f14888871e246c00075 P 6398c8993c3c45dcaab25b42142d5671 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:25.003481 17620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63962b169caf4f14888871e246c00075" candidate_uuid: "d43ed6b8663947d99a72cd922cc323aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c843193b801b4f1a88f630ed3ed0f375"
I20240503 02:00:25.003684 17620 raft_consensus.cc:3037] T 63962b169caf4f14888871e246c00075 P c843193b801b4f1a88f630ed3ed0f375 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:25.004325 17089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63962b169caf4f14888871e246c00075" candidate_uuid: "d43ed6b8663947d99a72cd922cc323aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89dd09f8aaee4b249ae3ce924a422a67"
I20240503 02:00:25.004396 17486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63962b169caf4f14888871e246c00075" candidate_uuid: "d43ed6b8663947d99a72cd922cc323aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16d437fe629643948c95d628926d5ffa"
I20240503 02:00:25.004505 17089 raft_consensus.cc:3037] T 63962b169caf4f14888871e246c00075 P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:25.004511 17486 raft_consensus.cc:3037] T 63962b169caf4f14888871e246c00075 P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:25.004809 17751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63962b169caf4f14888871e246c00075" candidate_uuid: "d43ed6b8663947d99a72cd922cc323aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "002cc3411acb4fdfa0b67c6f57f687a6"
I20240503 02:00:25.004886 17620 raft_consensus.cc:2454] T 63962b169caf4f14888871e246c00075 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d43ed6b8663947d99a72cd922cc323aa in term 1.
I20240503 02:00:25.004961 17751 raft_consensus.cc:3037] T 63962b169caf4f14888871e246c00075 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:25.005618 17486 raft_consensus.cc:2454] T 63962b169caf4f14888871e246c00075 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d43ed6b8663947d99a72cd922cc323aa in term 1.
I20240503 02:00:25.006073 17751 raft_consensus.cc:2454] T 63962b169caf4f14888871e246c00075 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d43ed6b8663947d99a72cd922cc323aa in term 1.
I20240503 02:00:25.006374 17289 leader_election.cc:302] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 002cc3411acb4fdfa0b67c6f57f687a6, 16d437fe629643948c95d628926d5ffa, c843193b801b4f1a88f630ed3ed0f375, d43ed6b8663947d99a72cd922cc323aa; no voters:
I20240503 02:00:25.006637 18238 raft_consensus.cc:2781] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:25.006862 18238 raft_consensus.cc:686] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa [term 1 LEADER]: Becoming Leader. State: Replica: d43ed6b8663947d99a72cd922cc323aa, State: Running, Role: LEADER
I20240503 02:00:25.007212 18238 consensus_queue.cc:237] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:25.009995 16815 catalog_manager.cc:5484] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa reported cstate change: term changed from 0 to 1, leader changed from <none> to d43ed6b8663947d99a72cd922cc323aa (127.0.99.132). New cstate: current_term: 1 leader_uuid: "d43ed6b8663947d99a72cd922cc323aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:25.017462 17224 raft_consensus.cc:2454] T 63962b169caf4f14888871e246c00075 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d43ed6b8663947d99a72cd922cc323aa in term 1.
I20240503 02:00:25.033914 17089 raft_consensus.cc:2454] T 63962b169caf4f14888871e246c00075 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d43ed6b8663947d99a72cd922cc323aa in term 1.
I20240503 02:00:25.034198 16958 raft_consensus.cc:2454] T 63962b169caf4f14888871e246c00075 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d43ed6b8663947d99a72cd922cc323aa in term 1.
W20240503 02:00:25.131101 18633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:00:25.131350 18633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:00:25.134799 18633 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240503 02:00:25.228665 17224 raft_consensus.cc:1261] T fdf0a4b9e17642849a4094d2f0b9a255 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:25.229336 18589 consensus_queue.cc:1035] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:25.232371 17354 raft_consensus.cc:1261] T d52afe1281064d6480427554ca4bb2f6 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:25.233036 17620 raft_consensus.cc:1261] T d52afe1281064d6480427554ca4bb2f6 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:25.233417 16958 raft_consensus.cc:1261] T d52afe1281064d6480427554ca4bb2f6 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:25.233893 18619 consensus_queue.cc:1035] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:25.234591 17223 raft_consensus.cc:1261] T d52afe1281064d6480427554ca4bb2f6 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:25.234994 17089 raft_consensus.cc:1261] T d52afe1281064d6480427554ca4bb2f6 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:25.235350 16959 raft_consensus.cc:1261] T fdf0a4b9e17642849a4094d2f0b9a255 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:25.235688 17091 raft_consensus.cc:1261] T fdf0a4b9e17642849a4094d2f0b9a255 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:25.236403 18583 consensus_queue.cc:1035] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:25.236730 18583 consensus_queue.cc:1035] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:25.236848 17751 raft_consensus.cc:1261] T d52afe1281064d6480427554ca4bb2f6 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:25.237838 17355 raft_consensus.cc:1261] T fdf0a4b9e17642849a4094d2f0b9a255 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:25.238062 18276 consensus_queue.cc:1035] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:25.238312 18276 consensus_queue.cc:1035] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:25.238480 17486 raft_consensus.cc:1261] T fdf0a4b9e17642849a4094d2f0b9a255 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:25.238531 18276 consensus_queue.cc:1035] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:25.238777 18276 consensus_queue.cc:1035] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:25.239427 18276 consensus_queue.cc:1035] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:25.239578 17620 raft_consensus.cc:1261] T fdf0a4b9e17642849a4094d2f0b9a255 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:25.240007 18529 consensus_queue.cc:1035] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:25.240365 18589 consensus_queue.cc:1035] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:25.241255 18230 consensus_queue.cc:1035] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:25.258924 17222 raft_consensus.cc:1261] T 63962b169caf4f14888871e246c00075 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:25.258942 17089 raft_consensus.cc:1261] T 63962b169caf4f14888871e246c00075 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:25.259475 18238 consensus_queue.cc:1035] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:25.260861 16960 raft_consensus.cc:1261] T 63962b169caf4f14888871e246c00075 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:25.261235 18631 consensus_queue.cc:1035] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:25.262544 17751 raft_consensus.cc:1261] T 63962b169caf4f14888871e246c00075 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:25.262972 17618 raft_consensus.cc:1261] T 63962b169caf4f14888871e246c00075 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:25.263759 17488 raft_consensus.cc:1261] T 63962b169caf4f14888871e246c00075 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:25.265228 18679 consensus_queue.cc:1035] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:25.265549 18631 consensus_queue.cc:1035] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:25.265844 18679 consensus_queue.cc:1035] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:25.267385 18680 consensus_queue.cc:1035] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:25.287467 18309 mvcc.cc:205] Tried to move back new op lower bound from 7023417856942133248 to 7023417855789830144. Current Snapshot: MvccSnapshot[applied={T|T < 7023417856942133248}]
I20240503 02:00:25.289716 18464 mvcc.cc:205] Tried to move back new op lower bound from 7023417856932184064 to 7023417855714304000. Current Snapshot: MvccSnapshot[applied={T|T < 7023417856932184064}]
I20240503 02:00:25.299973 18466 mvcc.cc:205] Tried to move back new op lower bound from 7023417856942133248 to 7023417855789830144. Current Snapshot: MvccSnapshot[applied={T|T < 7023417856942133248}]
I20240503 02:00:25.329771 18472 mvcc.cc:205] Tried to move back new op lower bound from 7023417857054019584 to 7023417856033345536. Current Snapshot: MvccSnapshot[applied={T|T < 7023417857054019584}]
I20240503 02:00:25.330029 18587 mvcc.cc:205] Tried to move back new op lower bound from 7023417857054019584 to 7023417856033345536. Current Snapshot: MvccSnapshot[applied={T|T < 7023417857054019584}]
I20240503 02:00:25.353493 18444 mvcc.cc:205] Tried to move back new op lower bound from 7023417857054019584 to 7023417856033345536. Current Snapshot: MvccSnapshot[applied={T|T < 7023417857054019584}]
INSERT report
rows total: 2000
time total: 245.185 ms
time per row: 0.122593 ms
I20240503 02:00:25.434947 16815 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39916:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 7
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240503 02:00:25.435601 16815 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_4 in case of a server failure: 8 tablet servers would be needed, 7 are available. Consider bringing up more tablet servers.
I20240503 02:00:25.442265 16938 tablet_service.cc:1467] Processing CreateTablet for tablet d81dcfefb4fd401b9cfd2ca36034a010 (DEFAULT_TABLE table=rebalance_test_table_4 [id=421dd60555d242a78638e28180581a9c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:25.442418 17072 tablet_service.cc:1467] Processing CreateTablet for tablet d81dcfefb4fd401b9cfd2ca36034a010 (DEFAULT_TABLE table=rebalance_test_table_4 [id=421dd60555d242a78638e28180581a9c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:25.442669 16938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d81dcfefb4fd401b9cfd2ca36034a010. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:25.442719 17072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d81dcfefb4fd401b9cfd2ca36034a010. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:25.444445 16939 tablet_service.cc:1467] Processing CreateTablet for tablet ae4259dd3dc745869256269632886f8a (DEFAULT_TABLE table=rebalance_test_table_4 [id=421dd60555d242a78638e28180581a9c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:25.444788 16939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae4259dd3dc745869256269632886f8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:25.444926 17466 tablet_service.cc:1467] Processing CreateTablet for tablet a39f284167e942188c181147e8a9e74e (DEFAULT_TABLE table=rebalance_test_table_4 [id=421dd60555d242a78638e28180581a9c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:25.445123 17468 tablet_service.cc:1467] Processing CreateTablet for tablet d81dcfefb4fd401b9cfd2ca36034a010 (DEFAULT_TABLE table=rebalance_test_table_4 [id=421dd60555d242a78638e28180581a9c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:25.445294 17466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a39f284167e942188c181147e8a9e74e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:25.445488 17467 tablet_service.cc:1467] Processing CreateTablet for tablet ae4259dd3dc745869256269632886f8a (DEFAULT_TABLE table=rebalance_test_table_4 [id=421dd60555d242a78638e28180581a9c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:25.445709 17467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae4259dd3dc745869256269632886f8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:25.447027 17335 tablet_service.cc:1467] Processing CreateTablet for tablet a39f284167e942188c181147e8a9e74e (DEFAULT_TABLE table=rebalance_test_table_4 [id=421dd60555d242a78638e28180581a9c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:25.447235 17334 tablet_service.cc:1467] Processing CreateTablet for tablet d81dcfefb4fd401b9cfd2ca36034a010 (DEFAULT_TABLE table=rebalance_test_table_4 [id=421dd60555d242a78638e28180581a9c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:25.447348 17335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a39f284167e942188c181147e8a9e74e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:25.447629 17336 tablet_service.cc:1467] Processing CreateTablet for tablet ae4259dd3dc745869256269632886f8a (DEFAULT_TABLE table=rebalance_test_table_4 [id=421dd60555d242a78638e28180581a9c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:25.447731 17730 tablet_service.cc:1467] Processing CreateTablet for tablet d81dcfefb4fd401b9cfd2ca36034a010 (DEFAULT_TABLE table=rebalance_test_table_4 [id=421dd60555d242a78638e28180581a9c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:25.447839 17336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae4259dd3dc745869256269632886f8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:25.448015 17730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d81dcfefb4fd401b9cfd2ca36034a010. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:25.449378 18696 tablet_bootstrap.cc:492] T d81dcfefb4fd401b9cfd2ca36034a010 P 89dd09f8aaee4b249ae3ce924a422a67: Bootstrap starting.
I20240503 02:00:25.450178 17072 tablet_service.cc:1467] Processing CreateTablet for tablet ae4259dd3dc745869256269632886f8a (DEFAULT_TABLE table=rebalance_test_table_4 [id=421dd60555d242a78638e28180581a9c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:25.450155 17204 tablet_service.cc:1467] Processing CreateTablet for tablet d81dcfefb4fd401b9cfd2ca36034a010 (DEFAULT_TABLE table=rebalance_test_table_4 [id=421dd60555d242a78638e28180581a9c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:25.450299 17732 tablet_service.cc:1467] Processing CreateTablet for tablet ae4259dd3dc745869256269632886f8a (DEFAULT_TABLE table=rebalance_test_table_4 [id=421dd60555d242a78638e28180581a9c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:25.450443 17072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae4259dd3dc745869256269632886f8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:25.450464 17204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d81dcfefb4fd401b9cfd2ca36034a010. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:25.450459 18696 tablet_bootstrap.cc:654] T d81dcfefb4fd401b9cfd2ca36034a010 P 89dd09f8aaee4b249ae3ce924a422a67: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:25.450557 17732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae4259dd3dc745869256269632886f8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:25.452211 17731 tablet_service.cc:1467] Processing CreateTablet for tablet a39f284167e942188c181147e8a9e74e (DEFAULT_TABLE table=rebalance_test_table_4 [id=421dd60555d242a78638e28180581a9c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:25.452430 17731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a39f284167e942188c181147e8a9e74e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:25.452927 18700 tablet_bootstrap.cc:492] T d81dcfefb4fd401b9cfd2ca36034a010 P 002cc3411acb4fdfa0b67c6f57f687a6: Bootstrap starting.
I20240503 02:00:25.452905 17203 tablet_service.cc:1467] Processing CreateTablet for tablet ae4259dd3dc745869256269632886f8a (DEFAULT_TABLE table=rebalance_test_table_4 [id=421dd60555d242a78638e28180581a9c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:25.453157 17203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae4259dd3dc745869256269632886f8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:25.454077 18700 tablet_bootstrap.cc:654] T d81dcfefb4fd401b9cfd2ca36034a010 P 002cc3411acb4fdfa0b67c6f57f687a6: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:25.454697 17600 tablet_service.cc:1467] Processing CreateTablet for tablet d81dcfefb4fd401b9cfd2ca36034a010 (DEFAULT_TABLE table=rebalance_test_table_4 [id=421dd60555d242a78638e28180581a9c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:25.454923 18699 tablet_bootstrap.cc:492] T a39f284167e942188c181147e8a9e74e P d43ed6b8663947d99a72cd922cc323aa: Bootstrap starting.
I20240503 02:00:25.455029 17600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d81dcfefb4fd401b9cfd2ca36034a010. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:25.455042 17598 tablet_service.cc:1467] Processing CreateTablet for tablet a39f284167e942188c181147e8a9e74e (DEFAULT_TABLE table=rebalance_test_table_4 [id=421dd60555d242a78638e28180581a9c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:25.455509 17598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a39f284167e942188c181147e8a9e74e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:25.455830 17334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d81dcfefb4fd401b9cfd2ca36034a010. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:25.455936 18699 tablet_bootstrap.cc:654] T a39f284167e942188c181147e8a9e74e P d43ed6b8663947d99a72cd922cc323aa: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:25.454697 17599 tablet_service.cc:1467] Processing CreateTablet for tablet ae4259dd3dc745869256269632886f8a (DEFAULT_TABLE table=rebalance_test_table_4 [id=421dd60555d242a78638e28180581a9c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:25.457479 18700 tablet_bootstrap.cc:492] T d81dcfefb4fd401b9cfd2ca36034a010 P 002cc3411acb4fdfa0b67c6f57f687a6: No bootstrap required, opened a new log
I20240503 02:00:25.457602 18700 ts_tablet_manager.cc:1367] T d81dcfefb4fd401b9cfd2ca36034a010 P 002cc3411acb4fdfa0b67c6f57f687a6: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 02:00:25.457619 17599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae4259dd3dc745869256269632886f8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:25.458200 18700 raft_consensus.cc:348] T d81dcfefb4fd401b9cfd2ca36034a010 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:25.458365 18700 raft_consensus.cc:374] T d81dcfefb4fd401b9cfd2ca36034a010 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:25.458431 18700 raft_consensus.cc:729] T d81dcfefb4fd401b9cfd2ca36034a010 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 002cc3411acb4fdfa0b67c6f57f687a6, State: Initialized, Role: FOLLOWER
I20240503 02:00:25.458587 18700 consensus_queue.cc:260] T d81dcfefb4fd401b9cfd2ca36034a010 P 002cc3411acb4fdfa0b67c6f57f687a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:25.458969 18700 ts_tablet_manager.cc:1398] T d81dcfefb4fd401b9cfd2ca36034a010 P 002cc3411acb4fdfa0b67c6f57f687a6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:25.459684 17468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d81dcfefb4fd401b9cfd2ca36034a010. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:25.459748 18700 tablet_bootstrap.cc:492] T ae4259dd3dc745869256269632886f8a P 002cc3411acb4fdfa0b67c6f57f687a6: Bootstrap starting.
I20240503 02:00:25.460776 18700 tablet_bootstrap.cc:654] T ae4259dd3dc745869256269632886f8a P 002cc3411acb4fdfa0b67c6f57f687a6: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:25.462244 18699 tablet_bootstrap.cc:492] T a39f284167e942188c181147e8a9e74e P d43ed6b8663947d99a72cd922cc323aa: No bootstrap required, opened a new log
I20240503 02:00:25.462343 18700 tablet_bootstrap.cc:492] T ae4259dd3dc745869256269632886f8a P 002cc3411acb4fdfa0b67c6f57f687a6: No bootstrap required, opened a new log
I20240503 02:00:25.462348 18697 tablet_bootstrap.cc:492] T ae4259dd3dc745869256269632886f8a P 6398c8993c3c45dcaab25b42142d5671: Bootstrap starting.
I20240503 02:00:25.462368 18699 ts_tablet_manager.cc:1367] T a39f284167e942188c181147e8a9e74e P d43ed6b8663947d99a72cd922cc323aa: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 02:00:25.462426 18700 ts_tablet_manager.cc:1367] T ae4259dd3dc745869256269632886f8a P 002cc3411acb4fdfa0b67c6f57f687a6: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 02:00:25.462917 18700 raft_consensus.cc:348] T ae4259dd3dc745869256269632886f8a P 002cc3411acb4fdfa0b67c6f57f687a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:25.463047 18700 raft_consensus.cc:374] T ae4259dd3dc745869256269632886f8a P 002cc3411acb4fdfa0b67c6f57f687a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:25.463088 18700 raft_consensus.cc:729] T ae4259dd3dc745869256269632886f8a P 002cc3411acb4fdfa0b67c6f57f687a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 002cc3411acb4fdfa0b67c6f57f687a6, State: Initialized, Role: FOLLOWER
I20240503 02:00:25.463325 18697 tablet_bootstrap.cc:654] T ae4259dd3dc745869256269632886f8a P 6398c8993c3c45dcaab25b42142d5671: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:25.463292 18699 raft_consensus.cc:348] T a39f284167e942188c181147e8a9e74e P d43ed6b8663947d99a72cd922cc323aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:25.463567 18699 raft_consensus.cc:374] T a39f284167e942188c181147e8a9e74e P d43ed6b8663947d99a72cd922cc323aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:25.463670 18699 raft_consensus.cc:729] T a39f284167e942188c181147e8a9e74e P d43ed6b8663947d99a72cd922cc323aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d43ed6b8663947d99a72cd922cc323aa, State: Initialized, Role: FOLLOWER
I20240503 02:00:25.463863 18700 consensus_queue.cc:260] T ae4259dd3dc745869256269632886f8a P 002cc3411acb4fdfa0b67c6f57f687a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:25.463877 18699 consensus_queue.cc:260] T a39f284167e942188c181147e8a9e74e P d43ed6b8663947d99a72cd922cc323aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:25.464352 18700 ts_tablet_manager.cc:1398] T ae4259dd3dc745869256269632886f8a P 002cc3411acb4fdfa0b67c6f57f687a6: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 02:00:25.464550 18700 tablet_bootstrap.cc:492] T a39f284167e942188c181147e8a9e74e P 002cc3411acb4fdfa0b67c6f57f687a6: Bootstrap starting.
I20240503 02:00:25.464843 16938 tablet_service.cc:1467] Processing CreateTablet for tablet a39f284167e942188c181147e8a9e74e (DEFAULT_TABLE table=rebalance_test_table_4 [id=421dd60555d242a78638e28180581a9c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:25.464907 18699 ts_tablet_manager.cc:1398] T a39f284167e942188c181147e8a9e74e P d43ed6b8663947d99a72cd922cc323aa: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:25.465128 16938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a39f284167e942188c181147e8a9e74e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:25.465250 18699 tablet_bootstrap.cc:492] T ae4259dd3dc745869256269632886f8a P d43ed6b8663947d99a72cd922cc323aa: Bootstrap starting.
I20240503 02:00:25.465979 18700 tablet_bootstrap.cc:654] T a39f284167e942188c181147e8a9e74e P 002cc3411acb4fdfa0b67c6f57f687a6: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:25.466918 18699 tablet_bootstrap.cc:654] T ae4259dd3dc745869256269632886f8a P d43ed6b8663947d99a72cd922cc323aa: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:25.467628 18700 tablet_bootstrap.cc:492] T a39f284167e942188c181147e8a9e74e P 002cc3411acb4fdfa0b67c6f57f687a6: No bootstrap required, opened a new log
I20240503 02:00:25.467731 18700 ts_tablet_manager.cc:1367] T a39f284167e942188c181147e8a9e74e P 002cc3411acb4fdfa0b67c6f57f687a6: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.002s
I20240503 02:00:25.468192 18705 tablet_bootstrap.cc:492] T d81dcfefb4fd401b9cfd2ca36034a010 P 5b8291b26a7f448b8636187d1caf86d0: Bootstrap starting.
I20240503 02:00:25.468215 18700 raft_consensus.cc:348] T a39f284167e942188c181147e8a9e74e P 002cc3411acb4fdfa0b67c6f57f687a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:25.468333 18700 raft_consensus.cc:374] T a39f284167e942188c181147e8a9e74e P 002cc3411acb4fdfa0b67c6f57f687a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:25.468364 18700 raft_consensus.cc:729] T a39f284167e942188c181147e8a9e74e P 002cc3411acb4fdfa0b67c6f57f687a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 002cc3411acb4fdfa0b67c6f57f687a6, State: Initialized, Role: FOLLOWER
I20240503 02:00:25.468508 18700 consensus_queue.cc:260] T a39f284167e942188c181147e8a9e74e P 002cc3411acb4fdfa0b67c6f57f687a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:25.468947 17072 tablet_service.cc:1467] Processing CreateTablet for tablet a39f284167e942188c181147e8a9e74e (DEFAULT_TABLE table=rebalance_test_table_4 [id=421dd60555d242a78638e28180581a9c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:25.469024 18700 ts_tablet_manager.cc:1398] T a39f284167e942188c181147e8a9e74e P 002cc3411acb4fdfa0b67c6f57f687a6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:25.469256 17072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a39f284167e942188c181147e8a9e74e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:25.469379 18705 tablet_bootstrap.cc:654] T d81dcfefb4fd401b9cfd2ca36034a010 P 5b8291b26a7f448b8636187d1caf86d0: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:25.469906 18699 tablet_bootstrap.cc:492] T ae4259dd3dc745869256269632886f8a P d43ed6b8663947d99a72cd922cc323aa: No bootstrap required, opened a new log
I20240503 02:00:25.470023 18699 ts_tablet_manager.cc:1367] T ae4259dd3dc745869256269632886f8a P d43ed6b8663947d99a72cd922cc323aa: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 02:00:25.470836 18699 raft_consensus.cc:348] T ae4259dd3dc745869256269632886f8a P d43ed6b8663947d99a72cd922cc323aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:25.470978 18699 raft_consensus.cc:374] T ae4259dd3dc745869256269632886f8a P d43ed6b8663947d99a72cd922cc323aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:25.471029 18699 raft_consensus.cc:729] T ae4259dd3dc745869256269632886f8a P d43ed6b8663947d99a72cd922cc323aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d43ed6b8663947d99a72cd922cc323aa, State: Initialized, Role: FOLLOWER
I20240503 02:00:25.471227 18699 consensus_queue.cc:260] T ae4259dd3dc745869256269632886f8a P d43ed6b8663947d99a72cd922cc323aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:25.471527 18698 tablet_bootstrap.cc:492] T a39f284167e942188c181147e8a9e74e P 16d437fe629643948c95d628926d5ffa: Bootstrap starting.
I20240503 02:00:25.471660 18699 ts_tablet_manager.cc:1398] T ae4259dd3dc745869256269632886f8a P d43ed6b8663947d99a72cd922cc323aa: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:25.473101 18699 tablet_bootstrap.cc:492] T d81dcfefb4fd401b9cfd2ca36034a010 P d43ed6b8663947d99a72cd922cc323aa: Bootstrap starting.
I20240503 02:00:25.473086 17204 tablet_service.cc:1467] Processing CreateTablet for tablet a39f284167e942188c181147e8a9e74e (DEFAULT_TABLE table=rebalance_test_table_4 [id=421dd60555d242a78638e28180581a9c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:25.473342 17204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a39f284167e942188c181147e8a9e74e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:25.473037 18697 tablet_bootstrap.cc:492] T ae4259dd3dc745869256269632886f8a P 6398c8993c3c45dcaab25b42142d5671: No bootstrap required, opened a new log
I20240503 02:00:25.473641 18697 ts_tablet_manager.cc:1367] T ae4259dd3dc745869256269632886f8a P 6398c8993c3c45dcaab25b42142d5671: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240503 02:00:25.474187 18697 raft_consensus.cc:348] T ae4259dd3dc745869256269632886f8a P 6398c8993c3c45dcaab25b42142d5671 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:25.474385 18697 raft_consensus.cc:374] T ae4259dd3dc745869256269632886f8a P 6398c8993c3c45dcaab25b42142d5671 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:25.474594 18697 raft_consensus.cc:729] T ae4259dd3dc745869256269632886f8a P 6398c8993c3c45dcaab25b42142d5671 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6398c8993c3c45dcaab25b42142d5671, State: Initialized, Role: FOLLOWER
I20240503 02:00:25.474843 18697 consensus_queue.cc:260] T ae4259dd3dc745869256269632886f8a P 6398c8993c3c45dcaab25b42142d5671 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:25.475051 18699 tablet_bootstrap.cc:654] T d81dcfefb4fd401b9cfd2ca36034a010 P d43ed6b8663947d99a72cd922cc323aa: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:25.476064 18699 tablet_bootstrap.cc:492] T d81dcfefb4fd401b9cfd2ca36034a010 P d43ed6b8663947d99a72cd922cc323aa: No bootstrap required, opened a new log
I20240503 02:00:25.476143 18699 ts_tablet_manager.cc:1367] T d81dcfefb4fd401b9cfd2ca36034a010 P d43ed6b8663947d99a72cd922cc323aa: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:00:25.476524 18699 raft_consensus.cc:348] T d81dcfefb4fd401b9cfd2ca36034a010 P d43ed6b8663947d99a72cd922cc323aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:25.476639 18699 raft_consensus.cc:374] T d81dcfefb4fd401b9cfd2ca36034a010 P d43ed6b8663947d99a72cd922cc323aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:25.476707 18699 raft_consensus.cc:729] T d81dcfefb4fd401b9cfd2ca36034a010 P d43ed6b8663947d99a72cd922cc323aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d43ed6b8663947d99a72cd922cc323aa, State: Initialized, Role: FOLLOWER
I20240503 02:00:25.476835 18699 consensus_queue.cc:260] T d81dcfefb4fd401b9cfd2ca36034a010 P d43ed6b8663947d99a72cd922cc323aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:25.477165 18699 ts_tablet_manager.cc:1398] T d81dcfefb4fd401b9cfd2ca36034a010 P d43ed6b8663947d99a72cd922cc323aa: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:25.477213 18697 ts_tablet_manager.cc:1398] T ae4259dd3dc745869256269632886f8a P 6398c8993c3c45dcaab25b42142d5671: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 02:00:25.477844 18704 tablet_bootstrap.cc:492] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375: Bootstrap starting.
I20240503 02:00:25.478561 18697 tablet_bootstrap.cc:492] T d81dcfefb4fd401b9cfd2ca36034a010 P 6398c8993c3c45dcaab25b42142d5671: Bootstrap starting.
I20240503 02:00:25.479617 18704 tablet_bootstrap.cc:654] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:25.480206 18697 tablet_bootstrap.cc:654] T d81dcfefb4fd401b9cfd2ca36034a010 P 6398c8993c3c45dcaab25b42142d5671: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:25.481047 18696 tablet_bootstrap.cc:492] T d81dcfefb4fd401b9cfd2ca36034a010 P 89dd09f8aaee4b249ae3ce924a422a67: No bootstrap required, opened a new log
I20240503 02:00:25.481168 18696 ts_tablet_manager.cc:1367] T d81dcfefb4fd401b9cfd2ca36034a010 P 89dd09f8aaee4b249ae3ce924a422a67: Time spent bootstrapping tablet: real 0.032s user 0.002s sys 0.000s
I20240503 02:00:25.481588 18696 raft_consensus.cc:348] T d81dcfefb4fd401b9cfd2ca36034a010 P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:25.481709 18696 raft_consensus.cc:374] T d81dcfefb4fd401b9cfd2ca36034a010 P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:25.481745 18696 raft_consensus.cc:729] T d81dcfefb4fd401b9cfd2ca36034a010 P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89dd09f8aaee4b249ae3ce924a422a67, State: Initialized, Role: FOLLOWER
I20240503 02:00:25.481885 18696 consensus_queue.cc:260] T d81dcfefb4fd401b9cfd2ca36034a010 P 89dd09f8aaee4b249ae3ce924a422a67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:25.482235 18696 ts_tablet_manager.cc:1398] T d81dcfefb4fd401b9cfd2ca36034a010 P 89dd09f8aaee4b249ae3ce924a422a67: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:25.483346 18698 tablet_bootstrap.cc:654] T a39f284167e942188c181147e8a9e74e P 16d437fe629643948c95d628926d5ffa: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:25.483536 18704 tablet_bootstrap.cc:492] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375: No bootstrap required, opened a new log
I20240503 02:00:25.483640 18704 ts_tablet_manager.cc:1367] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 02:00:25.484025 18704 raft_consensus.cc:348] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:25.484127 18704 raft_consensus.cc:374] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:25.484161 18704 raft_consensus.cc:729] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c843193b801b4f1a88f630ed3ed0f375, State: Initialized, Role: FOLLOWER
I20240503 02:00:25.484324 18704 consensus_queue.cc:260] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:25.484529 18696 tablet_bootstrap.cc:492] T ae4259dd3dc745869256269632886f8a P 89dd09f8aaee4b249ae3ce924a422a67: Bootstrap starting.
I20240503 02:00:25.485334 18698 tablet_bootstrap.cc:492] T a39f284167e942188c181147e8a9e74e P 16d437fe629643948c95d628926d5ffa: No bootstrap required, opened a new log
I20240503 02:00:25.485596 18698 ts_tablet_manager.cc:1367] T a39f284167e942188c181147e8a9e74e P 16d437fe629643948c95d628926d5ffa: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.000s
I20240503 02:00:25.486169 18698 raft_consensus.cc:348] T a39f284167e942188c181147e8a9e74e P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:25.486299 18696 tablet_bootstrap.cc:654] T ae4259dd3dc745869256269632886f8a P 89dd09f8aaee4b249ae3ce924a422a67: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:25.486377 18704 ts_tablet_manager.cc:1398] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 02:00:25.486392 18698 raft_consensus.cc:374] T a39f284167e942188c181147e8a9e74e P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:25.486515 18698 raft_consensus.cc:729] T a39f284167e942188c181147e8a9e74e P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16d437fe629643948c95d628926d5ffa, State: Initialized, Role: FOLLOWER
I20240503 02:00:25.486745 18698 consensus_queue.cc:260] T a39f284167e942188c181147e8a9e74e P 16d437fe629643948c95d628926d5ffa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:25.487205 18698 ts_tablet_manager.cc:1398] T a39f284167e942188c181147e8a9e74e P 16d437fe629643948c95d628926d5ffa: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:25.487583 18704 tablet_bootstrap.cc:492] T a39f284167e942188c181147e8a9e74e P c843193b801b4f1a88f630ed3ed0f375: Bootstrap starting.
I20240503 02:00:25.488665 18704 tablet_bootstrap.cc:654] T a39f284167e942188c181147e8a9e74e P c843193b801b4f1a88f630ed3ed0f375: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:25.488814 18698 tablet_bootstrap.cc:492] T d81dcfefb4fd401b9cfd2ca36034a010 P 16d437fe629643948c95d628926d5ffa: Bootstrap starting.
I20240503 02:00:25.489835 18698 tablet_bootstrap.cc:654] T d81dcfefb4fd401b9cfd2ca36034a010 P 16d437fe629643948c95d628926d5ffa: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:25.490747 18697 tablet_bootstrap.cc:492] T d81dcfefb4fd401b9cfd2ca36034a010 P 6398c8993c3c45dcaab25b42142d5671: No bootstrap required, opened a new log
I20240503 02:00:25.490857 18697 ts_tablet_manager.cc:1367] T d81dcfefb4fd401b9cfd2ca36034a010 P 6398c8993c3c45dcaab25b42142d5671: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240503 02:00:25.491322 18697 raft_consensus.cc:348] T d81dcfefb4fd401b9cfd2ca36034a010 P 6398c8993c3c45dcaab25b42142d5671 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:25.491437 18697 raft_consensus.cc:374] T d81dcfefb4fd401b9cfd2ca36034a010 P 6398c8993c3c45dcaab25b42142d5671 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:25.491478 18697 raft_consensus.cc:729] T d81dcfefb4fd401b9cfd2ca36034a010 P 6398c8993c3c45dcaab25b42142d5671 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6398c8993c3c45dcaab25b42142d5671, State: Initialized, Role: FOLLOWER
I20240503 02:00:25.491602 18697 consensus_queue.cc:260] T d81dcfefb4fd401b9cfd2ca36034a010 P 6398c8993c3c45dcaab25b42142d5671 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:25.491904 18697 ts_tablet_manager.cc:1398] T d81dcfefb4fd401b9cfd2ca36034a010 P 6398c8993c3c45dcaab25b42142d5671: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:25.492069 18697 tablet_bootstrap.cc:492] T a39f284167e942188c181147e8a9e74e P 6398c8993c3c45dcaab25b42142d5671: Bootstrap starting.
I20240503 02:00:25.492894 18697 tablet_bootstrap.cc:654] T a39f284167e942188c181147e8a9e74e P 6398c8993c3c45dcaab25b42142d5671: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:25.493456 18696 tablet_bootstrap.cc:492] T ae4259dd3dc745869256269632886f8a P 89dd09f8aaee4b249ae3ce924a422a67: No bootstrap required, opened a new log
I20240503 02:00:25.493577 18696 ts_tablet_manager.cc:1367] T ae4259dd3dc745869256269632886f8a P 89dd09f8aaee4b249ae3ce924a422a67: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 02:00:25.494112 18697 tablet_bootstrap.cc:492] T a39f284167e942188c181147e8a9e74e P 6398c8993c3c45dcaab25b42142d5671: No bootstrap required, opened a new log
I20240503 02:00:25.494206 18697 ts_tablet_manager.cc:1367] T a39f284167e942188c181147e8a9e74e P 6398c8993c3c45dcaab25b42142d5671: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:25.494549 18698 tablet_bootstrap.cc:492] T d81dcfefb4fd401b9cfd2ca36034a010 P 16d437fe629643948c95d628926d5ffa: No bootstrap required, opened a new log
I20240503 02:00:25.494572 18705 tablet_bootstrap.cc:492] T d81dcfefb4fd401b9cfd2ca36034a010 P 5b8291b26a7f448b8636187d1caf86d0: No bootstrap required, opened a new log
I20240503 02:00:25.494655 18698 ts_tablet_manager.cc:1367] T d81dcfefb4fd401b9cfd2ca36034a010 P 16d437fe629643948c95d628926d5ffa: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 02:00:25.494616 18697 raft_consensus.cc:348] T a39f284167e942188c181147e8a9e74e P 6398c8993c3c45dcaab25b42142d5671 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:25.494745 18697 raft_consensus.cc:374] T a39f284167e942188c181147e8a9e74e P 6398c8993c3c45dcaab25b42142d5671 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:25.494789 18697 raft_consensus.cc:729] T a39f284167e942188c181147e8a9e74e P 6398c8993c3c45dcaab25b42142d5671 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6398c8993c3c45dcaab25b42142d5671, State: Initialized, Role: FOLLOWER
I20240503 02:00:25.494915 18705 ts_tablet_manager.cc:1367] T d81dcfefb4fd401b9cfd2ca36034a010 P 5b8291b26a7f448b8636187d1caf86d0: Time spent bootstrapping tablet: real 0.027s user 0.002s sys 0.000s
I20240503 02:00:25.494925 18697 consensus_queue.cc:260] T a39f284167e942188c181147e8a9e74e P 6398c8993c3c45dcaab25b42142d5671 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:25.495208 18697 ts_tablet_manager.cc:1398] T a39f284167e942188c181147e8a9e74e P 6398c8993c3c45dcaab25b42142d5671: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:25.495402 18461 raft_consensus.cc:482] T ae4259dd3dc745869256269632886f8a P 6398c8993c3c45dcaab25b42142d5671 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:25.495486 18461 raft_consensus.cc:504] T ae4259dd3dc745869256269632886f8a P 6398c8993c3c45dcaab25b42142d5671 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:25.495051 18698 raft_consensus.cc:348] T d81dcfefb4fd401b9cfd2ca36034a010 P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:25.496016 18698 raft_consensus.cc:374] T d81dcfefb4fd401b9cfd2ca36034a010 P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:25.496062 18698 raft_consensus.cc:729] T d81dcfefb4fd401b9cfd2ca36034a010 P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16d437fe629643948c95d628926d5ffa, State: Initialized, Role: FOLLOWER
I20240503 02:00:25.496152 18461 leader_election.cc:288] T ae4259dd3dc745869256269632886f8a P 6398c8993c3c45dcaab25b42142d5671 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 16d437fe629643948c95d628926d5ffa (127.0.99.133:46581), d43ed6b8663947d99a72cd922cc323aa (127.0.99.132:40023), c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134:44343), 002cc3411acb4fdfa0b67c6f57f687a6 (127.0.99.135:38989), 89dd09f8aaee4b249ae3ce924a422a67 (127.0.99.130:38889), 5b8291b26a7f448b8636187d1caf86d0 (127.0.99.131:44479)
I20240503 02:00:25.496215 18698 consensus_queue.cc:260] T d81dcfefb4fd401b9cfd2ca36034a010 P 16d437fe629643948c95d628926d5ffa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:25.496604 17354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae4259dd3dc745869256269632886f8a" candidate_uuid: "6398c8993c3c45dcaab25b42142d5671" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d43ed6b8663947d99a72cd922cc323aa" is_pre_election: true
I20240503 02:00:25.496628 17222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae4259dd3dc745869256269632886f8a" candidate_uuid: "6398c8993c3c45dcaab25b42142d5671" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b8291b26a7f448b8636187d1caf86d0" is_pre_election: true
I20240503 02:00:25.496728 17091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae4259dd3dc745869256269632886f8a" candidate_uuid: "6398c8993c3c45dcaab25b42142d5671" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89dd09f8aaee4b249ae3ce924a422a67" is_pre_election: true
I20240503 02:00:25.496778 17354 raft_consensus.cc:2454] T ae4259dd3dc745869256269632886f8a P d43ed6b8663947d99a72cd922cc323aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6398c8993c3c45dcaab25b42142d5671 in term 0.
W20240503 02:00:25.497148 16894 leader_election.cc:341] T ae4259dd3dc745869256269632886f8a P 6398c8993c3c45dcaab25b42142d5671 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b8291b26a7f448b8636187d1caf86d0 (127.0.99.131:44479): Illegal state: must be running to vote when last-logged opid is not known
W20240503 02:00:25.497375 16896 leader_election.cc:341] T ae4259dd3dc745869256269632886f8a P 6398c8993c3c45dcaab25b42142d5671 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89dd09f8aaee4b249ae3ce924a422a67 (127.0.99.130:38889): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:00:25.497763 17488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae4259dd3dc745869256269632886f8a" candidate_uuid: "6398c8993c3c45dcaab25b42142d5671" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16d437fe629643948c95d628926d5ffa" is_pre_election: true
I20240503 02:00:25.498070 17618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae4259dd3dc745869256269632886f8a" candidate_uuid: "6398c8993c3c45dcaab25b42142d5671" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c843193b801b4f1a88f630ed3ed0f375" is_pre_election: true
W20240503 02:00:25.498205 16893 leader_election.cc:341] T ae4259dd3dc745869256269632886f8a P 6398c8993c3c45dcaab25b42142d5671 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 16d437fe629643948c95d628926d5ffa (127.0.99.133:46581): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:00:25.498178 17750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae4259dd3dc745869256269632886f8a" candidate_uuid: "6398c8993c3c45dcaab25b42142d5671" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" is_pre_election: true
I20240503 02:00:25.498324 17750 raft_consensus.cc:2454] T ae4259dd3dc745869256269632886f8a P 002cc3411acb4fdfa0b67c6f57f687a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6398c8993c3c45dcaab25b42142d5671 in term 0.
W20240503 02:00:25.498543 16893 leader_election.cc:341] T ae4259dd3dc745869256269632886f8a P 6398c8993c3c45dcaab25b42142d5671 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134:44343): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:00:25.498607 16893 leader_election.cc:302] T ae4259dd3dc745869256269632886f8a P 6398c8993c3c45dcaab25b42142d5671 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 6 responses out of 7 voters: 2 yes votes; 4 no votes. yes voters: 6398c8993c3c45dcaab25b42142d5671, d43ed6b8663947d99a72cd922cc323aa; no voters: 16d437fe629643948c95d628926d5ffa, 5b8291b26a7f448b8636187d1caf86d0, 89dd09f8aaee4b249ae3ce924a422a67, c843193b801b4f1a88f630ed3ed0f375
I20240503 02:00:25.498757 18696 raft_consensus.cc:348] T ae4259dd3dc745869256269632886f8a P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:25.498868 18696 raft_consensus.cc:374] T ae4259dd3dc745869256269632886f8a P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:25.498899 18696 raft_consensus.cc:729] T ae4259dd3dc745869256269632886f8a P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89dd09f8aaee4b249ae3ce924a422a67, State: Initialized, Role: FOLLOWER
I20240503 02:00:25.498943 18461 raft_consensus.cc:2726] T ae4259dd3dc745869256269632886f8a P 6398c8993c3c45dcaab25b42142d5671 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:00:25.499054 18696 consensus_queue.cc:260] T ae4259dd3dc745869256269632886f8a P 89dd09f8aaee4b249ae3ce924a422a67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:25.499382 18696 ts_tablet_manager.cc:1398] T ae4259dd3dc745869256269632886f8a P 89dd09f8aaee4b249ae3ce924a422a67: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240503 02:00:25.499583 18696 tablet_bootstrap.cc:492] T a39f284167e942188c181147e8a9e74e P 89dd09f8aaee4b249ae3ce924a422a67: Bootstrap starting.
I20240503 02:00:25.500581 18698 ts_tablet_manager.cc:1398] T d81dcfefb4fd401b9cfd2ca36034a010 P 16d437fe629643948c95d628926d5ffa: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 02:00:25.500777 18698 tablet_bootstrap.cc:492] T ae4259dd3dc745869256269632886f8a P 16d437fe629643948c95d628926d5ffa: Bootstrap starting.
I20240503 02:00:25.500792 18696 tablet_bootstrap.cc:654] T a39f284167e942188c181147e8a9e74e P 89dd09f8aaee4b249ae3ce924a422a67: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:25.501911 18696 tablet_bootstrap.cc:492] T a39f284167e942188c181147e8a9e74e P 89dd09f8aaee4b249ae3ce924a422a67: No bootstrap required, opened a new log
I20240503 02:00:25.501994 18696 ts_tablet_manager.cc:1367] T a39f284167e942188c181147e8a9e74e P 89dd09f8aaee4b249ae3ce924a422a67: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 02:00:25.502424 18696 raft_consensus.cc:348] T a39f284167e942188c181147e8a9e74e P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:25.502547 18696 raft_consensus.cc:374] T a39f284167e942188c181147e8a9e74e P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:25.502588 18696 raft_consensus.cc:729] T a39f284167e942188c181147e8a9e74e P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89dd09f8aaee4b249ae3ce924a422a67, State: Initialized, Role: FOLLOWER
I20240503 02:00:25.502609 18698 tablet_bootstrap.cc:654] T ae4259dd3dc745869256269632886f8a P 16d437fe629643948c95d628926d5ffa: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:25.502738 18696 consensus_queue.cc:260] T a39f284167e942188c181147e8a9e74e P 89dd09f8aaee4b249ae3ce924a422a67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:25.503055 18696 ts_tablet_manager.cc:1398] T a39f284167e942188c181147e8a9e74e P 89dd09f8aaee4b249ae3ce924a422a67: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:25.503726 18705 raft_consensus.cc:348] T d81dcfefb4fd401b9cfd2ca36034a010 P 5b8291b26a7f448b8636187d1caf86d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:25.503852 18705 raft_consensus.cc:374] T d81dcfefb4fd401b9cfd2ca36034a010 P 5b8291b26a7f448b8636187d1caf86d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:25.503876 18705 raft_consensus.cc:729] T d81dcfefb4fd401b9cfd2ca36034a010 P 5b8291b26a7f448b8636187d1caf86d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b8291b26a7f448b8636187d1caf86d0, State: Initialized, Role: FOLLOWER
I20240503 02:00:25.504024 18705 consensus_queue.cc:260] T d81dcfefb4fd401b9cfd2ca36034a010 P 5b8291b26a7f448b8636187d1caf86d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:25.504427 18705 ts_tablet_manager.cc:1398] T d81dcfefb4fd401b9cfd2ca36034a010 P 5b8291b26a7f448b8636187d1caf86d0: Time spent starting tablet: real 0.009s user 0.001s sys 0.000s
I20240503 02:00:25.504622 18705 tablet_bootstrap.cc:492] T ae4259dd3dc745869256269632886f8a P 5b8291b26a7f448b8636187d1caf86d0: Bootstrap starting.
I20240503 02:00:25.505513 18705 tablet_bootstrap.cc:654] T ae4259dd3dc745869256269632886f8a P 5b8291b26a7f448b8636187d1caf86d0: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:25.506520 18705 tablet_bootstrap.cc:492] T ae4259dd3dc745869256269632886f8a P 5b8291b26a7f448b8636187d1caf86d0: No bootstrap required, opened a new log
I20240503 02:00:25.506615 18705 ts_tablet_manager.cc:1367] T ae4259dd3dc745869256269632886f8a P 5b8291b26a7f448b8636187d1caf86d0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:25.507038 18705 raft_consensus.cc:348] T ae4259dd3dc745869256269632886f8a P 5b8291b26a7f448b8636187d1caf86d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:25.507138 18705 raft_consensus.cc:374] T ae4259dd3dc745869256269632886f8a P 5b8291b26a7f448b8636187d1caf86d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:25.507167 18705 raft_consensus.cc:729] T ae4259dd3dc745869256269632886f8a P 5b8291b26a7f448b8636187d1caf86d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b8291b26a7f448b8636187d1caf86d0, State: Initialized, Role: FOLLOWER
I20240503 02:00:25.507205 18698 tablet_bootstrap.cc:492] T ae4259dd3dc745869256269632886f8a P 16d437fe629643948c95d628926d5ffa: No bootstrap required, opened a new log
I20240503 02:00:25.507324 18698 ts_tablet_manager.cc:1367] T ae4259dd3dc745869256269632886f8a P 16d437fe629643948c95d628926d5ffa: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 02:00:25.507292 18705 consensus_queue.cc:260] T ae4259dd3dc745869256269632886f8a P 5b8291b26a7f448b8636187d1caf86d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:25.507596 18705 ts_tablet_manager.cc:1398] T ae4259dd3dc745869256269632886f8a P 5b8291b26a7f448b8636187d1caf86d0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:25.507787 18705 tablet_bootstrap.cc:492] T a39f284167e942188c181147e8a9e74e P 5b8291b26a7f448b8636187d1caf86d0: Bootstrap starting.
I20240503 02:00:25.507880 18698 raft_consensus.cc:348] T ae4259dd3dc745869256269632886f8a P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:25.508033 18698 raft_consensus.cc:374] T ae4259dd3dc745869256269632886f8a P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:25.508072 18698 raft_consensus.cc:729] T ae4259dd3dc745869256269632886f8a P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16d437fe629643948c95d628926d5ffa, State: Initialized, Role: FOLLOWER
I20240503 02:00:25.508220 18698 consensus_queue.cc:260] T ae4259dd3dc745869256269632886f8a P 16d437fe629643948c95d628926d5ffa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:25.508575 18705 tablet_bootstrap.cc:654] T a39f284167e942188c181147e8a9e74e P 5b8291b26a7f448b8636187d1caf86d0: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:25.508563 18698 ts_tablet_manager.cc:1398] T ae4259dd3dc745869256269632886f8a P 16d437fe629643948c95d628926d5ffa: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 02:00:25.508984 18704 tablet_bootstrap.cc:492] T a39f284167e942188c181147e8a9e74e P c843193b801b4f1a88f630ed3ed0f375: No bootstrap required, opened a new log
I20240503 02:00:25.509091 18704 ts_tablet_manager.cc:1367] T a39f284167e942188c181147e8a9e74e P c843193b801b4f1a88f630ed3ed0f375: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.000s
I20240503 02:00:25.509465 18704 raft_consensus.cc:348] T a39f284167e942188c181147e8a9e74e P c843193b801b4f1a88f630ed3ed0f375 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:25.509550 18704 raft_consensus.cc:374] T a39f284167e942188c181147e8a9e74e P c843193b801b4f1a88f630ed3ed0f375 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:25.509570 18704 raft_consensus.cc:729] T a39f284167e942188c181147e8a9e74e P c843193b801b4f1a88f630ed3ed0f375 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c843193b801b4f1a88f630ed3ed0f375, State: Initialized, Role: FOLLOWER
I20240503 02:00:25.509685 18704 consensus_queue.cc:260] T a39f284167e942188c181147e8a9e74e P c843193b801b4f1a88f630ed3ed0f375 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:25.510021 18704 ts_tablet_manager.cc:1398] T a39f284167e942188c181147e8a9e74e P c843193b801b4f1a88f630ed3ed0f375: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:25.510211 18704 tablet_bootstrap.cc:492] T ae4259dd3dc745869256269632886f8a P c843193b801b4f1a88f630ed3ed0f375: Bootstrap starting.
I20240503 02:00:25.510442 18705 tablet_bootstrap.cc:492] T a39f284167e942188c181147e8a9e74e P 5b8291b26a7f448b8636187d1caf86d0: No bootstrap required, opened a new log
I20240503 02:00:25.510507 18705 ts_tablet_manager.cc:1367] T a39f284167e942188c181147e8a9e74e P 5b8291b26a7f448b8636187d1caf86d0: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 02:00:25.510902 18705 raft_consensus.cc:348] T a39f284167e942188c181147e8a9e74e P 5b8291b26a7f448b8636187d1caf86d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:25.511049 18705 raft_consensus.cc:374] T a39f284167e942188c181147e8a9e74e P 5b8291b26a7f448b8636187d1caf86d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:25.511091 18704 tablet_bootstrap.cc:654] T ae4259dd3dc745869256269632886f8a P c843193b801b4f1a88f630ed3ed0f375: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:25.511103 18705 raft_consensus.cc:729] T a39f284167e942188c181147e8a9e74e P 5b8291b26a7f448b8636187d1caf86d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b8291b26a7f448b8636187d1caf86d0, State: Initialized, Role: FOLLOWER
I20240503 02:00:25.511345 18705 consensus_queue.cc:260] T a39f284167e942188c181147e8a9e74e P 5b8291b26a7f448b8636187d1caf86d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:25.511727 18705 ts_tablet_manager.cc:1398] T a39f284167e942188c181147e8a9e74e P 5b8291b26a7f448b8636187d1caf86d0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:25.512302 18704 tablet_bootstrap.cc:492] T ae4259dd3dc745869256269632886f8a P c843193b801b4f1a88f630ed3ed0f375: No bootstrap required, opened a new log
I20240503 02:00:25.512390 18704 ts_tablet_manager.cc:1367] T ae4259dd3dc745869256269632886f8a P c843193b801b4f1a88f630ed3ed0f375: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:25.512830 18704 raft_consensus.cc:348] T ae4259dd3dc745869256269632886f8a P c843193b801b4f1a88f630ed3ed0f375 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:25.512923 18704 raft_consensus.cc:374] T ae4259dd3dc745869256269632886f8a P c843193b801b4f1a88f630ed3ed0f375 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:25.512954 18704 raft_consensus.cc:729] T ae4259dd3dc745869256269632886f8a P c843193b801b4f1a88f630ed3ed0f375 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c843193b801b4f1a88f630ed3ed0f375, State: Initialized, Role: FOLLOWER
I20240503 02:00:25.513087 18704 consensus_queue.cc:260] T ae4259dd3dc745869256269632886f8a P c843193b801b4f1a88f630ed3ed0f375 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:25.513353 18704 ts_tablet_manager.cc:1398] T ae4259dd3dc745869256269632886f8a P c843193b801b4f1a88f630ed3ed0f375: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:25.522948 18678 raft_consensus.cc:482] T a39f284167e942188c181147e8a9e74e P d43ed6b8663947d99a72cd922cc323aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:25.523101 18678 raft_consensus.cc:504] T a39f284167e942188c181147e8a9e74e P d43ed6b8663947d99a72cd922cc323aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:25.523816 18678 leader_election.cc:288] T a39f284167e942188c181147e8a9e74e P d43ed6b8663947d99a72cd922cc323aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 16d437fe629643948c95d628926d5ffa (127.0.99.133:46581), c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134:44343), 002cc3411acb4fdfa0b67c6f57f687a6 (127.0.99.135:38989), 6398c8993c3c45dcaab25b42142d5671 (127.0.99.129:36983), 89dd09f8aaee4b249ae3ce924a422a67 (127.0.99.130:38889), 5b8291b26a7f448b8636187d1caf86d0 (127.0.99.131:44479)
I20240503 02:00:25.524257 17091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a39f284167e942188c181147e8a9e74e" candidate_uuid: "d43ed6b8663947d99a72cd922cc323aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89dd09f8aaee4b249ae3ce924a422a67" is_pre_election: true
I20240503 02:00:25.524248 17488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a39f284167e942188c181147e8a9e74e" candidate_uuid: "d43ed6b8663947d99a72cd922cc323aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16d437fe629643948c95d628926d5ffa" is_pre_election: true
I20240503 02:00:25.524289 17222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a39f284167e942188c181147e8a9e74e" candidate_uuid: "d43ed6b8663947d99a72cd922cc323aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b8291b26a7f448b8636187d1caf86d0" is_pre_election: true
I20240503 02:00:25.524421 17091 raft_consensus.cc:2454] T a39f284167e942188c181147e8a9e74e P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d43ed6b8663947d99a72cd922cc323aa in term 0.
I20240503 02:00:25.524421 17488 raft_consensus.cc:2454] T a39f284167e942188c181147e8a9e74e P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d43ed6b8663947d99a72cd922cc323aa in term 0.
I20240503 02:00:25.524443 17222 raft_consensus.cc:2454] T a39f284167e942188c181147e8a9e74e P 5b8291b26a7f448b8636187d1caf86d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d43ed6b8663947d99a72cd922cc323aa in term 0.
I20240503 02:00:25.524513 17750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a39f284167e942188c181147e8a9e74e" candidate_uuid: "d43ed6b8663947d99a72cd922cc323aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" is_pre_election: true
I20240503 02:00:25.524699 17750 raft_consensus.cc:2454] T a39f284167e942188c181147e8a9e74e P 002cc3411acb4fdfa0b67c6f57f687a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d43ed6b8663947d99a72cd922cc323aa in term 0.
I20240503 02:00:25.524765 16960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a39f284167e942188c181147e8a9e74e" candidate_uuid: "d43ed6b8663947d99a72cd922cc323aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6398c8993c3c45dcaab25b42142d5671" is_pre_election: true
I20240503 02:00:25.524906 16960 raft_consensus.cc:2454] T a39f284167e942188c181147e8a9e74e P 6398c8993c3c45dcaab25b42142d5671 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d43ed6b8663947d99a72cd922cc323aa in term 0.
I20240503 02:00:25.525121 17292 leader_election.cc:302] T a39f284167e942188c181147e8a9e74e P d43ed6b8663947d99a72cd922cc323aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 16d437fe629643948c95d628926d5ffa, 5b8291b26a7f448b8636187d1caf86d0, 89dd09f8aaee4b249ae3ce924a422a67, d43ed6b8663947d99a72cd922cc323aa; no voters:
I20240503 02:00:25.525221 17618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a39f284167e942188c181147e8a9e74e" candidate_uuid: "d43ed6b8663947d99a72cd922cc323aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c843193b801b4f1a88f630ed3ed0f375" is_pre_election: true
I20240503 02:00:25.525337 18678 raft_consensus.cc:2781] T a39f284167e942188c181147e8a9e74e P d43ed6b8663947d99a72cd922cc323aa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:25.525367 17618 raft_consensus.cc:2454] T a39f284167e942188c181147e8a9e74e P c843193b801b4f1a88f630ed3ed0f375 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d43ed6b8663947d99a72cd922cc323aa in term 0.
I20240503 02:00:25.525422 18678 raft_consensus.cc:482] T a39f284167e942188c181147e8a9e74e P d43ed6b8663947d99a72cd922cc323aa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:25.525480 18678 raft_consensus.cc:3037] T a39f284167e942188c181147e8a9e74e P d43ed6b8663947d99a72cd922cc323aa [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:25.526573 18678 raft_consensus.cc:504] T a39f284167e942188c181147e8a9e74e P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:25.527298 18678 leader_election.cc:288] T a39f284167e942188c181147e8a9e74e P d43ed6b8663947d99a72cd922cc323aa [CANDIDATE]: Term 1 election: Requested vote from peers 16d437fe629643948c95d628926d5ffa (127.0.99.133:46581), c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134:44343), 002cc3411acb4fdfa0b67c6f57f687a6 (127.0.99.135:38989), 6398c8993c3c45dcaab25b42142d5671 (127.0.99.129:36983), 89dd09f8aaee4b249ae3ce924a422a67 (127.0.99.130:38889), 5b8291b26a7f448b8636187d1caf86d0 (127.0.99.131:44479)
I20240503 02:00:25.527726 16960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a39f284167e942188c181147e8a9e74e" candidate_uuid: "d43ed6b8663947d99a72cd922cc323aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6398c8993c3c45dcaab25b42142d5671"
I20240503 02:00:25.527765 17222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a39f284167e942188c181147e8a9e74e" candidate_uuid: "d43ed6b8663947d99a72cd922cc323aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b8291b26a7f448b8636187d1caf86d0"
I20240503 02:00:25.527855 16960 raft_consensus.cc:3037] T a39f284167e942188c181147e8a9e74e P 6398c8993c3c45dcaab25b42142d5671 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:25.527874 17222 raft_consensus.cc:3037] T a39f284167e942188c181147e8a9e74e P 5b8291b26a7f448b8636187d1caf86d0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:25.527909 17488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a39f284167e942188c181147e8a9e74e" candidate_uuid: "d43ed6b8663947d99a72cd922cc323aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16d437fe629643948c95d628926d5ffa"
I20240503 02:00:25.528016 17091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a39f284167e942188c181147e8a9e74e" candidate_uuid: "d43ed6b8663947d99a72cd922cc323aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89dd09f8aaee4b249ae3ce924a422a67"
I20240503 02:00:25.528107 17488 raft_consensus.cc:3037] T a39f284167e942188c181147e8a9e74e P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:25.528121 17091 raft_consensus.cc:3037] T a39f284167e942188c181147e8a9e74e P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:25.529301 17091 raft_consensus.cc:2454] T a39f284167e942188c181147e8a9e74e P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d43ed6b8663947d99a72cd922cc323aa in term 1.
I20240503 02:00:25.529301 16960 raft_consensus.cc:2454] T a39f284167e942188c181147e8a9e74e P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d43ed6b8663947d99a72cd922cc323aa in term 1.
I20240503 02:00:25.529314 17222 raft_consensus.cc:2454] T a39f284167e942188c181147e8a9e74e P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d43ed6b8663947d99a72cd922cc323aa in term 1.
I20240503 02:00:25.529358 17488 raft_consensus.cc:2454] T a39f284167e942188c181147e8a9e74e P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d43ed6b8663947d99a72cd922cc323aa in term 1.
I20240503 02:00:25.529843 17291 leader_election.cc:302] T a39f284167e942188c181147e8a9e74e P d43ed6b8663947d99a72cd922cc323aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 5b8291b26a7f448b8636187d1caf86d0, 6398c8993c3c45dcaab25b42142d5671, 89dd09f8aaee4b249ae3ce924a422a67, d43ed6b8663947d99a72cd922cc323aa; no voters:
I20240503 02:00:25.530117 18678 raft_consensus.cc:2781] T a39f284167e942188c181147e8a9e74e P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:25.530112 17618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a39f284167e942188c181147e8a9e74e" candidate_uuid: "d43ed6b8663947d99a72cd922cc323aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c843193b801b4f1a88f630ed3ed0f375"
I20240503 02:00:25.530225 17618 raft_consensus.cc:3037] T a39f284167e942188c181147e8a9e74e P c843193b801b4f1a88f630ed3ed0f375 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:25.530310 18678 raft_consensus.cc:686] T a39f284167e942188c181147e8a9e74e P d43ed6b8663947d99a72cd922cc323aa [term 1 LEADER]: Becoming Leader. State: Replica: d43ed6b8663947d99a72cd922cc323aa, State: Running, Role: LEADER
I20240503 02:00:25.530313 17750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a39f284167e942188c181147e8a9e74e" candidate_uuid: "d43ed6b8663947d99a72cd922cc323aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "002cc3411acb4fdfa0b67c6f57f687a6"
I20240503 02:00:25.530436 17750 raft_consensus.cc:3037] T a39f284167e942188c181147e8a9e74e P 002cc3411acb4fdfa0b67c6f57f687a6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:25.530496 18678 consensus_queue.cc:237] T a39f284167e942188c181147e8a9e74e P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:25.531347 17750 raft_consensus.cc:2454] T a39f284167e942188c181147e8a9e74e P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d43ed6b8663947d99a72cd922cc323aa in term 1.
I20240503 02:00:25.531447 17618 raft_consensus.cc:2454] T a39f284167e942188c181147e8a9e74e P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d43ed6b8663947d99a72cd922cc323aa in term 1.
I20240503 02:00:25.532931 16817 catalog_manager.cc:5484] T a39f284167e942188c181147e8a9e74e P d43ed6b8663947d99a72cd922cc323aa reported cstate change: term changed from 0 to 1, leader changed from <none> to d43ed6b8663947d99a72cd922cc323aa (127.0.99.132). New cstate: current_term: 1 leader_uuid: "d43ed6b8663947d99a72cd922cc323aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:25.566017 18561 raft_consensus.cc:482] T ae4259dd3dc745869256269632886f8a P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:25.566217 18561 raft_consensus.cc:504] T ae4259dd3dc745869256269632886f8a P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:25.566994 18561 leader_election.cc:288] T ae4259dd3dc745869256269632886f8a P 89dd09f8aaee4b249ae3ce924a422a67 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 16d437fe629643948c95d628926d5ffa (127.0.99.133:46581), d43ed6b8663947d99a72cd922cc323aa (127.0.99.132:40023), c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134:44343), 002cc3411acb4fdfa0b67c6f57f687a6 (127.0.99.135:38989), 6398c8993c3c45dcaab25b42142d5671 (127.0.99.129:36983), 5b8291b26a7f448b8636187d1caf86d0 (127.0.99.131:44479)
I20240503 02:00:25.567368 16960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae4259dd3dc745869256269632886f8a" candidate_uuid: "89dd09f8aaee4b249ae3ce924a422a67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6398c8993c3c45dcaab25b42142d5671" is_pre_election: true
I20240503 02:00:25.567451 17618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae4259dd3dc745869256269632886f8a" candidate_uuid: "89dd09f8aaee4b249ae3ce924a422a67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c843193b801b4f1a88f630ed3ed0f375" is_pre_election: true
I20240503 02:00:25.567519 17488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae4259dd3dc745869256269632886f8a" candidate_uuid: "89dd09f8aaee4b249ae3ce924a422a67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16d437fe629643948c95d628926d5ffa" is_pre_election: true
I20240503 02:00:25.567559 16960 raft_consensus.cc:2454] T ae4259dd3dc745869256269632886f8a P 6398c8993c3c45dcaab25b42142d5671 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89dd09f8aaee4b249ae3ce924a422a67 in term 0.
I20240503 02:00:25.567622 17618 raft_consensus.cc:2454] T ae4259dd3dc745869256269632886f8a P c843193b801b4f1a88f630ed3ed0f375 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89dd09f8aaee4b249ae3ce924a422a67 in term 0.
I20240503 02:00:25.567687 17488 raft_consensus.cc:2454] T ae4259dd3dc745869256269632886f8a P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89dd09f8aaee4b249ae3ce924a422a67 in term 0.
I20240503 02:00:25.567682 17354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae4259dd3dc745869256269632886f8a" candidate_uuid: "89dd09f8aaee4b249ae3ce924a422a67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d43ed6b8663947d99a72cd922cc323aa" is_pre_election: true
I20240503 02:00:25.567842 17354 raft_consensus.cc:2454] T ae4259dd3dc745869256269632886f8a P d43ed6b8663947d99a72cd922cc323aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89dd09f8aaee4b249ae3ce924a422a67 in term 0.
I20240503 02:00:25.567946 17222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae4259dd3dc745869256269632886f8a" candidate_uuid: "89dd09f8aaee4b249ae3ce924a422a67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b8291b26a7f448b8636187d1caf86d0" is_pre_election: true
I20240503 02:00:25.568113 17222 raft_consensus.cc:2454] T ae4259dd3dc745869256269632886f8a P 5b8291b26a7f448b8636187d1caf86d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89dd09f8aaee4b249ae3ce924a422a67 in term 0.
I20240503 02:00:25.568176 17026 leader_election.cc:302] T ae4259dd3dc745869256269632886f8a P 89dd09f8aaee4b249ae3ce924a422a67 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 5 responses out of 7 voters: 5 yes votes; 0 no votes. yes voters: 16d437fe629643948c95d628926d5ffa, 6398c8993c3c45dcaab25b42142d5671, 89dd09f8aaee4b249ae3ce924a422a67, c843193b801b4f1a88f630ed3ed0f375, d43ed6b8663947d99a72cd922cc323aa; no voters:
I20240503 02:00:25.568437 18561 raft_consensus.cc:2781] T ae4259dd3dc745869256269632886f8a P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:25.568419 17750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae4259dd3dc745869256269632886f8a" candidate_uuid: "89dd09f8aaee4b249ae3ce924a422a67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" is_pre_election: true
I20240503 02:00:25.568495 18561 raft_consensus.cc:482] T ae4259dd3dc745869256269632886f8a P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:25.568555 18561 raft_consensus.cc:3037] T ae4259dd3dc745869256269632886f8a P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:25.568576 17750 raft_consensus.cc:2454] T ae4259dd3dc745869256269632886f8a P 002cc3411acb4fdfa0b67c6f57f687a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89dd09f8aaee4b249ae3ce924a422a67 in term 0.
I20240503 02:00:25.569623 18561 raft_consensus.cc:504] T ae4259dd3dc745869256269632886f8a P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:25.570484 17354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae4259dd3dc745869256269632886f8a" candidate_uuid: "89dd09f8aaee4b249ae3ce924a422a67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d43ed6b8663947d99a72cd922cc323aa"
I20240503 02:00:25.570621 17354 raft_consensus.cc:3037] T ae4259dd3dc745869256269632886f8a P d43ed6b8663947d99a72cd922cc323aa [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:25.570619 17222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae4259dd3dc745869256269632886f8a" candidate_uuid: "89dd09f8aaee4b249ae3ce924a422a67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b8291b26a7f448b8636187d1caf86d0"
I20240503 02:00:25.570721 17222 raft_consensus.cc:3037] T ae4259dd3dc745869256269632886f8a P 5b8291b26a7f448b8636187d1caf86d0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:25.570773 17750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae4259dd3dc745869256269632886f8a" candidate_uuid: "89dd09f8aaee4b249ae3ce924a422a67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "002cc3411acb4fdfa0b67c6f57f687a6"
I20240503 02:00:25.570760 16960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae4259dd3dc745869256269632886f8a" candidate_uuid: "89dd09f8aaee4b249ae3ce924a422a67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6398c8993c3c45dcaab25b42142d5671"
I20240503 02:00:25.570858 16960 raft_consensus.cc:3037] T ae4259dd3dc745869256269632886f8a P 6398c8993c3c45dcaab25b42142d5671 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:25.570869 17750 raft_consensus.cc:3037] T ae4259dd3dc745869256269632886f8a P 002cc3411acb4fdfa0b67c6f57f687a6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:25.571779 17354 raft_consensus.cc:2454] T ae4259dd3dc745869256269632886f8a P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89dd09f8aaee4b249ae3ce924a422a67 in term 1.
I20240503 02:00:25.571779 17222 raft_consensus.cc:2454] T ae4259dd3dc745869256269632886f8a P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89dd09f8aaee4b249ae3ce924a422a67 in term 1.
I20240503 02:00:25.571900 16960 raft_consensus.cc:2454] T ae4259dd3dc745869256269632886f8a P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89dd09f8aaee4b249ae3ce924a422a67 in term 1.
I20240503 02:00:25.571908 17750 raft_consensus.cc:2454] T ae4259dd3dc745869256269632886f8a P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89dd09f8aaee4b249ae3ce924a422a67 in term 1.
I20240503 02:00:25.572142 17618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae4259dd3dc745869256269632886f8a" candidate_uuid: "89dd09f8aaee4b249ae3ce924a422a67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c843193b801b4f1a88f630ed3ed0f375"
I20240503 02:00:25.572285 17618 raft_consensus.cc:3037] T ae4259dd3dc745869256269632886f8a P c843193b801b4f1a88f630ed3ed0f375 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:25.572355 17488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae4259dd3dc745869256269632886f8a" candidate_uuid: "89dd09f8aaee4b249ae3ce924a422a67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16d437fe629643948c95d628926d5ffa"
I20240503 02:00:25.572453 17488 raft_consensus.cc:3037] T ae4259dd3dc745869256269632886f8a P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:25.573240 17618 raft_consensus.cc:2454] T ae4259dd3dc745869256269632886f8a P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89dd09f8aaee4b249ae3ce924a422a67 in term 1.
I20240503 02:00:25.573320 17488 raft_consensus.cc:2454] T ae4259dd3dc745869256269632886f8a P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89dd09f8aaee4b249ae3ce924a422a67 in term 1.
I20240503 02:00:25.573447 17026 leader_election.cc:302] T ae4259dd3dc745869256269632886f8a P 89dd09f8aaee4b249ae3ce924a422a67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 002cc3411acb4fdfa0b67c6f57f687a6, 6398c8993c3c45dcaab25b42142d5671, 89dd09f8aaee4b249ae3ce924a422a67, d43ed6b8663947d99a72cd922cc323aa; no voters:
I20240503 02:00:25.573594 18561 leader_election.cc:288] T ae4259dd3dc745869256269632886f8a P 89dd09f8aaee4b249ae3ce924a422a67 [CANDIDATE]: Term 1 election: Requested vote from peers 16d437fe629643948c95d628926d5ffa (127.0.99.133:46581), d43ed6b8663947d99a72cd922cc323aa (127.0.99.132:40023), c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134:44343), 002cc3411acb4fdfa0b67c6f57f687a6 (127.0.99.135:38989), 6398c8993c3c45dcaab25b42142d5671 (127.0.99.129:36983), 5b8291b26a7f448b8636187d1caf86d0 (127.0.99.131:44479)
I20240503 02:00:25.573679 18560 raft_consensus.cc:2781] T ae4259dd3dc745869256269632886f8a P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:25.573901 18560 raft_consensus.cc:686] T ae4259dd3dc745869256269632886f8a P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 LEADER]: Becoming Leader. State: Replica: 89dd09f8aaee4b249ae3ce924a422a67, State: Running, Role: LEADER
I20240503 02:00:25.574115 18560 consensus_queue.cc:237] T ae4259dd3dc745869256269632886f8a P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:25.576720 16817 catalog_manager.cc:5484] T ae4259dd3dc745869256269632886f8a P 89dd09f8aaee4b249ae3ce924a422a67 reported cstate change: term changed from 0 to 1, leader changed from <none> to 89dd09f8aaee4b249ae3ce924a422a67 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "89dd09f8aaee4b249ae3ce924a422a67" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:25.586055 18584 raft_consensus.cc:482] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:25.586247 18584 raft_consensus.cc:504] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:25.587908 18584 leader_election.cc:288] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6398c8993c3c45dcaab25b42142d5671 (127.0.99.129:36983), 89dd09f8aaee4b249ae3ce924a422a67 (127.0.99.130:38889), 5b8291b26a7f448b8636187d1caf86d0 (127.0.99.131:44479), d43ed6b8663947d99a72cd922cc323aa (127.0.99.132:40023), 16d437fe629643948c95d628926d5ffa (127.0.99.133:46581), 002cc3411acb4fdfa0b67c6f57f687a6 (127.0.99.135:38989)
I20240503 02:00:25.593766 17091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d81dcfefb4fd401b9cfd2ca36034a010" candidate_uuid: "c843193b801b4f1a88f630ed3ed0f375" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89dd09f8aaee4b249ae3ce924a422a67" is_pre_election: true
I20240503 02:00:25.593973 17091 raft_consensus.cc:2454] T d81dcfefb4fd401b9cfd2ca36034a010 P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c843193b801b4f1a88f630ed3ed0f375 in term 0.
I20240503 02:00:25.594419 17750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d81dcfefb4fd401b9cfd2ca36034a010" candidate_uuid: "c843193b801b4f1a88f630ed3ed0f375" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" is_pre_election: true
I20240503 02:00:25.594602 17750 raft_consensus.cc:2454] T d81dcfefb4fd401b9cfd2ca36034a010 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c843193b801b4f1a88f630ed3ed0f375 in term 0.
I20240503 02:00:25.596598 16960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d81dcfefb4fd401b9cfd2ca36034a010" candidate_uuid: "c843193b801b4f1a88f630ed3ed0f375" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6398c8993c3c45dcaab25b42142d5671" is_pre_election: true
I20240503 02:00:25.596625 17354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d81dcfefb4fd401b9cfd2ca36034a010" candidate_uuid: "c843193b801b4f1a88f630ed3ed0f375" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d43ed6b8663947d99a72cd922cc323aa" is_pre_election: true
I20240503 02:00:25.596812 17354 raft_consensus.cc:2454] T d81dcfefb4fd401b9cfd2ca36034a010 P d43ed6b8663947d99a72cd922cc323aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c843193b801b4f1a88f630ed3ed0f375 in term 0.
I20240503 02:00:25.596832 16960 raft_consensus.cc:2454] T d81dcfefb4fd401b9cfd2ca36034a010 P 6398c8993c3c45dcaab25b42142d5671 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c843193b801b4f1a88f630ed3ed0f375 in term 0.
I20240503 02:00:25.597292 17555 leader_election.cc:302] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 002cc3411acb4fdfa0b67c6f57f687a6, 6398c8993c3c45dcaab25b42142d5671, 89dd09f8aaee4b249ae3ce924a422a67, c843193b801b4f1a88f630ed3ed0f375; no voters:
I20240503 02:00:25.597635 18584 raft_consensus.cc:2781] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:25.597705 18584 raft_consensus.cc:482] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:25.597756 18584 raft_consensus.cc:3037] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:25.597783 17488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d81dcfefb4fd401b9cfd2ca36034a010" candidate_uuid: "c843193b801b4f1a88f630ed3ed0f375" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16d437fe629643948c95d628926d5ffa" is_pre_election: true
I20240503 02:00:25.597939 17488 raft_consensus.cc:2454] T d81dcfefb4fd401b9cfd2ca36034a010 P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c843193b801b4f1a88f630ed3ed0f375 in term 0.
I20240503 02:00:25.598863 17222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d81dcfefb4fd401b9cfd2ca36034a010" candidate_uuid: "c843193b801b4f1a88f630ed3ed0f375" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b8291b26a7f448b8636187d1caf86d0" is_pre_election: true
I20240503 02:00:25.599011 17222 raft_consensus.cc:2454] T d81dcfefb4fd401b9cfd2ca36034a010 P 5b8291b26a7f448b8636187d1caf86d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c843193b801b4f1a88f630ed3ed0f375 in term 0.
I20240503 02:00:25.599053 18584 raft_consensus.cc:504] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:25.599728 18584 leader_election.cc:288] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 [CANDIDATE]: Term 1 election: Requested vote from peers 6398c8993c3c45dcaab25b42142d5671 (127.0.99.129:36983), 89dd09f8aaee4b249ae3ce924a422a67 (127.0.99.130:38889), 5b8291b26a7f448b8636187d1caf86d0 (127.0.99.131:44479), d43ed6b8663947d99a72cd922cc323aa (127.0.99.132:40023), 16d437fe629643948c95d628926d5ffa (127.0.99.133:46581), 002cc3411acb4fdfa0b67c6f57f687a6 (127.0.99.135:38989)
I20240503 02:00:25.599799 16960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d81dcfefb4fd401b9cfd2ca36034a010" candidate_uuid: "c843193b801b4f1a88f630ed3ed0f375" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6398c8993c3c45dcaab25b42142d5671"
I20240503 02:00:25.599905 16960 raft_consensus.cc:3037] T d81dcfefb4fd401b9cfd2ca36034a010 P 6398c8993c3c45dcaab25b42142d5671 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:25.599951 17091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d81dcfefb4fd401b9cfd2ca36034a010" candidate_uuid: "c843193b801b4f1a88f630ed3ed0f375" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89dd09f8aaee4b249ae3ce924a422a67"
I20240503 02:00:25.600072 17091 raft_consensus.cc:3037] T d81dcfefb4fd401b9cfd2ca36034a010 P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:25.600363 17750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d81dcfefb4fd401b9cfd2ca36034a010" candidate_uuid: "c843193b801b4f1a88f630ed3ed0f375" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "002cc3411acb4fdfa0b67c6f57f687a6"
I20240503 02:00:25.600440 17222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d81dcfefb4fd401b9cfd2ca36034a010" candidate_uuid: "c843193b801b4f1a88f630ed3ed0f375" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b8291b26a7f448b8636187d1caf86d0"
I20240503 02:00:25.600502 17750 raft_consensus.cc:3037] T d81dcfefb4fd401b9cfd2ca36034a010 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:25.600560 17222 raft_consensus.cc:3037] T d81dcfefb4fd401b9cfd2ca36034a010 P 5b8291b26a7f448b8636187d1caf86d0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:25.601101 16960 raft_consensus.cc:2454] T d81dcfefb4fd401b9cfd2ca36034a010 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c843193b801b4f1a88f630ed3ed0f375 in term 1.
I20240503 02:00:25.601348 17091 raft_consensus.cc:2454] T d81dcfefb4fd401b9cfd2ca36034a010 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c843193b801b4f1a88f630ed3ed0f375 in term 1.
I20240503 02:00:25.601502 17750 raft_consensus.cc:2454] T d81dcfefb4fd401b9cfd2ca36034a010 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c843193b801b4f1a88f630ed3ed0f375 in term 1.
I20240503 02:00:25.601688 17488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d81dcfefb4fd401b9cfd2ca36034a010" candidate_uuid: "c843193b801b4f1a88f630ed3ed0f375" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16d437fe629643948c95d628926d5ffa"
I20240503 02:00:25.601819 17222 raft_consensus.cc:2454] T d81dcfefb4fd401b9cfd2ca36034a010 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c843193b801b4f1a88f630ed3ed0f375 in term 1.
I20240503 02:00:25.601864 17488 raft_consensus.cc:3037] T d81dcfefb4fd401b9cfd2ca36034a010 P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:25.602155 17554 leader_election.cc:302] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 002cc3411acb4fdfa0b67c6f57f687a6, 5b8291b26a7f448b8636187d1caf86d0, 6398c8993c3c45dcaab25b42142d5671, c843193b801b4f1a88f630ed3ed0f375; no voters:
I20240503 02:00:25.602399 17354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d81dcfefb4fd401b9cfd2ca36034a010" candidate_uuid: "c843193b801b4f1a88f630ed3ed0f375" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d43ed6b8663947d99a72cd922cc323aa"
I20240503 02:00:25.602532 17354 raft_consensus.cc:3037] T d81dcfefb4fd401b9cfd2ca36034a010 P d43ed6b8663947d99a72cd922cc323aa [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:25.602797 17488 raft_consensus.cc:2454] T d81dcfefb4fd401b9cfd2ca36034a010 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c843193b801b4f1a88f630ed3ed0f375 in term 1.
I20240503 02:00:25.603029 18584 raft_consensus.cc:2781] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:25.603317 18584 raft_consensus.cc:686] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 [term 1 LEADER]: Becoming Leader. State: Replica: c843193b801b4f1a88f630ed3ed0f375, State: Running, Role: LEADER
I20240503 02:00:25.603518 17354 raft_consensus.cc:2454] T d81dcfefb4fd401b9cfd2ca36034a010 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c843193b801b4f1a88f630ed3ed0f375 in term 1.
I20240503 02:00:25.603677 18584 consensus_queue.cc:237] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:25.606595 16817 catalog_manager.cc:5484] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 reported cstate change: term changed from 0 to 1, leader changed from <none> to c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134). New cstate: current_term: 1 leader_uuid: "c843193b801b4f1a88f630ed3ed0f375" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } health_report { overall_health: UNKNOWN } } }
W20240503 02:00:25.734961 18738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:00:25.735229 18738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:00:25.738323 18738 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240503 02:00:25.834221 17222 raft_consensus.cc:1261] T ae4259dd3dc745869256269632886f8a P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:25.834771 18560 consensus_queue.cc:1035] T ae4259dd3dc745869256269632886f8a P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:25.837826 17222 raft_consensus.cc:1261] T a39f284167e942188c181147e8a9e74e P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:25.838573 18238 consensus_queue.cc:1035] T a39f284167e942188c181147e8a9e74e P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:25.839668 17091 raft_consensus.cc:1261] T a39f284167e942188c181147e8a9e74e P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:25.839769 17750 raft_consensus.cc:1261] T a39f284167e942188c181147e8a9e74e P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:25.839993 17751 raft_consensus.cc:1261] T ae4259dd3dc745869256269632886f8a P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:25.840381 16960 raft_consensus.cc:1261] T a39f284167e942188c181147e8a9e74e P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:25.841022 17618 raft_consensus.cc:1261] T a39f284167e942188c181147e8a9e74e P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:25.841028 18631 consensus_queue.cc:1035] T a39f284167e942188c181147e8a9e74e P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:25.841320 18631 consensus_queue.cc:1035] T a39f284167e942188c181147e8a9e74e P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:25.842267 18560 consensus_queue.cc:1035] T ae4259dd3dc745869256269632886f8a P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:25.842993 17751 raft_consensus.cc:1261] T d81dcfefb4fd401b9cfd2ca36034a010 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer c843193b801b4f1a88f630ed3ed0f375: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:25.843493 18584 consensus_queue.cc:1035] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:25.845650 16960 raft_consensus.cc:1261] T ae4259dd3dc745869256269632886f8a P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:25.846648 18678 consensus_queue.cc:1035] T a39f284167e942188c181147e8a9e74e P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:25.846952 18678 consensus_queue.cc:1035] T a39f284167e942188c181147e8a9e74e P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:25.847674 17488 raft_consensus.cc:1261] T a39f284167e942188c181147e8a9e74e P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:25.848430 18238 consensus_queue.cc:1035] T a39f284167e942188c181147e8a9e74e P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:25.852602 18561 consensus_queue.cc:1035] T ae4259dd3dc745869256269632886f8a P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:25.890166 17354 raft_consensus.cc:1261] T ae4259dd3dc745869256269632886f8a P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:25.890899 18560 consensus_queue.cc:1035] T ae4259dd3dc745869256269632886f8a P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:25.892391 17486 raft_consensus.cc:1261] T ae4259dd3dc745869256269632886f8a P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:25.893488 18561 consensus_queue.cc:1035] T ae4259dd3dc745869256269632886f8a P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:25.896955 17618 raft_consensus.cc:1261] T ae4259dd3dc745869256269632886f8a P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:25.897553 18561 consensus_queue.cc:1035] T ae4259dd3dc745869256269632886f8a P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:25.909260 17355 raft_consensus.cc:1261] T d81dcfefb4fd401b9cfd2ca36034a010 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Refusing update from remote peer c843193b801b4f1a88f630ed3ed0f375: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:25.909739 17487 raft_consensus.cc:1261] T d81dcfefb4fd401b9cfd2ca36034a010 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Refusing update from remote peer c843193b801b4f1a88f630ed3ed0f375: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:25.910161 18584 consensus_queue.cc:1035] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:25.910909 17223 raft_consensus.cc:1261] T d81dcfefb4fd401b9cfd2ca36034a010 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer c843193b801b4f1a88f630ed3ed0f375: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:25.911314 18584 consensus_queue.cc:1035] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:25.914786 18736 consensus_queue.cc:1035] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:25.916191 17090 raft_consensus.cc:1261] T d81dcfefb4fd401b9cfd2ca36034a010 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer c843193b801b4f1a88f630ed3ed0f375: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:25.917158 16960 raft_consensus.cc:1261] T d81dcfefb4fd401b9cfd2ca36034a010 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer c843193b801b4f1a88f630ed3ed0f375: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:25.917510 18584 consensus_queue.cc:1035] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:25.923318 18584 consensus_queue.cc:1035] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:25.924806 18675 mvcc.cc:205] Tried to move back new op lower bound from 7023417859425280000 to 7023417858176364544. Current Snapshot: MvccSnapshot[applied={T|T < 7023417859425280000}]
INSERT report
rows total: 2000
time total: 240.398 ms
time per row: 0.120199 ms
I20240503 02:00:26.027809 16813 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39916:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 7
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240503 02:00:26.028414 16813 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_5 in case of a server failure: 8 tablet servers would be needed, 7 are available. Consider bringing up more tablet servers.
I20240503 02:00:26.034751 17334 tablet_service.cc:1467] Processing CreateTablet for tablet 78d106d705df4089a4dfae4d5dc533cc (DEFAULT_TABLE table=rebalance_test_table_5 [id=90cb01160ab843bbae459ab57f0425ed]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:26.034864 17467 tablet_service.cc:1467] Processing CreateTablet for tablet 78d106d705df4089a4dfae4d5dc533cc (DEFAULT_TABLE table=rebalance_test_table_5 [id=90cb01160ab843bbae459ab57f0425ed]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:26.035149 17334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78d106d705df4089a4dfae4d5dc533cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:26.035259 17467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78d106d705df4089a4dfae4d5dc533cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:26.037016 16938 tablet_service.cc:1467] Processing CreateTablet for tablet 78d106d705df4089a4dfae4d5dc533cc (DEFAULT_TABLE table=rebalance_test_table_5 [id=90cb01160ab843bbae459ab57f0425ed]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:26.037307 16938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78d106d705df4089a4dfae4d5dc533cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:26.038004 18811 tablet_bootstrap.cc:492] T 78d106d705df4089a4dfae4d5dc533cc P 16d437fe629643948c95d628926d5ffa: Bootstrap starting.
I20240503 02:00:26.039054 18811 tablet_bootstrap.cc:654] T 78d106d705df4089a4dfae4d5dc533cc P 16d437fe629643948c95d628926d5ffa: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:26.039789 18813 tablet_bootstrap.cc:492] T 78d106d705df4089a4dfae4d5dc533cc P 6398c8993c3c45dcaab25b42142d5671: Bootstrap starting.
I20240503 02:00:26.039911 16939 tablet_service.cc:1467] Processing CreateTablet for tablet 42a900125241432a8bb31dad0b940968 (DEFAULT_TABLE table=rebalance_test_table_5 [id=90cb01160ab843bbae459ab57f0425ed]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:26.040180 16939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42a900125241432a8bb31dad0b940968. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:26.040336 18811 tablet_bootstrap.cc:492] T 78d106d705df4089a4dfae4d5dc533cc P 16d437fe629643948c95d628926d5ffa: No bootstrap required, opened a new log
I20240503 02:00:26.040486 18811 ts_tablet_manager.cc:1367] T 78d106d705df4089a4dfae4d5dc533cc P 16d437fe629643948c95d628926d5ffa: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:26.040552 18813 tablet_bootstrap.cc:654] T 78d106d705df4089a4dfae4d5dc533cc P 6398c8993c3c45dcaab25b42142d5671: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:26.041144 18811 raft_consensus.cc:348] T 78d106d705df4089a4dfae4d5dc533cc P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:26.041342 18811 raft_consensus.cc:374] T 78d106d705df4089a4dfae4d5dc533cc P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:26.041405 18811 raft_consensus.cc:729] T 78d106d705df4089a4dfae4d5dc533cc P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16d437fe629643948c95d628926d5ffa, State: Initialized, Role: FOLLOWER
I20240503 02:00:26.041465 17204 tablet_service.cc:1467] Processing CreateTablet for tablet 78d106d705df4089a4dfae4d5dc533cc (DEFAULT_TABLE table=rebalance_test_table_5 [id=90cb01160ab843bbae459ab57f0425ed]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:26.041608 18811 consensus_queue.cc:260] T 78d106d705df4089a4dfae4d5dc533cc P 16d437fe629643948c95d628926d5ffa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:26.041788 17204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78d106d705df4089a4dfae4d5dc533cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:26.041833 17334 tablet_service.cc:1467] Processing CreateTablet for tablet 42a900125241432a8bb31dad0b940968 (DEFAULT_TABLE table=rebalance_test_table_5 [id=90cb01160ab843bbae459ab57f0425ed]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:26.042023 18811 ts_tablet_manager.cc:1398] T 78d106d705df4089a4dfae4d5dc533cc P 16d437fe629643948c95d628926d5ffa: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:26.042066 17334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42a900125241432a8bb31dad0b940968. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:26.042904 17599 tablet_service.cc:1467] Processing CreateTablet for tablet 78d106d705df4089a4dfae4d5dc533cc (DEFAULT_TABLE table=rebalance_test_table_5 [id=90cb01160ab843bbae459ab57f0425ed]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:26.043223 17599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78d106d705df4089a4dfae4d5dc533cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:26.043617 17072 tablet_service.cc:1467] Processing CreateTablet for tablet 78d106d705df4089a4dfae4d5dc533cc (DEFAULT_TABLE table=rebalance_test_table_5 [id=90cb01160ab843bbae459ab57f0425ed]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:26.043895 17072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78d106d705df4089a4dfae4d5dc533cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:26.044150 17336 tablet_service.cc:1467] Processing CreateTablet for tablet ec69ad86be3d4793b10aca98dc4c1736 (DEFAULT_TABLE table=rebalance_test_table_5 [id=90cb01160ab843bbae459ab57f0425ed]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:26.044359 17336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec69ad86be3d4793b10aca98dc4c1736. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:26.044896 17071 tablet_service.cc:1467] Processing CreateTablet for tablet 42a900125241432a8bb31dad0b940968 (DEFAULT_TABLE table=rebalance_test_table_5 [id=90cb01160ab843bbae459ab57f0425ed]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:26.045115 17071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42a900125241432a8bb31dad0b940968. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:26.045653 18816 tablet_bootstrap.cc:492] T 78d106d705df4089a4dfae4d5dc533cc P c843193b801b4f1a88f630ed3ed0f375: Bootstrap starting.
I20240503 02:00:26.045709 17598 tablet_service.cc:1467] Processing CreateTablet for tablet 42a900125241432a8bb31dad0b940968 (DEFAULT_TABLE table=rebalance_test_table_5 [id=90cb01160ab843bbae459ab57f0425ed]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:26.045914 17598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42a900125241432a8bb31dad0b940968. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:26.046615 18816 tablet_bootstrap.cc:654] T 78d106d705df4089a4dfae4d5dc533cc P c843193b801b4f1a88f630ed3ed0f375: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:26.047219 17731 tablet_service.cc:1467] Processing CreateTablet for tablet 78d106d705df4089a4dfae4d5dc533cc (DEFAULT_TABLE table=rebalance_test_table_5 [id=90cb01160ab843bbae459ab57f0425ed]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:26.047521 17731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78d106d705df4089a4dfae4d5dc533cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:26.048051 17600 tablet_service.cc:1467] Processing CreateTablet for tablet ec69ad86be3d4793b10aca98dc4c1736 (DEFAULT_TABLE table=rebalance_test_table_5 [id=90cb01160ab843bbae459ab57f0425ed]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:26.048267 17600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec69ad86be3d4793b10aca98dc4c1736. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:26.048966 17730 tablet_service.cc:1467] Processing CreateTablet for tablet ec69ad86be3d4793b10aca98dc4c1736 (DEFAULT_TABLE table=rebalance_test_table_5 [id=90cb01160ab843bbae459ab57f0425ed]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:26.049289 17732 tablet_service.cc:1467] Processing CreateTablet for tablet 42a900125241432a8bb31dad0b940968 (DEFAULT_TABLE table=rebalance_test_table_5 [id=90cb01160ab843bbae459ab57f0425ed]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:26.041465 17203 tablet_service.cc:1467] Processing CreateTablet for tablet 42a900125241432a8bb31dad0b940968 (DEFAULT_TABLE table=rebalance_test_table_5 [id=90cb01160ab843bbae459ab57f0425ed]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:26.049734 17203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42a900125241432a8bb31dad0b940968. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:26.050571 17468 tablet_service.cc:1467] Processing CreateTablet for tablet ec69ad86be3d4793b10aca98dc4c1736 (DEFAULT_TABLE table=rebalance_test_table_5 [id=90cb01160ab843bbae459ab57f0425ed]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:26.050853 17468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec69ad86be3d4793b10aca98dc4c1736. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:26.051846 17732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42a900125241432a8bb31dad0b940968. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:26.051963 17071 tablet_service.cc:1467] Processing CreateTablet for tablet ec69ad86be3d4793b10aca98dc4c1736 (DEFAULT_TABLE table=rebalance_test_table_5 [id=90cb01160ab843bbae459ab57f0425ed]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:26.052203 17071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec69ad86be3d4793b10aca98dc4c1736. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:26.053097 17730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec69ad86be3d4793b10aca98dc4c1736. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:26.054003 17203 tablet_service.cc:1467] Processing CreateTablet for tablet ec69ad86be3d4793b10aca98dc4c1736 (DEFAULT_TABLE table=rebalance_test_table_5 [id=90cb01160ab843bbae459ab57f0425ed]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:26.054227 17203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec69ad86be3d4793b10aca98dc4c1736. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:26.055343 16938 tablet_service.cc:1467] Processing CreateTablet for tablet ec69ad86be3d4793b10aca98dc4c1736 (DEFAULT_TABLE table=rebalance_test_table_5 [id=90cb01160ab843bbae459ab57f0425ed]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:26.055625 16938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec69ad86be3d4793b10aca98dc4c1736. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:26.055691 17467 tablet_service.cc:1467] Processing CreateTablet for tablet 42a900125241432a8bb31dad0b940968 (DEFAULT_TABLE table=rebalance_test_table_5 [id=90cb01160ab843bbae459ab57f0425ed]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:26.056017 17467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42a900125241432a8bb31dad0b940968. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:26.056314 18811 tablet_bootstrap.cc:492] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa: Bootstrap starting.
I20240503 02:00:26.057513 18811 tablet_bootstrap.cc:654] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:26.058599 18815 tablet_bootstrap.cc:492] T 78d106d705df4089a4dfae4d5dc533cc P 5b8291b26a7f448b8636187d1caf86d0: Bootstrap starting.
I20240503 02:00:26.058681 18819 tablet_bootstrap.cc:492] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6: Bootstrap starting.
I20240503 02:00:26.058830 18811 tablet_bootstrap.cc:492] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa: No bootstrap required, opened a new log
I20240503 02:00:26.058938 18811 ts_tablet_manager.cc:1367] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240503 02:00:26.058952 18817 tablet_bootstrap.cc:492] T 42a900125241432a8bb31dad0b940968 P 89dd09f8aaee4b249ae3ce924a422a67: Bootstrap starting.
I20240503 02:00:26.059443 18811 raft_consensus.cc:348] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:26.059564 18815 tablet_bootstrap.cc:654] T 78d106d705df4089a4dfae4d5dc533cc P 5b8291b26a7f448b8636187d1caf86d0: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:26.059607 18811 raft_consensus.cc:374] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:26.059669 18811 raft_consensus.cc:729] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16d437fe629643948c95d628926d5ffa, State: Initialized, Role: FOLLOWER
I20240503 02:00:26.059875 18817 tablet_bootstrap.cc:654] T 42a900125241432a8bb31dad0b940968 P 89dd09f8aaee4b249ae3ce924a422a67: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:26.059854 18811 consensus_queue.cc:260] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:26.060174 18811 ts_tablet_manager.cc:1398] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:26.060338 18811 tablet_bootstrap.cc:492] T 42a900125241432a8bb31dad0b940968 P 16d437fe629643948c95d628926d5ffa: Bootstrap starting.
I20240503 02:00:26.060988 18819 tablet_bootstrap.cc:654] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:26.061321 18811 tablet_bootstrap.cc:654] T 42a900125241432a8bb31dad0b940968 P 16d437fe629643948c95d628926d5ffa: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:26.061352 18810 tablet_bootstrap.cc:492] T 78d106d705df4089a4dfae4d5dc533cc P d43ed6b8663947d99a72cd922cc323aa: Bootstrap starting.
I20240503 02:00:26.062721 18819 tablet_bootstrap.cc:492] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6: No bootstrap required, opened a new log
I20240503 02:00:26.062844 18819 ts_tablet_manager.cc:1367] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 02:00:26.063392 18819 raft_consensus.cc:348] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:26.063534 18819 raft_consensus.cc:374] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:26.063539 18810 tablet_bootstrap.cc:654] T 78d106d705df4089a4dfae4d5dc533cc P d43ed6b8663947d99a72cd922cc323aa: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:26.063570 18819 raft_consensus.cc:729] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 002cc3411acb4fdfa0b67c6f57f687a6, State: Initialized, Role: FOLLOWER
I20240503 02:00:26.063717 18819 consensus_queue.cc:260] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:26.064102 18819 ts_tablet_manager.cc:1398] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:26.064271 18819 tablet_bootstrap.cc:492] T 42a900125241432a8bb31dad0b940968 P 002cc3411acb4fdfa0b67c6f57f687a6: Bootstrap starting.
I20240503 02:00:26.064342 18817 tablet_bootstrap.cc:492] T 42a900125241432a8bb31dad0b940968 P 89dd09f8aaee4b249ae3ce924a422a67: No bootstrap required, opened a new log
I20240503 02:00:26.064428 18811 tablet_bootstrap.cc:492] T 42a900125241432a8bb31dad0b940968 P 16d437fe629643948c95d628926d5ffa: No bootstrap required, opened a new log
I20240503 02:00:26.064455 18817 ts_tablet_manager.cc:1367] T 42a900125241432a8bb31dad0b940968 P 89dd09f8aaee4b249ae3ce924a422a67: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 02:00:26.064500 18811 ts_tablet_manager.cc:1367] T 42a900125241432a8bb31dad0b940968 P 16d437fe629643948c95d628926d5ffa: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240503 02:00:26.064966 18811 raft_consensus.cc:348] T 42a900125241432a8bb31dad0b940968 P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:26.065075 18816 tablet_bootstrap.cc:492] T 78d106d705df4089a4dfae4d5dc533cc P c843193b801b4f1a88f630ed3ed0f375: No bootstrap required, opened a new log
I20240503 02:00:26.065081 18811 raft_consensus.cc:374] T 42a900125241432a8bb31dad0b940968 P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:26.064997 18817 raft_consensus.cc:348] T 42a900125241432a8bb31dad0b940968 P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:26.065132 18811 raft_consensus.cc:729] T 42a900125241432a8bb31dad0b940968 P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16d437fe629643948c95d628926d5ffa, State: Initialized, Role: FOLLOWER
I20240503 02:00:26.065181 18816 ts_tablet_manager.cc:1367] T 78d106d705df4089a4dfae4d5dc533cc P c843193b801b4f1a88f630ed3ed0f375: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.000s
I20240503 02:00:26.065246 18811 consensus_queue.cc:260] T 42a900125241432a8bb31dad0b940968 P 16d437fe629643948c95d628926d5ffa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:26.065526 18811 ts_tablet_manager.cc:1398] T 42a900125241432a8bb31dad0b940968 P 16d437fe629643948c95d628926d5ffa: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:26.065737 18816 raft_consensus.cc:348] T 78d106d705df4089a4dfae4d5dc533cc P c843193b801b4f1a88f630ed3ed0f375 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:26.065901 18816 raft_consensus.cc:374] T 78d106d705df4089a4dfae4d5dc533cc P c843193b801b4f1a88f630ed3ed0f375 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:26.065939 18816 raft_consensus.cc:729] T 78d106d705df4089a4dfae4d5dc533cc P c843193b801b4f1a88f630ed3ed0f375 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c843193b801b4f1a88f630ed3ed0f375, State: Initialized, Role: FOLLOWER
I20240503 02:00:26.066109 18816 consensus_queue.cc:260] T 78d106d705df4089a4dfae4d5dc533cc P c843193b801b4f1a88f630ed3ed0f375 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:26.066246 18819 tablet_bootstrap.cc:654] T 42a900125241432a8bb31dad0b940968 P 002cc3411acb4fdfa0b67c6f57f687a6: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:26.067627 18816 ts_tablet_manager.cc:1398] T 78d106d705df4089a4dfae4d5dc533cc P c843193b801b4f1a88f630ed3ed0f375: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:26.065148 18817 raft_consensus.cc:374] T 42a900125241432a8bb31dad0b940968 P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:26.067826 18816 tablet_bootstrap.cc:492] T 42a900125241432a8bb31dad0b940968 P c843193b801b4f1a88f630ed3ed0f375: Bootstrap starting.
I20240503 02:00:26.067864 18817 raft_consensus.cc:729] T 42a900125241432a8bb31dad0b940968 P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89dd09f8aaee4b249ae3ce924a422a67, State: Initialized, Role: FOLLOWER
I20240503 02:00:26.068002 18817 consensus_queue.cc:260] T 42a900125241432a8bb31dad0b940968 P 89dd09f8aaee4b249ae3ce924a422a67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:26.069087 18816 tablet_bootstrap.cc:654] T 42a900125241432a8bb31dad0b940968 P c843193b801b4f1a88f630ed3ed0f375: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:26.070163 18813 tablet_bootstrap.cc:492] T 78d106d705df4089a4dfae4d5dc533cc P 6398c8993c3c45dcaab25b42142d5671: No bootstrap required, opened a new log
I20240503 02:00:26.070231 18817 ts_tablet_manager.cc:1398] T 42a900125241432a8bb31dad0b940968 P 89dd09f8aaee4b249ae3ce924a422a67: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240503 02:00:26.070279 18813 ts_tablet_manager.cc:1367] T 78d106d705df4089a4dfae4d5dc533cc P 6398c8993c3c45dcaab25b42142d5671: Time spent bootstrapping tablet: real 0.031s user 0.000s sys 0.001s
I20240503 02:00:26.070827 18813 raft_consensus.cc:348] T 78d106d705df4089a4dfae4d5dc533cc P 6398c8993c3c45dcaab25b42142d5671 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:26.070926 18819 tablet_bootstrap.cc:492] T 42a900125241432a8bb31dad0b940968 P 002cc3411acb4fdfa0b67c6f57f687a6: No bootstrap required, opened a new log
I20240503 02:00:26.070981 18813 raft_consensus.cc:374] T 78d106d705df4089a4dfae4d5dc533cc P 6398c8993c3c45dcaab25b42142d5671 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:26.071028 18813 raft_consensus.cc:729] T 78d106d705df4089a4dfae4d5dc533cc P 6398c8993c3c45dcaab25b42142d5671 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6398c8993c3c45dcaab25b42142d5671, State: Initialized, Role: FOLLOWER
I20240503 02:00:26.071038 18817 tablet_bootstrap.cc:492] T 78d106d705df4089a4dfae4d5dc533cc P 89dd09f8aaee4b249ae3ce924a422a67: Bootstrap starting.
I20240503 02:00:26.071167 18819 ts_tablet_manager.cc:1367] T 42a900125241432a8bb31dad0b940968 P 002cc3411acb4fdfa0b67c6f57f687a6: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 02:00:26.071172 18813 consensus_queue.cc:260] T 78d106d705df4089a4dfae4d5dc533cc P 6398c8993c3c45dcaab25b42142d5671 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:26.071511 18813 ts_tablet_manager.cc:1398] T 78d106d705df4089a4dfae4d5dc533cc P 6398c8993c3c45dcaab25b42142d5671: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:26.071682 18813 tablet_bootstrap.cc:492] T 42a900125241432a8bb31dad0b940968 P 6398c8993c3c45dcaab25b42142d5671: Bootstrap starting.
I20240503 02:00:26.072293 18817 tablet_bootstrap.cc:654] T 78d106d705df4089a4dfae4d5dc533cc P 89dd09f8aaee4b249ae3ce924a422a67: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:26.072754 18813 tablet_bootstrap.cc:654] T 42a900125241432a8bb31dad0b940968 P 6398c8993c3c45dcaab25b42142d5671: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:26.073814 18817 tablet_bootstrap.cc:492] T 78d106d705df4089a4dfae4d5dc533cc P 89dd09f8aaee4b249ae3ce924a422a67: No bootstrap required, opened a new log
I20240503 02:00:26.073834 18810 tablet_bootstrap.cc:492] T 78d106d705df4089a4dfae4d5dc533cc P d43ed6b8663947d99a72cd922cc323aa: No bootstrap required, opened a new log
I20240503 02:00:26.073905 18817 ts_tablet_manager.cc:1367] T 78d106d705df4089a4dfae4d5dc533cc P 89dd09f8aaee4b249ae3ce924a422a67: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240503 02:00:26.073932 18810 ts_tablet_manager.cc:1367] T 78d106d705df4089a4dfae4d5dc533cc P d43ed6b8663947d99a72cd922cc323aa: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240503 02:00:26.074246 18819 raft_consensus.cc:348] T 42a900125241432a8bb31dad0b940968 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:26.074357 18817 raft_consensus.cc:348] T 78d106d705df4089a4dfae4d5dc533cc P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:26.074512 18817 raft_consensus.cc:374] T 78d106d705df4089a4dfae4d5dc533cc P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:26.074409 18810 raft_consensus.cc:348] T 78d106d705df4089a4dfae4d5dc533cc P d43ed6b8663947d99a72cd922cc323aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:26.074553 18817 raft_consensus.cc:729] T 78d106d705df4089a4dfae4d5dc533cc P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89dd09f8aaee4b249ae3ce924a422a67, State: Initialized, Role: FOLLOWER
I20240503 02:00:26.074558 18810 raft_consensus.cc:374] T 78d106d705df4089a4dfae4d5dc533cc P d43ed6b8663947d99a72cd922cc323aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:26.074595 18810 raft_consensus.cc:729] T 78d106d705df4089a4dfae4d5dc533cc P d43ed6b8663947d99a72cd922cc323aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d43ed6b8663947d99a72cd922cc323aa, State: Initialized, Role: FOLLOWER
I20240503 02:00:26.074695 18817 consensus_queue.cc:260] T 78d106d705df4089a4dfae4d5dc533cc P 89dd09f8aaee4b249ae3ce924a422a67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:26.074466 18819 raft_consensus.cc:374] T 42a900125241432a8bb31dad0b940968 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:26.074939 18819 raft_consensus.cc:729] T 42a900125241432a8bb31dad0b940968 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 002cc3411acb4fdfa0b67c6f57f687a6, State: Initialized, Role: FOLLOWER
I20240503 02:00:26.075093 18813 tablet_bootstrap.cc:492] T 42a900125241432a8bb31dad0b940968 P 6398c8993c3c45dcaab25b42142d5671: No bootstrap required, opened a new log
I20240503 02:00:26.075186 18813 ts_tablet_manager.cc:1367] T 42a900125241432a8bb31dad0b940968 P 6398c8993c3c45dcaab25b42142d5671: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240503 02:00:26.075127 18819 consensus_queue.cc:260] T 42a900125241432a8bb31dad0b940968 P 002cc3411acb4fdfa0b67c6f57f687a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:26.075480 18819 ts_tablet_manager.cc:1398] T 42a900125241432a8bb31dad0b940968 P 002cc3411acb4fdfa0b67c6f57f687a6: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 02:00:26.075660 18819 tablet_bootstrap.cc:492] T ec69ad86be3d4793b10aca98dc4c1736 P 002cc3411acb4fdfa0b67c6f57f687a6: Bootstrap starting.
I20240503 02:00:26.075625 18813 raft_consensus.cc:348] T 42a900125241432a8bb31dad0b940968 P 6398c8993c3c45dcaab25b42142d5671 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:26.075764 18813 raft_consensus.cc:374] T 42a900125241432a8bb31dad0b940968 P 6398c8993c3c45dcaab25b42142d5671 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:26.075917 18813 raft_consensus.cc:729] T 42a900125241432a8bb31dad0b940968 P 6398c8993c3c45dcaab25b42142d5671 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6398c8993c3c45dcaab25b42142d5671, State: Initialized, Role: FOLLOWER
I20240503 02:00:26.076059 18813 consensus_queue.cc:260] T 42a900125241432a8bb31dad0b940968 P 6398c8993c3c45dcaab25b42142d5671 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:26.076460 18813 ts_tablet_manager.cc:1398] T 42a900125241432a8bb31dad0b940968 P 6398c8993c3c45dcaab25b42142d5671: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:26.076597 18819 tablet_bootstrap.cc:654] T ec69ad86be3d4793b10aca98dc4c1736 P 002cc3411acb4fdfa0b67c6f57f687a6: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:26.076658 18813 tablet_bootstrap.cc:492] T ec69ad86be3d4793b10aca98dc4c1736 P 6398c8993c3c45dcaab25b42142d5671: Bootstrap starting.
I20240503 02:00:26.077746 18813 tablet_bootstrap.cc:654] T ec69ad86be3d4793b10aca98dc4c1736 P 6398c8993c3c45dcaab25b42142d5671: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:26.077853 18817 ts_tablet_manager.cc:1398] T 78d106d705df4089a4dfae4d5dc533cc P 89dd09f8aaee4b249ae3ce924a422a67: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240503 02:00:26.078133 18819 tablet_bootstrap.cc:492] T ec69ad86be3d4793b10aca98dc4c1736 P 002cc3411acb4fdfa0b67c6f57f687a6: No bootstrap required, opened a new log
I20240503 02:00:26.078214 18819 ts_tablet_manager.cc:1367] T ec69ad86be3d4793b10aca98dc4c1736 P 002cc3411acb4fdfa0b67c6f57f687a6: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:26.078465 18810 consensus_queue.cc:260] T 78d106d705df4089a4dfae4d5dc533cc P d43ed6b8663947d99a72cd922cc323aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:26.078627 18819 raft_consensus.cc:348] T ec69ad86be3d4793b10aca98dc4c1736 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:26.078778 18819 raft_consensus.cc:374] T ec69ad86be3d4793b10aca98dc4c1736 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:26.078824 18819 raft_consensus.cc:729] T ec69ad86be3d4793b10aca98dc4c1736 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 002cc3411acb4fdfa0b67c6f57f687a6, State: Initialized, Role: FOLLOWER
I20240503 02:00:26.078965 18819 consensus_queue.cc:260] T ec69ad86be3d4793b10aca98dc4c1736 P 002cc3411acb4fdfa0b67c6f57f687a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:26.079285 18819 ts_tablet_manager.cc:1398] T ec69ad86be3d4793b10aca98dc4c1736 P 002cc3411acb4fdfa0b67c6f57f687a6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:26.079774 18810 ts_tablet_manager.cc:1398] T 78d106d705df4089a4dfae4d5dc533cc P d43ed6b8663947d99a72cd922cc323aa: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240503 02:00:26.080027 18810 tablet_bootstrap.cc:492] T 42a900125241432a8bb31dad0b940968 P d43ed6b8663947d99a72cd922cc323aa: Bootstrap starting.
I20240503 02:00:26.081364 18817 tablet_bootstrap.cc:492] T ec69ad86be3d4793b10aca98dc4c1736 P 89dd09f8aaee4b249ae3ce924a422a67: Bootstrap starting.
I20240503 02:00:26.081881 18810 tablet_bootstrap.cc:654] T 42a900125241432a8bb31dad0b940968 P d43ed6b8663947d99a72cd922cc323aa: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:26.082633 18817 tablet_bootstrap.cc:654] T ec69ad86be3d4793b10aca98dc4c1736 P 89dd09f8aaee4b249ae3ce924a422a67: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:26.083534 18810 tablet_bootstrap.cc:492] T 42a900125241432a8bb31dad0b940968 P d43ed6b8663947d99a72cd922cc323aa: No bootstrap required, opened a new log
I20240503 02:00:26.083631 18810 ts_tablet_manager.cc:1367] T 42a900125241432a8bb31dad0b940968 P d43ed6b8663947d99a72cd922cc323aa: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:00:26.083740 18815 tablet_bootstrap.cc:492] T 78d106d705df4089a4dfae4d5dc533cc P 5b8291b26a7f448b8636187d1caf86d0: No bootstrap required, opened a new log
I20240503 02:00:26.083886 18815 ts_tablet_manager.cc:1367] T 78d106d705df4089a4dfae4d5dc533cc P 5b8291b26a7f448b8636187d1caf86d0: Time spent bootstrapping tablet: real 0.025s user 0.002s sys 0.000s
I20240503 02:00:26.084111 18810 raft_consensus.cc:348] T 42a900125241432a8bb31dad0b940968 P d43ed6b8663947d99a72cd922cc323aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:26.084239 18810 raft_consensus.cc:374] T 42a900125241432a8bb31dad0b940968 P d43ed6b8663947d99a72cd922cc323aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:26.084282 18810 raft_consensus.cc:729] T 42a900125241432a8bb31dad0b940968 P d43ed6b8663947d99a72cd922cc323aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d43ed6b8663947d99a72cd922cc323aa, State: Initialized, Role: FOLLOWER
I20240503 02:00:26.084429 18810 consensus_queue.cc:260] T 42a900125241432a8bb31dad0b940968 P d43ed6b8663947d99a72cd922cc323aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:26.084442 18815 raft_consensus.cc:348] T 78d106d705df4089a4dfae4d5dc533cc P 5b8291b26a7f448b8636187d1caf86d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:26.084570 18815 raft_consensus.cc:374] T 78d106d705df4089a4dfae4d5dc533cc P 5b8291b26a7f448b8636187d1caf86d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:26.084620 18815 raft_consensus.cc:729] T 78d106d705df4089a4dfae4d5dc533cc P 5b8291b26a7f448b8636187d1caf86d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b8291b26a7f448b8636187d1caf86d0, State: Initialized, Role: FOLLOWER
I20240503 02:00:26.084795 18815 consensus_queue.cc:260] T 78d106d705df4089a4dfae4d5dc533cc P 5b8291b26a7f448b8636187d1caf86d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:26.085148 18815 ts_tablet_manager.cc:1398] T 78d106d705df4089a4dfae4d5dc533cc P 5b8291b26a7f448b8636187d1caf86d0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:26.085305 18815 tablet_bootstrap.cc:492] T 42a900125241432a8bb31dad0b940968 P 5b8291b26a7f448b8636187d1caf86d0: Bootstrap starting.
I20240503 02:00:26.085928 18810 ts_tablet_manager.cc:1398] T 42a900125241432a8bb31dad0b940968 P d43ed6b8663947d99a72cd922cc323aa: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:26.086257 18815 tablet_bootstrap.cc:654] T 42a900125241432a8bb31dad0b940968 P 5b8291b26a7f448b8636187d1caf86d0: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:26.086614 18810 tablet_bootstrap.cc:492] T ec69ad86be3d4793b10aca98dc4c1736 P d43ed6b8663947d99a72cd922cc323aa: Bootstrap starting.
I20240503 02:00:26.087478 18815 tablet_bootstrap.cc:492] T 42a900125241432a8bb31dad0b940968 P 5b8291b26a7f448b8636187d1caf86d0: No bootstrap required, opened a new log
I20240503 02:00:26.087569 18815 ts_tablet_manager.cc:1367] T 42a900125241432a8bb31dad0b940968 P 5b8291b26a7f448b8636187d1caf86d0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:26.088083 18810 tablet_bootstrap.cc:654] T ec69ad86be3d4793b10aca98dc4c1736 P d43ed6b8663947d99a72cd922cc323aa: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:26.088023 18815 raft_consensus.cc:348] T 42a900125241432a8bb31dad0b940968 P 5b8291b26a7f448b8636187d1caf86d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:26.088153 18815 raft_consensus.cc:374] T 42a900125241432a8bb31dad0b940968 P 5b8291b26a7f448b8636187d1caf86d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:26.088186 18815 raft_consensus.cc:729] T 42a900125241432a8bb31dad0b940968 P 5b8291b26a7f448b8636187d1caf86d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b8291b26a7f448b8636187d1caf86d0, State: Initialized, Role: FOLLOWER
I20240503 02:00:26.088336 18815 consensus_queue.cc:260] T 42a900125241432a8bb31dad0b940968 P 5b8291b26a7f448b8636187d1caf86d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:26.088709 18815 ts_tablet_manager.cc:1398] T 42a900125241432a8bb31dad0b940968 P 5b8291b26a7f448b8636187d1caf86d0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:26.089135 18815 tablet_bootstrap.cc:492] T ec69ad86be3d4793b10aca98dc4c1736 P 5b8291b26a7f448b8636187d1caf86d0: Bootstrap starting.
I20240503 02:00:26.089205 18813 tablet_bootstrap.cc:492] T ec69ad86be3d4793b10aca98dc4c1736 P 6398c8993c3c45dcaab25b42142d5671: No bootstrap required, opened a new log
I20240503 02:00:26.089316 18813 ts_tablet_manager.cc:1367] T ec69ad86be3d4793b10aca98dc4c1736 P 6398c8993c3c45dcaab25b42142d5671: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240503 02:00:26.089648 18816 tablet_bootstrap.cc:492] T 42a900125241432a8bb31dad0b940968 P c843193b801b4f1a88f630ed3ed0f375: No bootstrap required, opened a new log
I20240503 02:00:26.089753 18816 ts_tablet_manager.cc:1367] T 42a900125241432a8bb31dad0b940968 P c843193b801b4f1a88f630ed3ed0f375: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.000s
I20240503 02:00:26.089918 18813 raft_consensus.cc:348] T ec69ad86be3d4793b10aca98dc4c1736 P 6398c8993c3c45dcaab25b42142d5671 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:26.090032 18813 raft_consensus.cc:374] T ec69ad86be3d4793b10aca98dc4c1736 P 6398c8993c3c45dcaab25b42142d5671 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:26.090055 18813 raft_consensus.cc:729] T ec69ad86be3d4793b10aca98dc4c1736 P 6398c8993c3c45dcaab25b42142d5671 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6398c8993c3c45dcaab25b42142d5671, State: Initialized, Role: FOLLOWER
I20240503 02:00:26.090042 18815 tablet_bootstrap.cc:654] T ec69ad86be3d4793b10aca98dc4c1736 P 5b8291b26a7f448b8636187d1caf86d0: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:26.090185 18813 consensus_queue.cc:260] T ec69ad86be3d4793b10aca98dc4c1736 P 6398c8993c3c45dcaab25b42142d5671 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:26.090422 18813 ts_tablet_manager.cc:1398] T ec69ad86be3d4793b10aca98dc4c1736 P 6398c8993c3c45dcaab25b42142d5671: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:26.090353 18816 raft_consensus.cc:348] T 42a900125241432a8bb31dad0b940968 P c843193b801b4f1a88f630ed3ed0f375 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:26.090487 18816 raft_consensus.cc:374] T 42a900125241432a8bb31dad0b940968 P c843193b801b4f1a88f630ed3ed0f375 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:26.090534 18816 raft_consensus.cc:729] T 42a900125241432a8bb31dad0b940968 P c843193b801b4f1a88f630ed3ed0f375 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c843193b801b4f1a88f630ed3ed0f375, State: Initialized, Role: FOLLOWER
I20240503 02:00:26.090688 18816 consensus_queue.cc:260] T 42a900125241432a8bb31dad0b940968 P c843193b801b4f1a88f630ed3ed0f375 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:26.091181 18816 ts_tablet_manager.cc:1398] T 42a900125241432a8bb31dad0b940968 P c843193b801b4f1a88f630ed3ed0f375: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 02:00:26.091532 18816 tablet_bootstrap.cc:492] T ec69ad86be3d4793b10aca98dc4c1736 P c843193b801b4f1a88f630ed3ed0f375: Bootstrap starting.
I20240503 02:00:26.092556 18816 tablet_bootstrap.cc:654] T ec69ad86be3d4793b10aca98dc4c1736 P c843193b801b4f1a88f630ed3ed0f375: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:26.093453 18589 raft_consensus.cc:482] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:26.093570 18589 raft_consensus.cc:504] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:26.093766 18810 tablet_bootstrap.cc:492] T ec69ad86be3d4793b10aca98dc4c1736 P d43ed6b8663947d99a72cd922cc323aa: No bootstrap required, opened a new log
I20240503 02:00:26.093927 18810 ts_tablet_manager.cc:1367] T ec69ad86be3d4793b10aca98dc4c1736 P d43ed6b8663947d99a72cd922cc323aa: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.001s
I20240503 02:00:26.094308 18589 leader_election.cc:288] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d43ed6b8663947d99a72cd922cc323aa (127.0.99.132:40023), 16d437fe629643948c95d628926d5ffa (127.0.99.133:46581), c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134:44343), 89dd09f8aaee4b249ae3ce924a422a67 (127.0.99.130:38889), 6398c8993c3c45dcaab25b42142d5671 (127.0.99.129:36983), 5b8291b26a7f448b8636187d1caf86d0 (127.0.99.131:44479)
I20240503 02:00:26.094483 18810 raft_consensus.cc:348] T ec69ad86be3d4793b10aca98dc4c1736 P d43ed6b8663947d99a72cd922cc323aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:26.094749 18810 raft_consensus.cc:374] T ec69ad86be3d4793b10aca98dc4c1736 P d43ed6b8663947d99a72cd922cc323aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:26.094781 18815 tablet_bootstrap.cc:492] T ec69ad86be3d4793b10aca98dc4c1736 P 5b8291b26a7f448b8636187d1caf86d0: No bootstrap required, opened a new log
I20240503 02:00:26.094820 18810 raft_consensus.cc:729] T ec69ad86be3d4793b10aca98dc4c1736 P d43ed6b8663947d99a72cd922cc323aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d43ed6b8663947d99a72cd922cc323aa, State: Initialized, Role: FOLLOWER
I20240503 02:00:26.094858 18815 ts_tablet_manager.cc:1367] T ec69ad86be3d4793b10aca98dc4c1736 P 5b8291b26a7f448b8636187d1caf86d0: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 02:00:26.094993 18810 consensus_queue.cc:260] T ec69ad86be3d4793b10aca98dc4c1736 P d43ed6b8663947d99a72cd922cc323aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:26.095299 16960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78d106d705df4089a4dfae4d5dc533cc" candidate_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6398c8993c3c45dcaab25b42142d5671" is_pre_election: true
I20240503 02:00:26.095345 17355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78d106d705df4089a4dfae4d5dc533cc" candidate_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d43ed6b8663947d99a72cd922cc323aa" is_pre_election: true
I20240503 02:00:26.095346 18815 raft_consensus.cc:348] T ec69ad86be3d4793b10aca98dc4c1736 P 5b8291b26a7f448b8636187d1caf86d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:26.095427 16960 raft_consensus.cc:2454] T 78d106d705df4089a4dfae4d5dc533cc P 6398c8993c3c45dcaab25b42142d5671 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 002cc3411acb4fdfa0b67c6f57f687a6 in term 0.
I20240503 02:00:26.095474 18815 raft_consensus.cc:374] T ec69ad86be3d4793b10aca98dc4c1736 P 5b8291b26a7f448b8636187d1caf86d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:26.095479 17355 raft_consensus.cc:2454] T 78d106d705df4089a4dfae4d5dc533cc P d43ed6b8663947d99a72cd922cc323aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 002cc3411acb4fdfa0b67c6f57f687a6 in term 0.
I20240503 02:00:26.095510 18815 raft_consensus.cc:729] T ec69ad86be3d4793b10aca98dc4c1736 P 5b8291b26a7f448b8636187d1caf86d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b8291b26a7f448b8636187d1caf86d0, State: Initialized, Role: FOLLOWER
I20240503 02:00:26.095647 18815 consensus_queue.cc:260] T ec69ad86be3d4793b10aca98dc4c1736 P 5b8291b26a7f448b8636187d1caf86d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:26.095925 18815 ts_tablet_manager.cc:1398] T ec69ad86be3d4793b10aca98dc4c1736 P 5b8291b26a7f448b8636187d1caf86d0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:26.095942 18810 ts_tablet_manager.cc:1398] T ec69ad86be3d4793b10aca98dc4c1736 P d43ed6b8663947d99a72cd922cc323aa: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 02:00:26.096040 17223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78d106d705df4089a4dfae4d5dc533cc" candidate_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b8291b26a7f448b8636187d1caf86d0" is_pre_election: true
I20240503 02:00:26.096181 17223 raft_consensus.cc:2454] T 78d106d705df4089a4dfae4d5dc533cc P 5b8291b26a7f448b8636187d1caf86d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 002cc3411acb4fdfa0b67c6f57f687a6 in term 0.
I20240503 02:00:26.096331 17090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78d106d705df4089a4dfae4d5dc533cc" candidate_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89dd09f8aaee4b249ae3ce924a422a67" is_pre_election: true
I20240503 02:00:26.096454 17090 raft_consensus.cc:2454] T 78d106d705df4089a4dfae4d5dc533cc P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 002cc3411acb4fdfa0b67c6f57f687a6 in term 0.
I20240503 02:00:26.096859 17686 leader_election.cc:302] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 002cc3411acb4fdfa0b67c6f57f687a6, 5b8291b26a7f448b8636187d1caf86d0, 6398c8993c3c45dcaab25b42142d5671, d43ed6b8663947d99a72cd922cc323aa; no voters:
I20240503 02:00:26.096987 18816 tablet_bootstrap.cc:492] T ec69ad86be3d4793b10aca98dc4c1736 P c843193b801b4f1a88f630ed3ed0f375: No bootstrap required, opened a new log
I20240503 02:00:26.097025 18589 raft_consensus.cc:2781] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:26.097074 18816 ts_tablet_manager.cc:1367] T ec69ad86be3d4793b10aca98dc4c1736 P c843193b801b4f1a88f630ed3ed0f375: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20240503 02:00:26.097085 18589 raft_consensus.cc:482] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:26.097134 18589 raft_consensus.cc:3037] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:26.097561 18816 raft_consensus.cc:348] T ec69ad86be3d4793b10aca98dc4c1736 P c843193b801b4f1a88f630ed3ed0f375 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:26.097688 18816 raft_consensus.cc:374] T ec69ad86be3d4793b10aca98dc4c1736 P c843193b801b4f1a88f630ed3ed0f375 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:26.097728 18816 raft_consensus.cc:729] T ec69ad86be3d4793b10aca98dc4c1736 P c843193b801b4f1a88f630ed3ed0f375 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c843193b801b4f1a88f630ed3ed0f375, State: Initialized, Role: FOLLOWER
I20240503 02:00:26.097898 18816 consensus_queue.cc:260] T ec69ad86be3d4793b10aca98dc4c1736 P c843193b801b4f1a88f630ed3ed0f375 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:26.098009 17486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78d106d705df4089a4dfae4d5dc533cc" candidate_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16d437fe629643948c95d628926d5ffa" is_pre_election: true
I20240503 02:00:26.098158 17486 raft_consensus.cc:2454] T 78d106d705df4089a4dfae4d5dc533cc P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 002cc3411acb4fdfa0b67c6f57f687a6 in term 0.
I20240503 02:00:26.098241 18816 ts_tablet_manager.cc:1398] T ec69ad86be3d4793b10aca98dc4c1736 P c843193b801b4f1a88f630ed3ed0f375: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20240503 02:00:26.098206 18589 raft_consensus.cc:504] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:26.098348 18817 tablet_bootstrap.cc:492] T ec69ad86be3d4793b10aca98dc4c1736 P 89dd09f8aaee4b249ae3ce924a422a67: No bootstrap required, opened a new log
I20240503 02:00:26.098446 18817 ts_tablet_manager.cc:1367] T ec69ad86be3d4793b10aca98dc4c1736 P 89dd09f8aaee4b249ae3ce924a422a67: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.002s
I20240503 02:00:26.098402 17618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78d106d705df4089a4dfae4d5dc533cc" candidate_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c843193b801b4f1a88f630ed3ed0f375" is_pre_election: true
I20240503 02:00:26.098552 17618 raft_consensus.cc:2454] T 78d106d705df4089a4dfae4d5dc533cc P c843193b801b4f1a88f630ed3ed0f375 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 002cc3411acb4fdfa0b67c6f57f687a6 in term 0.
I20240503 02:00:26.098806 18589 leader_election.cc:288] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 [CANDIDATE]: Term 1 election: Requested vote from peers d43ed6b8663947d99a72cd922cc323aa (127.0.99.132:40023), 16d437fe629643948c95d628926d5ffa (127.0.99.133:46581), c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134:44343), 89dd09f8aaee4b249ae3ce924a422a67 (127.0.99.130:38889), 6398c8993c3c45dcaab25b42142d5671 (127.0.99.129:36983), 5b8291b26a7f448b8636187d1caf86d0 (127.0.99.131:44479)
I20240503 02:00:26.098989 18817 raft_consensus.cc:348] T ec69ad86be3d4793b10aca98dc4c1736 P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:26.099129 18817 raft_consensus.cc:374] T ec69ad86be3d4793b10aca98dc4c1736 P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:26.099151 18817 raft_consensus.cc:729] T ec69ad86be3d4793b10aca98dc4c1736 P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89dd09f8aaee4b249ae3ce924a422a67, State: Initialized, Role: FOLLOWER
I20240503 02:00:26.099262 18817 consensus_queue.cc:260] T ec69ad86be3d4793b10aca98dc4c1736 P 89dd09f8aaee4b249ae3ce924a422a67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:26.099447 16960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78d106d705df4089a4dfae4d5dc533cc" candidate_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6398c8993c3c45dcaab25b42142d5671"
I20240503 02:00:26.099556 16960 raft_consensus.cc:3037] T 78d106d705df4089a4dfae4d5dc533cc P 6398c8993c3c45dcaab25b42142d5671 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:26.099571 18817 ts_tablet_manager.cc:1398] T ec69ad86be3d4793b10aca98dc4c1736 P 89dd09f8aaee4b249ae3ce924a422a67: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:26.100082 17355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78d106d705df4089a4dfae4d5dc533cc" candidate_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d43ed6b8663947d99a72cd922cc323aa"
I20240503 02:00:26.100206 17355 raft_consensus.cc:3037] T 78d106d705df4089a4dfae4d5dc533cc P d43ed6b8663947d99a72cd922cc323aa [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:26.100636 16960 raft_consensus.cc:2454] T 78d106d705df4089a4dfae4d5dc533cc P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 002cc3411acb4fdfa0b67c6f57f687a6 in term 1.
I20240503 02:00:26.100920 17223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78d106d705df4089a4dfae4d5dc533cc" candidate_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b8291b26a7f448b8636187d1caf86d0"
I20240503 02:00:26.101033 17223 raft_consensus.cc:3037] T 78d106d705df4089a4dfae4d5dc533cc P 5b8291b26a7f448b8636187d1caf86d0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:26.101351 17355 raft_consensus.cc:2454] T 78d106d705df4089a4dfae4d5dc533cc P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 002cc3411acb4fdfa0b67c6f57f687a6 in term 1.
I20240503 02:00:26.102016 17223 raft_consensus.cc:2454] T 78d106d705df4089a4dfae4d5dc533cc P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 002cc3411acb4fdfa0b67c6f57f687a6 in term 1.
I20240503 02:00:26.102285 17686 leader_election.cc:302] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 002cc3411acb4fdfa0b67c6f57f687a6, 5b8291b26a7f448b8636187d1caf86d0, 6398c8993c3c45dcaab25b42142d5671, d43ed6b8663947d99a72cd922cc323aa; no voters:
I20240503 02:00:26.102505 18589 raft_consensus.cc:2781] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:26.102701 18589 raft_consensus.cc:686] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 LEADER]: Becoming Leader. State: Replica: 002cc3411acb4fdfa0b67c6f57f687a6, State: Running, Role: LEADER
I20240503 02:00:26.102877 18589 consensus_queue.cc:237] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:26.103551 17090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78d106d705df4089a4dfae4d5dc533cc" candidate_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89dd09f8aaee4b249ae3ce924a422a67"
I20240503 02:00:26.103688 17090 raft_consensus.cc:3037] T 78d106d705df4089a4dfae4d5dc533cc P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:26.104907 17090 raft_consensus.cc:2454] T 78d106d705df4089a4dfae4d5dc533cc P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 002cc3411acb4fdfa0b67c6f57f687a6 in term 1.
I20240503 02:00:26.105422 17486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78d106d705df4089a4dfae4d5dc533cc" candidate_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16d437fe629643948c95d628926d5ffa"
I20240503 02:00:26.105546 17486 raft_consensus.cc:3037] T 78d106d705df4089a4dfae4d5dc533cc P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:26.105707 16813 catalog_manager.cc:5484] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 002cc3411acb4fdfa0b67c6f57f687a6 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:26.106519 17486 raft_consensus.cc:2454] T 78d106d705df4089a4dfae4d5dc533cc P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 002cc3411acb4fdfa0b67c6f57f687a6 in term 1.
I20240503 02:00:26.106758 17618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78d106d705df4089a4dfae4d5dc533cc" candidate_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c843193b801b4f1a88f630ed3ed0f375"
I20240503 02:00:26.106899 17618 raft_consensus.cc:3037] T 78d106d705df4089a4dfae4d5dc533cc P c843193b801b4f1a88f630ed3ed0f375 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:26.107627 17618 raft_consensus.cc:2454] T 78d106d705df4089a4dfae4d5dc533cc P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 002cc3411acb4fdfa0b67c6f57f687a6 in term 1.
I20240503 02:00:26.115321 18809 raft_consensus.cc:482] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:26.115439 18809 raft_consensus.cc:504] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:26.116139 18809 leader_election.cc:288] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d43ed6b8663947d99a72cd922cc323aa (127.0.99.132:40023), c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134:44343), 002cc3411acb4fdfa0b67c6f57f687a6 (127.0.99.135:38989), 6398c8993c3c45dcaab25b42142d5671 (127.0.99.129:36983), 89dd09f8aaee4b249ae3ce924a422a67 (127.0.99.130:38889), 5b8291b26a7f448b8636187d1caf86d0 (127.0.99.131:44479)
I20240503 02:00:26.116439 17355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec69ad86be3d4793b10aca98dc4c1736" candidate_uuid: "16d437fe629643948c95d628926d5ffa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d43ed6b8663947d99a72cd922cc323aa" is_pre_election: true
I20240503 02:00:26.116429 17751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec69ad86be3d4793b10aca98dc4c1736" candidate_uuid: "16d437fe629643948c95d628926d5ffa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" is_pre_election: true
I20240503 02:00:26.116528 16960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec69ad86be3d4793b10aca98dc4c1736" candidate_uuid: "16d437fe629643948c95d628926d5ffa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6398c8993c3c45dcaab25b42142d5671" is_pre_election: true
I20240503 02:00:26.116582 17751 raft_consensus.cc:2454] T ec69ad86be3d4793b10aca98dc4c1736 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16d437fe629643948c95d628926d5ffa in term 0.
I20240503 02:00:26.116588 17355 raft_consensus.cc:2454] T ec69ad86be3d4793b10aca98dc4c1736 P d43ed6b8663947d99a72cd922cc323aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16d437fe629643948c95d628926d5ffa in term 0.
I20240503 02:00:26.116629 16960 raft_consensus.cc:2454] T ec69ad86be3d4793b10aca98dc4c1736 P 6398c8993c3c45dcaab25b42142d5671 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16d437fe629643948c95d628926d5ffa in term 0.
I20240503 02:00:26.117108 17422 leader_election.cc:302] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 002cc3411acb4fdfa0b67c6f57f687a6, 16d437fe629643948c95d628926d5ffa, 6398c8993c3c45dcaab25b42142d5671, d43ed6b8663947d99a72cd922cc323aa; no voters:
I20240503 02:00:26.117239 17618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec69ad86be3d4793b10aca98dc4c1736" candidate_uuid: "16d437fe629643948c95d628926d5ffa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c843193b801b4f1a88f630ed3ed0f375" is_pre_election: true
I20240503 02:00:26.117376 17618 raft_consensus.cc:2454] T ec69ad86be3d4793b10aca98dc4c1736 P c843193b801b4f1a88f630ed3ed0f375 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16d437fe629643948c95d628926d5ffa in term 0.
I20240503 02:00:26.117415 18809 raft_consensus.cc:2781] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:26.117460 18809 raft_consensus.cc:482] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:26.117424 17223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec69ad86be3d4793b10aca98dc4c1736" candidate_uuid: "16d437fe629643948c95d628926d5ffa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b8291b26a7f448b8636187d1caf86d0" is_pre_election: true
I20240503 02:00:26.117506 18809 raft_consensus.cc:3037] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:26.117555 17223 raft_consensus.cc:2454] T ec69ad86be3d4793b10aca98dc4c1736 P 5b8291b26a7f448b8636187d1caf86d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16d437fe629643948c95d628926d5ffa in term 0.
I20240503 02:00:26.118351 18809 raft_consensus.cc:504] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:26.119031 18809 leader_election.cc:288] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa [CANDIDATE]: Term 1 election: Requested vote from peers d43ed6b8663947d99a72cd922cc323aa (127.0.99.132:40023), c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134:44343), 002cc3411acb4fdfa0b67c6f57f687a6 (127.0.99.135:38989), 6398c8993c3c45dcaab25b42142d5671 (127.0.99.129:36983), 89dd09f8aaee4b249ae3ce924a422a67 (127.0.99.130:38889), 5b8291b26a7f448b8636187d1caf86d0 (127.0.99.131:44479)
I20240503 02:00:26.119148 17355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec69ad86be3d4793b10aca98dc4c1736" candidate_uuid: "16d437fe629643948c95d628926d5ffa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d43ed6b8663947d99a72cd922cc323aa"
I20240503 02:00:26.119246 17355 raft_consensus.cc:3037] T ec69ad86be3d4793b10aca98dc4c1736 P d43ed6b8663947d99a72cd922cc323aa [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:26.119509 16960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec69ad86be3d4793b10aca98dc4c1736" candidate_uuid: "16d437fe629643948c95d628926d5ffa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6398c8993c3c45dcaab25b42142d5671"
I20240503 02:00:26.119625 16960 raft_consensus.cc:3037] T ec69ad86be3d4793b10aca98dc4c1736 P 6398c8993c3c45dcaab25b42142d5671 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:26.120290 17355 raft_consensus.cc:2454] T ec69ad86be3d4793b10aca98dc4c1736 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16d437fe629643948c95d628926d5ffa in term 1.
I20240503 02:00:26.120736 17751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec69ad86be3d4793b10aca98dc4c1736" candidate_uuid: "16d437fe629643948c95d628926d5ffa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "002cc3411acb4fdfa0b67c6f57f687a6"
I20240503 02:00:26.120854 17751 raft_consensus.cc:3037] T ec69ad86be3d4793b10aca98dc4c1736 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:26.120934 17618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec69ad86be3d4793b10aca98dc4c1736" candidate_uuid: "16d437fe629643948c95d628926d5ffa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c843193b801b4f1a88f630ed3ed0f375"
I20240503 02:00:26.121039 17618 raft_consensus.cc:3037] T ec69ad86be3d4793b10aca98dc4c1736 P c843193b801b4f1a88f630ed3ed0f375 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:26.121126 17091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec69ad86be3d4793b10aca98dc4c1736" candidate_uuid: "16d437fe629643948c95d628926d5ffa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89dd09f8aaee4b249ae3ce924a422a67"
I20240503 02:00:26.121241 17091 raft_consensus.cc:3037] T ec69ad86be3d4793b10aca98dc4c1736 P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:26.122004 17618 raft_consensus.cc:2454] T ec69ad86be3d4793b10aca98dc4c1736 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16d437fe629643948c95d628926d5ffa in term 1.
I20240503 02:00:26.122042 17751 raft_consensus.cc:2454] T ec69ad86be3d4793b10aca98dc4c1736 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16d437fe629643948c95d628926d5ffa in term 1.
I20240503 02:00:26.122213 17090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec69ad86be3d4793b10aca98dc4c1736" candidate_uuid: "16d437fe629643948c95d628926d5ffa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89dd09f8aaee4b249ae3ce924a422a67" is_pre_election: true
I20240503 02:00:26.122440 16960 raft_consensus.cc:2454] T ec69ad86be3d4793b10aca98dc4c1736 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16d437fe629643948c95d628926d5ffa in term 1.
I20240503 02:00:26.122820 17421 leader_election.cc:302] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 002cc3411acb4fdfa0b67c6f57f687a6, 16d437fe629643948c95d628926d5ffa, 6398c8993c3c45dcaab25b42142d5671, c843193b801b4f1a88f630ed3ed0f375; no voters:
I20240503 02:00:26.122984 18809 raft_consensus.cc:2781] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:26.123103 18809 raft_consensus.cc:686] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa [term 1 LEADER]: Becoming Leader. State: Replica: 16d437fe629643948c95d628926d5ffa, State: Running, Role: LEADER
I20240503 02:00:26.123476 17223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec69ad86be3d4793b10aca98dc4c1736" candidate_uuid: "16d437fe629643948c95d628926d5ffa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b8291b26a7f448b8636187d1caf86d0"
I20240503 02:00:26.123476 18809 consensus_queue.cc:237] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:26.123589 17223 raft_consensus.cc:3037] T ec69ad86be3d4793b10aca98dc4c1736 P 5b8291b26a7f448b8636187d1caf86d0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:26.122246 17091 raft_consensus.cc:2454] T ec69ad86be3d4793b10aca98dc4c1736 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16d437fe629643948c95d628926d5ffa in term 1.
I20240503 02:00:26.124581 17223 raft_consensus.cc:2454] T ec69ad86be3d4793b10aca98dc4c1736 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16d437fe629643948c95d628926d5ffa in term 1.
I20240503 02:00:26.126053 16811 catalog_manager.cc:5484] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa reported cstate change: term changed from 0 to 1, leader changed from <none> to 16d437fe629643948c95d628926d5ffa (127.0.99.133). New cstate: current_term: 1 leader_uuid: "16d437fe629643948c95d628926d5ffa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:26.229305 18809 raft_consensus.cc:482] T 42a900125241432a8bb31dad0b940968 P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:26.229449 18809 raft_consensus.cc:504] T 42a900125241432a8bb31dad0b940968 P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:26.230096 18809 leader_election.cc:288] T 42a900125241432a8bb31dad0b940968 P 16d437fe629643948c95d628926d5ffa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d43ed6b8663947d99a72cd922cc323aa (127.0.99.132:40023), c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134:44343), 002cc3411acb4fdfa0b67c6f57f687a6 (127.0.99.135:38989), 6398c8993c3c45dcaab25b42142d5671 (127.0.99.129:36983), 89dd09f8aaee4b249ae3ce924a422a67 (127.0.99.130:38889), 5b8291b26a7f448b8636187d1caf86d0 (127.0.99.131:44479)
I20240503 02:00:26.230450 17355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42a900125241432a8bb31dad0b940968" candidate_uuid: "16d437fe629643948c95d628926d5ffa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d43ed6b8663947d99a72cd922cc323aa" is_pre_election: true
I20240503 02:00:26.230603 17355 raft_consensus.cc:2454] T 42a900125241432a8bb31dad0b940968 P d43ed6b8663947d99a72cd922cc323aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16d437fe629643948c95d628926d5ffa in term 0.
I20240503 02:00:26.230546 17618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42a900125241432a8bb31dad0b940968" candidate_uuid: "16d437fe629643948c95d628926d5ffa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c843193b801b4f1a88f630ed3ed0f375" is_pre_election: true
I20240503 02:00:26.230762 17618 raft_consensus.cc:2454] T 42a900125241432a8bb31dad0b940968 P c843193b801b4f1a88f630ed3ed0f375 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16d437fe629643948c95d628926d5ffa in term 0.
I20240503 02:00:26.230782 16960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42a900125241432a8bb31dad0b940968" candidate_uuid: "16d437fe629643948c95d628926d5ffa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6398c8993c3c45dcaab25b42142d5671" is_pre_election: true
I20240503 02:00:26.230839 17091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42a900125241432a8bb31dad0b940968" candidate_uuid: "16d437fe629643948c95d628926d5ffa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89dd09f8aaee4b249ae3ce924a422a67" is_pre_election: true
I20240503 02:00:26.230926 16960 raft_consensus.cc:2454] T 42a900125241432a8bb31dad0b940968 P 6398c8993c3c45dcaab25b42142d5671 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16d437fe629643948c95d628926d5ffa in term 0.
I20240503 02:00:26.230976 17091 raft_consensus.cc:2454] T 42a900125241432a8bb31dad0b940968 P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16d437fe629643948c95d628926d5ffa in term 0.
I20240503 02:00:26.231119 17751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42a900125241432a8bb31dad0b940968" candidate_uuid: "16d437fe629643948c95d628926d5ffa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" is_pre_election: true
I20240503 02:00:26.231276 17751 raft_consensus.cc:2454] T 42a900125241432a8bb31dad0b940968 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16d437fe629643948c95d628926d5ffa in term 0.
I20240503 02:00:26.231293 17422 leader_election.cc:302] T 42a900125241432a8bb31dad0b940968 P 16d437fe629643948c95d628926d5ffa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 16d437fe629643948c95d628926d5ffa, 6398c8993c3c45dcaab25b42142d5671, c843193b801b4f1a88f630ed3ed0f375, d43ed6b8663947d99a72cd922cc323aa; no voters:
I20240503 02:00:26.231382 17223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42a900125241432a8bb31dad0b940968" candidate_uuid: "16d437fe629643948c95d628926d5ffa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b8291b26a7f448b8636187d1caf86d0" is_pre_election: true
I20240503 02:00:26.231442 18809 raft_consensus.cc:2781] T 42a900125241432a8bb31dad0b940968 P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:26.231499 18809 raft_consensus.cc:482] T 42a900125241432a8bb31dad0b940968 P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:26.231534 18809 raft_consensus.cc:3037] T 42a900125241432a8bb31dad0b940968 P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:26.231546 17223 raft_consensus.cc:2454] T 42a900125241432a8bb31dad0b940968 P 5b8291b26a7f448b8636187d1caf86d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16d437fe629643948c95d628926d5ffa in term 0.
I20240503 02:00:26.232431 18809 raft_consensus.cc:504] T 42a900125241432a8bb31dad0b940968 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:26.233175 17355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42a900125241432a8bb31dad0b940968" candidate_uuid: "16d437fe629643948c95d628926d5ffa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d43ed6b8663947d99a72cd922cc323aa"
I20240503 02:00:26.233281 17355 raft_consensus.cc:3037] T 42a900125241432a8bb31dad0b940968 P d43ed6b8663947d99a72cd922cc323aa [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:26.233332 17091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42a900125241432a8bb31dad0b940968" candidate_uuid: "16d437fe629643948c95d628926d5ffa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89dd09f8aaee4b249ae3ce924a422a67"
I20240503 02:00:26.233417 17091 raft_consensus.cc:3037] T 42a900125241432a8bb31dad0b940968 P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:26.234203 17751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42a900125241432a8bb31dad0b940968" candidate_uuid: "16d437fe629643948c95d628926d5ffa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "002cc3411acb4fdfa0b67c6f57f687a6"
I20240503 02:00:26.234290 17355 raft_consensus.cc:2454] T 42a900125241432a8bb31dad0b940968 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16d437fe629643948c95d628926d5ffa in term 1.
I20240503 02:00:26.234311 17091 raft_consensus.cc:2454] T 42a900125241432a8bb31dad0b940968 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16d437fe629643948c95d628926d5ffa in term 1.
I20240503 02:00:26.234319 17751 raft_consensus.cc:3037] T 42a900125241432a8bb31dad0b940968 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:26.234391 17618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42a900125241432a8bb31dad0b940968" candidate_uuid: "16d437fe629643948c95d628926d5ffa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c843193b801b4f1a88f630ed3ed0f375"
I20240503 02:00:26.234485 17618 raft_consensus.cc:3037] T 42a900125241432a8bb31dad0b940968 P c843193b801b4f1a88f630ed3ed0f375 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:26.234721 18809 leader_election.cc:288] T 42a900125241432a8bb31dad0b940968 P 16d437fe629643948c95d628926d5ffa [CANDIDATE]: Term 1 election: Requested vote from peers d43ed6b8663947d99a72cd922cc323aa (127.0.99.132:40023), c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134:44343), 002cc3411acb4fdfa0b67c6f57f687a6 (127.0.99.135:38989), 6398c8993c3c45dcaab25b42142d5671 (127.0.99.129:36983), 89dd09f8aaee4b249ae3ce924a422a67 (127.0.99.130:38889), 5b8291b26a7f448b8636187d1caf86d0 (127.0.99.131:44479)
I20240503 02:00:26.235299 17223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42a900125241432a8bb31dad0b940968" candidate_uuid: "16d437fe629643948c95d628926d5ffa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b8291b26a7f448b8636187d1caf86d0"
I20240503 02:00:26.235453 17223 raft_consensus.cc:3037] T 42a900125241432a8bb31dad0b940968 P 5b8291b26a7f448b8636187d1caf86d0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:26.235446 16960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42a900125241432a8bb31dad0b940968" candidate_uuid: "16d437fe629643948c95d628926d5ffa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6398c8993c3c45dcaab25b42142d5671"
I20240503 02:00:26.235543 16960 raft_consensus.cc:3037] T 42a900125241432a8bb31dad0b940968 P 6398c8993c3c45dcaab25b42142d5671 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:26.235634 17618 raft_consensus.cc:2454] T 42a900125241432a8bb31dad0b940968 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16d437fe629643948c95d628926d5ffa in term 1.
I20240503 02:00:26.235913 17421 leader_election.cc:302] T 42a900125241432a8bb31dad0b940968 P 16d437fe629643948c95d628926d5ffa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 16d437fe629643948c95d628926d5ffa, 89dd09f8aaee4b249ae3ce924a422a67, c843193b801b4f1a88f630ed3ed0f375, d43ed6b8663947d99a72cd922cc323aa; no voters:
I20240503 02:00:26.236106 18809 raft_consensus.cc:2781] T 42a900125241432a8bb31dad0b940968 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:26.236186 18809 raft_consensus.cc:686] T 42a900125241432a8bb31dad0b940968 P 16d437fe629643948c95d628926d5ffa [term 1 LEADER]: Becoming Leader. State: Replica: 16d437fe629643948c95d628926d5ffa, State: Running, Role: LEADER
I20240503 02:00:26.236346 18809 consensus_queue.cc:237] T 42a900125241432a8bb31dad0b940968 P 16d437fe629643948c95d628926d5ffa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:26.236913 17223 raft_consensus.cc:2454] T 42a900125241432a8bb31dad0b940968 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16d437fe629643948c95d628926d5ffa in term 1.
I20240503 02:00:26.236979 17751 raft_consensus.cc:2454] T 42a900125241432a8bb31dad0b940968 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16d437fe629643948c95d628926d5ffa in term 1.
I20240503 02:00:26.237838 16960 raft_consensus.cc:2454] T 42a900125241432a8bb31dad0b940968 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16d437fe629643948c95d628926d5ffa in term 1.
I20240503 02:00:26.239284 16811 catalog_manager.cc:5484] T 42a900125241432a8bb31dad0b940968 P 16d437fe629643948c95d628926d5ffa reported cstate change: term changed from 0 to 1, leader changed from <none> to 16d437fe629643948c95d628926d5ffa (127.0.99.133). New cstate: current_term: 1 leader_uuid: "16d437fe629643948c95d628926d5ffa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } health_report { overall_health: UNKNOWN } } }
W20240503 02:00:26.369160 18842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:00:26.369402 18842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:00:26.372269 18842 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_5'
I20240503 02:00:26.470439 17223 raft_consensus.cc:1261] T 78d106d705df4089a4dfae4d5dc533cc P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:26.471226 16960 raft_consensus.cc:1261] T 78d106d705df4089a4dfae4d5dc533cc P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:26.472537 17091 raft_consensus.cc:1261] T 42a900125241432a8bb31dad0b940968 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:26.472496 18838 consensus_queue.cc:1035] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:26.472954 18809 consensus_queue.cc:1035] T 42a900125241432a8bb31dad0b940968 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:26.473273 16960 raft_consensus.cc:1261] T 42a900125241432a8bb31dad0b940968 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:26.473742 18809 consensus_queue.cc:1035] T 42a900125241432a8bb31dad0b940968 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:26.476248 17618 raft_consensus.cc:1261] T 42a900125241432a8bb31dad0b940968 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:26.476677 17355 raft_consensus.cc:1261] T 42a900125241432a8bb31dad0b940968 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:26.477246 18809 consensus_queue.cc:1035] T 42a900125241432a8bb31dad0b940968 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:26.478094 18664 consensus_queue.cc:1035] T 42a900125241432a8bb31dad0b940968 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:26.479346 17223 raft_consensus.cc:1261] T 42a900125241432a8bb31dad0b940968 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:26.480530 17751 raft_consensus.cc:1261] T 42a900125241432a8bb31dad0b940968 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:26.480872 18838 consensus_queue.cc:1035] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:26.481638 18878 consensus_queue.cc:1035] T 42a900125241432a8bb31dad0b940968 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:26.482439 18878 consensus_queue.cc:1035] T 42a900125241432a8bb31dad0b940968 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:26.484023 17090 raft_consensus.cc:1261] T 78d106d705df4089a4dfae4d5dc533cc P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:26.485683 17620 raft_consensus.cc:1261] T 78d106d705df4089a4dfae4d5dc533cc P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:26.488245 17355 raft_consensus.cc:1261] T 78d106d705df4089a4dfae4d5dc533cc P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:26.491120 18589 consensus_queue.cc:1035] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:26.491295 18838 consensus_queue.cc:1035] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:26.491508 18589 consensus_queue.cc:1035] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:26.492024 17486 raft_consensus.cc:1261] T 78d106d705df4089a4dfae4d5dc533cc P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:26.492853 18589 consensus_queue.cc:1035] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:26.505051 17222 raft_consensus.cc:1261] T ec69ad86be3d4793b10aca98dc4c1736 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:26.505467 18664 consensus_queue.cc:1035] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:26.507503 16959 raft_consensus.cc:1261] T ec69ad86be3d4793b10aca98dc4c1736 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:26.508244 18664 consensus_queue.cc:1035] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:26.510927 17089 raft_consensus.cc:1261] T ec69ad86be3d4793b10aca98dc4c1736 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:26.511524 18809 consensus_queue.cc:1035] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:26.512567 17354 raft_consensus.cc:1261] T ec69ad86be3d4793b10aca98dc4c1736 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:26.513222 17619 raft_consensus.cc:1261] T ec69ad86be3d4793b10aca98dc4c1736 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:26.513319 17750 raft_consensus.cc:1261] T ec69ad86be3d4793b10aca98dc4c1736 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:26.513691 18809 consensus_queue.cc:1035] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:26.513890 18878 consensus_queue.cc:1035] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:26.523703 18809 consensus_queue.cc:1035] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 241.944 ms
time per row: 0.120972 ms
I20240503 02:00:26.665557 16813 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39916:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 7
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240503 02:00:26.666191 16813 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_6 in case of a server failure: 8 tablet servers would be needed, 7 are available. Consider bringing up more tablet servers.
I20240503 02:00:26.672971 17467 tablet_service.cc:1467] Processing CreateTablet for tablet 3f67591d6a3744ab90a3e4d2375a59db (DEFAULT_TABLE table=rebalance_test_table_6 [id=fc19f05fb43b48068855575b19b79556]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:26.673087 17600 tablet_service.cc:1467] Processing CreateTablet for tablet 3f67591d6a3744ab90a3e4d2375a59db (DEFAULT_TABLE table=rebalance_test_table_6 [id=fc19f05fb43b48068855575b19b79556]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:26.673353 17467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f67591d6a3744ab90a3e4d2375a59db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:26.673524 17600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f67591d6a3744ab90a3e4d2375a59db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:26.673507 16938 tablet_service.cc:1467] Processing CreateTablet for tablet 3f67591d6a3744ab90a3e4d2375a59db (DEFAULT_TABLE table=rebalance_test_table_6 [id=fc19f05fb43b48068855575b19b79556]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:26.673799 16938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f67591d6a3744ab90a3e4d2375a59db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:26.676225 17730 tablet_service.cc:1467] Processing CreateTablet for tablet 3f67591d6a3744ab90a3e4d2375a59db (DEFAULT_TABLE table=rebalance_test_table_6 [id=fc19f05fb43b48068855575b19b79556]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:26.676295 17336 tablet_service.cc:1467] Processing CreateTablet for tablet 3f67591d6a3744ab90a3e4d2375a59db (DEFAULT_TABLE table=rebalance_test_table_6 [id=fc19f05fb43b48068855575b19b79556]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:26.676538 17730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f67591d6a3744ab90a3e4d2375a59db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:26.676581 17336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f67591d6a3744ab90a3e4d2375a59db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:26.676594 16939 tablet_service.cc:1467] Processing CreateTablet for tablet 4782de9826d6460e9e993e01327034e4 (DEFAULT_TABLE table=rebalance_test_table_6 [id=fc19f05fb43b48068855575b19b79556]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:26.676844 16939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4782de9826d6460e9e993e01327034e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:26.677560 17334 tablet_service.cc:1467] Processing CreateTablet for tablet ab536115ac7a4ba193f718d421f4000d (DEFAULT_TABLE table=rebalance_test_table_6 [id=fc19f05fb43b48068855575b19b79556]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:26.678316 17731 tablet_service.cc:1467] Processing CreateTablet for tablet 4782de9826d6460e9e993e01327034e4 (DEFAULT_TABLE table=rebalance_test_table_6 [id=fc19f05fb43b48068855575b19b79556]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:26.678599 17731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4782de9826d6460e9e993e01327034e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:26.679000 17335 tablet_service.cc:1467] Processing CreateTablet for tablet 4782de9826d6460e9e993e01327034e4 (DEFAULT_TABLE table=rebalance_test_table_6 [id=fc19f05fb43b48068855575b19b79556]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:26.679397 17732 tablet_service.cc:1467] Processing CreateTablet for tablet ab536115ac7a4ba193f718d421f4000d (DEFAULT_TABLE table=rebalance_test_table_6 [id=fc19f05fb43b48068855575b19b79556]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:26.679602 17732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab536115ac7a4ba193f718d421f4000d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:26.679574 17468 tablet_service.cc:1467] Processing CreateTablet for tablet 4782de9826d6460e9e993e01327034e4 (DEFAULT_TABLE table=rebalance_test_table_6 [id=fc19f05fb43b48068855575b19b79556]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:26.679868 17468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4782de9826d6460e9e993e01327034e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:26.681757 17203 tablet_service.cc:1467] Processing CreateTablet for tablet 3f67591d6a3744ab90a3e4d2375a59db (DEFAULT_TABLE table=rebalance_test_table_6 [id=fc19f05fb43b48068855575b19b79556]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:26.682036 17203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f67591d6a3744ab90a3e4d2375a59db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:26.682397 17598 tablet_service.cc:1467] Processing CreateTablet for tablet 4782de9826d6460e9e993e01327034e4 (DEFAULT_TABLE table=rebalance_test_table_6 [id=fc19f05fb43b48068855575b19b79556]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:26.682349 16938 tablet_service.cc:1467] Processing CreateTablet for tablet ab536115ac7a4ba193f718d421f4000d (DEFAULT_TABLE table=rebalance_test_table_6 [id=fc19f05fb43b48068855575b19b79556]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:26.682652 17598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4782de9826d6460e9e993e01327034e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:26.682652 16938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab536115ac7a4ba193f718d421f4000d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:26.683609 17334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab536115ac7a4ba193f718d421f4000d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:26.684456 17335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4782de9826d6460e9e993e01327034e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:26.685015 17204 tablet_service.cc:1467] Processing CreateTablet for tablet ab536115ac7a4ba193f718d421f4000d (DEFAULT_TABLE table=rebalance_test_table_6 [id=fc19f05fb43b48068855575b19b79556]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:26.685356 17204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab536115ac7a4ba193f718d421f4000d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:26.686122 17071 tablet_service.cc:1467] Processing CreateTablet for tablet 3f67591d6a3744ab90a3e4d2375a59db (DEFAULT_TABLE table=rebalance_test_table_6 [id=fc19f05fb43b48068855575b19b79556]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:26.686123 17072 tablet_service.cc:1467] Processing CreateTablet for tablet ab536115ac7a4ba193f718d421f4000d (DEFAULT_TABLE table=rebalance_test_table_6 [id=fc19f05fb43b48068855575b19b79556]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:26.686436 17071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f67591d6a3744ab90a3e4d2375a59db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:26.686615 17072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab536115ac7a4ba193f718d421f4000d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:26.686852 17600 tablet_service.cc:1467] Processing CreateTablet for tablet ab536115ac7a4ba193f718d421f4000d (DEFAULT_TABLE table=rebalance_test_table_6 [id=fc19f05fb43b48068855575b19b79556]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:26.687088 17600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab536115ac7a4ba193f718d421f4000d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:26.687964 17202 tablet_service.cc:1467] Processing CreateTablet for tablet 4782de9826d6460e9e993e01327034e4 (DEFAULT_TABLE table=rebalance_test_table_6 [id=fc19f05fb43b48068855575b19b79556]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:26.688236 17202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4782de9826d6460e9e993e01327034e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:26.690620 18937 tablet_bootstrap.cc:492] T ab536115ac7a4ba193f718d421f4000d P 89dd09f8aaee4b249ae3ce924a422a67: Bootstrap starting.
I20240503 02:00:26.690822 17467 tablet_service.cc:1467] Processing CreateTablet for tablet ab536115ac7a4ba193f718d421f4000d (DEFAULT_TABLE table=rebalance_test_table_6 [id=fc19f05fb43b48068855575b19b79556]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:26.690879 17070 tablet_service.cc:1467] Processing CreateTablet for tablet 4782de9826d6460e9e993e01327034e4 (DEFAULT_TABLE table=rebalance_test_table_6 [id=fc19f05fb43b48068855575b19b79556]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:26.691092 17467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab536115ac7a4ba193f718d421f4000d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:26.691215 17070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4782de9826d6460e9e993e01327034e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:26.691892 18937 tablet_bootstrap.cc:654] T ab536115ac7a4ba193f718d421f4000d P 89dd09f8aaee4b249ae3ce924a422a67: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:26.694128 18938 tablet_bootstrap.cc:492] T 4782de9826d6460e9e993e01327034e4 P 002cc3411acb4fdfa0b67c6f57f687a6: Bootstrap starting.
I20240503 02:00:26.694636 18937 tablet_bootstrap.cc:492] T ab536115ac7a4ba193f718d421f4000d P 89dd09f8aaee4b249ae3ce924a422a67: No bootstrap required, opened a new log
I20240503 02:00:26.694716 18937 ts_tablet_manager.cc:1367] T ab536115ac7a4ba193f718d421f4000d P 89dd09f8aaee4b249ae3ce924a422a67: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:00:26.694953 18938 tablet_bootstrap.cc:654] T 4782de9826d6460e9e993e01327034e4 P 002cc3411acb4fdfa0b67c6f57f687a6: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:26.695243 18937 raft_consensus.cc:348] T ab536115ac7a4ba193f718d421f4000d P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:26.695394 18937 raft_consensus.cc:374] T ab536115ac7a4ba193f718d421f4000d P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:26.695434 18937 raft_consensus.cc:729] T ab536115ac7a4ba193f718d421f4000d P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89dd09f8aaee4b249ae3ce924a422a67, State: Initialized, Role: FOLLOWER
I20240503 02:00:26.695556 18937 consensus_queue.cc:260] T ab536115ac7a4ba193f718d421f4000d P 89dd09f8aaee4b249ae3ce924a422a67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:26.695628 18935 tablet_bootstrap.cc:492] T 3f67591d6a3744ab90a3e4d2375a59db P d43ed6b8663947d99a72cd922cc323aa: Bootstrap starting.
I20240503 02:00:26.695976 18937 ts_tablet_manager.cc:1398] T ab536115ac7a4ba193f718d421f4000d P 89dd09f8aaee4b249ae3ce924a422a67: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:26.696291 18937 tablet_bootstrap.cc:492] T 3f67591d6a3744ab90a3e4d2375a59db P 89dd09f8aaee4b249ae3ce924a422a67: Bootstrap starting.
I20240503 02:00:26.696525 18935 tablet_bootstrap.cc:654] T 3f67591d6a3744ab90a3e4d2375a59db P d43ed6b8663947d99a72cd922cc323aa: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:26.697357 18932 tablet_bootstrap.cc:492] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa: Bootstrap starting.
I20240503 02:00:26.698153 18936 tablet_bootstrap.cc:492] T 3f67591d6a3744ab90a3e4d2375a59db P 5b8291b26a7f448b8636187d1caf86d0: Bootstrap starting.
I20240503 02:00:26.698457 18937 tablet_bootstrap.cc:654] T 3f67591d6a3744ab90a3e4d2375a59db P 89dd09f8aaee4b249ae3ce924a422a67: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:26.699443 18932 tablet_bootstrap.cc:654] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:26.700373 18936 tablet_bootstrap.cc:654] T 3f67591d6a3744ab90a3e4d2375a59db P 5b8291b26a7f448b8636187d1caf86d0: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:26.700596 18933 tablet_bootstrap.cc:492] T 3f67591d6a3744ab90a3e4d2375a59db P 6398c8993c3c45dcaab25b42142d5671: Bootstrap starting.
I20240503 02:00:26.701813 18933 tablet_bootstrap.cc:654] T 3f67591d6a3744ab90a3e4d2375a59db P 6398c8993c3c45dcaab25b42142d5671: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:26.702221 18938 tablet_bootstrap.cc:492] T 4782de9826d6460e9e993e01327034e4 P 002cc3411acb4fdfa0b67c6f57f687a6: No bootstrap required, opened a new log
I20240503 02:00:26.702349 18938 ts_tablet_manager.cc:1367] T 4782de9826d6460e9e993e01327034e4 P 002cc3411acb4fdfa0b67c6f57f687a6: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 02:00:26.702842 18938 raft_consensus.cc:348] T 4782de9826d6460e9e993e01327034e4 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:26.703033 18938 raft_consensus.cc:374] T 4782de9826d6460e9e993e01327034e4 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:26.703099 18938 raft_consensus.cc:729] T 4782de9826d6460e9e993e01327034e4 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 002cc3411acb4fdfa0b67c6f57f687a6, State: Initialized, Role: FOLLOWER
I20240503 02:00:26.703258 18938 consensus_queue.cc:260] T 4782de9826d6460e9e993e01327034e4 P 002cc3411acb4fdfa0b67c6f57f687a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:26.704067 18938 ts_tablet_manager.cc:1398] T 4782de9826d6460e9e993e01327034e4 P 002cc3411acb4fdfa0b67c6f57f687a6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:26.704246 18938 tablet_bootstrap.cc:492] T ab536115ac7a4ba193f718d421f4000d P 002cc3411acb4fdfa0b67c6f57f687a6: Bootstrap starting.
I20240503 02:00:26.705264 18938 tablet_bootstrap.cc:654] T ab536115ac7a4ba193f718d421f4000d P 002cc3411acb4fdfa0b67c6f57f687a6: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:26.706283 18932 tablet_bootstrap.cc:492] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa: No bootstrap required, opened a new log
I20240503 02:00:26.706396 18932 ts_tablet_manager.cc:1367] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20240503 02:00:26.706892 18932 raft_consensus.cc:348] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:26.707046 18932 raft_consensus.cc:374] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:26.707150 18932 raft_consensus.cc:729] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16d437fe629643948c95d628926d5ffa, State: Initialized, Role: FOLLOWER
I20240503 02:00:26.707343 18932 consensus_queue.cc:260] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:26.707645 18932 ts_tablet_manager.cc:1398] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:26.707844 18932 tablet_bootstrap.cc:492] T 4782de9826d6460e9e993e01327034e4 P 16d437fe629643948c95d628926d5ffa: Bootstrap starting.
I20240503 02:00:26.708685 18932 tablet_bootstrap.cc:654] T 4782de9826d6460e9e993e01327034e4 P 16d437fe629643948c95d628926d5ffa: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:26.708983 18934 tablet_bootstrap.cc:492] T 3f67591d6a3744ab90a3e4d2375a59db P c843193b801b4f1a88f630ed3ed0f375: Bootstrap starting.
I20240503 02:00:26.710201 18935 tablet_bootstrap.cc:492] T 3f67591d6a3744ab90a3e4d2375a59db P d43ed6b8663947d99a72cd922cc323aa: No bootstrap required, opened a new log
I20240503 02:00:26.710312 18935 ts_tablet_manager.cc:1367] T 3f67591d6a3744ab90a3e4d2375a59db P d43ed6b8663947d99a72cd922cc323aa: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.000s
I20240503 02:00:26.710753 18935 raft_consensus.cc:348] T 3f67591d6a3744ab90a3e4d2375a59db P d43ed6b8663947d99a72cd922cc323aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:26.710880 18935 raft_consensus.cc:374] T 3f67591d6a3744ab90a3e4d2375a59db P d43ed6b8663947d99a72cd922cc323aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:26.710913 18935 raft_consensus.cc:729] T 3f67591d6a3744ab90a3e4d2375a59db P d43ed6b8663947d99a72cd922cc323aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d43ed6b8663947d99a72cd922cc323aa, State: Initialized, Role: FOLLOWER
I20240503 02:00:26.711081 18935 consensus_queue.cc:260] T 3f67591d6a3744ab90a3e4d2375a59db P d43ed6b8663947d99a72cd922cc323aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:26.711922 18935 ts_tablet_manager.cc:1398] T 3f67591d6a3744ab90a3e4d2375a59db P d43ed6b8663947d99a72cd922cc323aa: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:26.712110 18935 tablet_bootstrap.cc:492] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa: Bootstrap starting.
I20240503 02:00:26.713574 18935 tablet_bootstrap.cc:654] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:26.713843 18934 tablet_bootstrap.cc:654] T 3f67591d6a3744ab90a3e4d2375a59db P c843193b801b4f1a88f630ed3ed0f375: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:26.715126 18932 tablet_bootstrap.cc:492] T 4782de9826d6460e9e993e01327034e4 P 16d437fe629643948c95d628926d5ffa: No bootstrap required, opened a new log
I20240503 02:00:26.715231 18938 tablet_bootstrap.cc:492] T ab536115ac7a4ba193f718d421f4000d P 002cc3411acb4fdfa0b67c6f57f687a6: No bootstrap required, opened a new log
I20240503 02:00:26.715236 18934 tablet_bootstrap.cc:492] T 3f67591d6a3744ab90a3e4d2375a59db P c843193b801b4f1a88f630ed3ed0f375: No bootstrap required, opened a new log
I20240503 02:00:26.715299 18932 ts_tablet_manager.cc:1367] T 4782de9826d6460e9e993e01327034e4 P 16d437fe629643948c95d628926d5ffa: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 02:00:26.715345 18934 ts_tablet_manager.cc:1367] T 3f67591d6a3744ab90a3e4d2375a59db P c843193b801b4f1a88f630ed3ed0f375: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 02:00:26.715472 18938 ts_tablet_manager.cc:1367] T ab536115ac7a4ba193f718d421f4000d P 002cc3411acb4fdfa0b67c6f57f687a6: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240503 02:00:26.715924 18932 raft_consensus.cc:348] T 4782de9826d6460e9e993e01327034e4 P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:26.715927 18934 raft_consensus.cc:348] T 3f67591d6a3744ab90a3e4d2375a59db P c843193b801b4f1a88f630ed3ed0f375 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:26.716130 18932 raft_consensus.cc:374] T 4782de9826d6460e9e993e01327034e4 P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:26.716171 18934 raft_consensus.cc:374] T 3f67591d6a3744ab90a3e4d2375a59db P c843193b801b4f1a88f630ed3ed0f375 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:26.716257 18934 raft_consensus.cc:729] T 3f67591d6a3744ab90a3e4d2375a59db P c843193b801b4f1a88f630ed3ed0f375 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c843193b801b4f1a88f630ed3ed0f375, State: Initialized, Role: FOLLOWER
I20240503 02:00:26.716224 18932 raft_consensus.cc:729] T 4782de9826d6460e9e993e01327034e4 P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16d437fe629643948c95d628926d5ffa, State: Initialized, Role: FOLLOWER
I20240503 02:00:26.716456 18938 raft_consensus.cc:348] T ab536115ac7a4ba193f718d421f4000d P 002cc3411acb4fdfa0b67c6f57f687a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:26.716462 18932 consensus_queue.cc:260] T 4782de9826d6460e9e993e01327034e4 P 16d437fe629643948c95d628926d5ffa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:26.716451 18934 consensus_queue.cc:260] T 3f67591d6a3744ab90a3e4d2375a59db P c843193b801b4f1a88f630ed3ed0f375 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:26.716718 18938 raft_consensus.cc:374] T ab536115ac7a4ba193f718d421f4000d P 002cc3411acb4fdfa0b67c6f57f687a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:26.716810 18938 raft_consensus.cc:729] T ab536115ac7a4ba193f718d421f4000d P 002cc3411acb4fdfa0b67c6f57f687a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 002cc3411acb4fdfa0b67c6f57f687a6, State: Initialized, Role: FOLLOWER
I20240503 02:00:26.716982 18938 consensus_queue.cc:260] T ab536115ac7a4ba193f718d421f4000d P 002cc3411acb4fdfa0b67c6f57f687a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:26.717389 18938 ts_tablet_manager.cc:1398] T ab536115ac7a4ba193f718d421f4000d P 002cc3411acb4fdfa0b67c6f57f687a6: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:26.717531 18934 ts_tablet_manager.cc:1398] T 3f67591d6a3744ab90a3e4d2375a59db P c843193b801b4f1a88f630ed3ed0f375: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:26.717650 18938 tablet_bootstrap.cc:492] T 3f67591d6a3744ab90a3e4d2375a59db P 002cc3411acb4fdfa0b67c6f57f687a6: Bootstrap starting.
I20240503 02:00:26.718673 18938 tablet_bootstrap.cc:654] T 3f67591d6a3744ab90a3e4d2375a59db P 002cc3411acb4fdfa0b67c6f57f687a6: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:26.718732 18932 ts_tablet_manager.cc:1398] T 4782de9826d6460e9e993e01327034e4 P 16d437fe629643948c95d628926d5ffa: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 02:00:26.719058 18932 tablet_bootstrap.cc:492] T ab536115ac7a4ba193f718d421f4000d P 16d437fe629643948c95d628926d5ffa: Bootstrap starting.
I20240503 02:00:26.719352 18936 tablet_bootstrap.cc:492] T 3f67591d6a3744ab90a3e4d2375a59db P 5b8291b26a7f448b8636187d1caf86d0: No bootstrap required, opened a new log
I20240503 02:00:26.719532 18936 ts_tablet_manager.cc:1367] T 3f67591d6a3744ab90a3e4d2375a59db P 5b8291b26a7f448b8636187d1caf86d0: Time spent bootstrapping tablet: real 0.021s user 0.001s sys 0.002s
I20240503 02:00:26.720079 18936 raft_consensus.cc:348] T 3f67591d6a3744ab90a3e4d2375a59db P 5b8291b26a7f448b8636187d1caf86d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:26.720288 18936 raft_consensus.cc:374] T 3f67591d6a3744ab90a3e4d2375a59db P 5b8291b26a7f448b8636187d1caf86d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:26.720299 18932 tablet_bootstrap.cc:654] T ab536115ac7a4ba193f718d421f4000d P 16d437fe629643948c95d628926d5ffa: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:26.720392 18936 raft_consensus.cc:729] T 3f67591d6a3744ab90a3e4d2375a59db P 5b8291b26a7f448b8636187d1caf86d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b8291b26a7f448b8636187d1caf86d0, State: Initialized, Role: FOLLOWER
I20240503 02:00:26.720613 18936 consensus_queue.cc:260] T 3f67591d6a3744ab90a3e4d2375a59db P 5b8291b26a7f448b8636187d1caf86d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:26.721156 18936 ts_tablet_manager.cc:1398] T 3f67591d6a3744ab90a3e4d2375a59db P 5b8291b26a7f448b8636187d1caf86d0: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:26.721380 18936 tablet_bootstrap.cc:492] T ab536115ac7a4ba193f718d421f4000d P 5b8291b26a7f448b8636187d1caf86d0: Bootstrap starting.
I20240503 02:00:26.723359 18936 tablet_bootstrap.cc:654] T ab536115ac7a4ba193f718d421f4000d P 5b8291b26a7f448b8636187d1caf86d0: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:26.725236 18934 tablet_bootstrap.cc:492] T ab536115ac7a4ba193f718d421f4000d P c843193b801b4f1a88f630ed3ed0f375: Bootstrap starting.
I20240503 02:00:26.726222 18938 tablet_bootstrap.cc:492] T 3f67591d6a3744ab90a3e4d2375a59db P 002cc3411acb4fdfa0b67c6f57f687a6: No bootstrap required, opened a new log
I20240503 02:00:26.726259 18934 tablet_bootstrap.cc:654] T ab536115ac7a4ba193f718d421f4000d P c843193b801b4f1a88f630ed3ed0f375: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:26.726269 18935 tablet_bootstrap.cc:492] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa: No bootstrap required, opened a new log
I20240503 02:00:26.726377 18935 ts_tablet_manager.cc:1367] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240503 02:00:26.726495 18938 ts_tablet_manager.cc:1367] T 3f67591d6a3744ab90a3e4d2375a59db P 002cc3411acb4fdfa0b67c6f57f687a6: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 02:00:26.726850 18935 raft_consensus.cc:348] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:26.726996 18935 raft_consensus.cc:374] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:26.727030 18935 raft_consensus.cc:729] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d43ed6b8663947d99a72cd922cc323aa, State: Initialized, Role: FOLLOWER
I20240503 02:00:26.727169 18935 consensus_queue.cc:260] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:26.727473 18935 ts_tablet_manager.cc:1398] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:26.727625 18935 tablet_bootstrap.cc:492] T 4782de9826d6460e9e993e01327034e4 P d43ed6b8663947d99a72cd922cc323aa: Bootstrap starting.
I20240503 02:00:26.728508 18935 tablet_bootstrap.cc:654] T 4782de9826d6460e9e993e01327034e4 P d43ed6b8663947d99a72cd922cc323aa: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:26.728541 18936 tablet_bootstrap.cc:492] T ab536115ac7a4ba193f718d421f4000d P 5b8291b26a7f448b8636187d1caf86d0: No bootstrap required, opened a new log
I20240503 02:00:26.728798 18936 ts_tablet_manager.cc:1367] T ab536115ac7a4ba193f718d421f4000d P 5b8291b26a7f448b8636187d1caf86d0: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.001s
I20240503 02:00:26.729642 18935 tablet_bootstrap.cc:492] T 4782de9826d6460e9e993e01327034e4 P d43ed6b8663947d99a72cd922cc323aa: No bootstrap required, opened a new log
I20240503 02:00:26.729723 18935 ts_tablet_manager.cc:1367] T 4782de9826d6460e9e993e01327034e4 P d43ed6b8663947d99a72cd922cc323aa: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 02:00:26.729920 18936 raft_consensus.cc:348] T ab536115ac7a4ba193f718d421f4000d P 5b8291b26a7f448b8636187d1caf86d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:26.730146 18936 raft_consensus.cc:374] T ab536115ac7a4ba193f718d421f4000d P 5b8291b26a7f448b8636187d1caf86d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:26.730141 18935 raft_consensus.cc:348] T 4782de9826d6460e9e993e01327034e4 P d43ed6b8663947d99a72cd922cc323aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:26.730229 18935 raft_consensus.cc:374] T 4782de9826d6460e9e993e01327034e4 P d43ed6b8663947d99a72cd922cc323aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:26.730258 18935 raft_consensus.cc:729] T 4782de9826d6460e9e993e01327034e4 P d43ed6b8663947d99a72cd922cc323aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d43ed6b8663947d99a72cd922cc323aa, State: Initialized, Role: FOLLOWER
I20240503 02:00:26.730314 18936 raft_consensus.cc:729] T ab536115ac7a4ba193f718d421f4000d P 5b8291b26a7f448b8636187d1caf86d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b8291b26a7f448b8636187d1caf86d0, State: Initialized, Role: FOLLOWER
I20240503 02:00:26.730365 18935 consensus_queue.cc:260] T 4782de9826d6460e9e993e01327034e4 P d43ed6b8663947d99a72cd922cc323aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:26.730676 18935 ts_tablet_manager.cc:1398] T 4782de9826d6460e9e993e01327034e4 P d43ed6b8663947d99a72cd922cc323aa: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:26.730499 18936 consensus_queue.cc:260] T ab536115ac7a4ba193f718d421f4000d P 5b8291b26a7f448b8636187d1caf86d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:26.731130 18936 ts_tablet_manager.cc:1398] T ab536115ac7a4ba193f718d421f4000d P 5b8291b26a7f448b8636187d1caf86d0: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 02:00:26.731331 18936 tablet_bootstrap.cc:492] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0: Bootstrap starting.
I20240503 02:00:26.732309 18936 tablet_bootstrap.cc:654] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:26.732820 18938 raft_consensus.cc:348] T 3f67591d6a3744ab90a3e4d2375a59db P 002cc3411acb4fdfa0b67c6f57f687a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:26.733009 18938 raft_consensus.cc:374] T 3f67591d6a3744ab90a3e4d2375a59db P 002cc3411acb4fdfa0b67c6f57f687a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:26.733063 18938 raft_consensus.cc:729] T 3f67591d6a3744ab90a3e4d2375a59db P 002cc3411acb4fdfa0b67c6f57f687a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 002cc3411acb4fdfa0b67c6f57f687a6, State: Initialized, Role: FOLLOWER
I20240503 02:00:26.733183 18933 tablet_bootstrap.cc:492] T 3f67591d6a3744ab90a3e4d2375a59db P 6398c8993c3c45dcaab25b42142d5671: No bootstrap required, opened a new log
I20240503 02:00:26.733292 18933 ts_tablet_manager.cc:1367] T 3f67591d6a3744ab90a3e4d2375a59db P 6398c8993c3c45dcaab25b42142d5671: Time spent bootstrapping tablet: real 0.033s user 0.002s sys 0.000s
I20240503 02:00:26.733249 18938 consensus_queue.cc:260] T 3f67591d6a3744ab90a3e4d2375a59db P 002cc3411acb4fdfa0b67c6f57f687a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:26.733723 18938 ts_tablet_manager.cc:1398] T 3f67591d6a3744ab90a3e4d2375a59db P 002cc3411acb4fdfa0b67c6f57f687a6: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240503 02:00:26.733783 18933 raft_consensus.cc:348] T 3f67591d6a3744ab90a3e4d2375a59db P 6398c8993c3c45dcaab25b42142d5671 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:26.733945 18933 raft_consensus.cc:374] T 3f67591d6a3744ab90a3e4d2375a59db P 6398c8993c3c45dcaab25b42142d5671 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:26.733985 18933 raft_consensus.cc:729] T 3f67591d6a3744ab90a3e4d2375a59db P 6398c8993c3c45dcaab25b42142d5671 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6398c8993c3c45dcaab25b42142d5671, State: Initialized, Role: FOLLOWER
I20240503 02:00:26.734138 18933 consensus_queue.cc:260] T 3f67591d6a3744ab90a3e4d2375a59db P 6398c8993c3c45dcaab25b42142d5671 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:26.734505 18933 ts_tablet_manager.cc:1398] T 3f67591d6a3744ab90a3e4d2375a59db P 6398c8993c3c45dcaab25b42142d5671: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 02:00:26.734684 18933 tablet_bootstrap.cc:492] T 4782de9826d6460e9e993e01327034e4 P 6398c8993c3c45dcaab25b42142d5671: Bootstrap starting.
I20240503 02:00:26.735415 18932 tablet_bootstrap.cc:492] T ab536115ac7a4ba193f718d421f4000d P 16d437fe629643948c95d628926d5ffa: No bootstrap required, opened a new log
I20240503 02:00:26.735493 18932 ts_tablet_manager.cc:1367] T ab536115ac7a4ba193f718d421f4000d P 16d437fe629643948c95d628926d5ffa: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.001s
I20240503 02:00:26.735509 18937 tablet_bootstrap.cc:492] T 3f67591d6a3744ab90a3e4d2375a59db P 89dd09f8aaee4b249ae3ce924a422a67: No bootstrap required, opened a new log
I20240503 02:00:26.735615 18937 ts_tablet_manager.cc:1367] T 3f67591d6a3744ab90a3e4d2375a59db P 89dd09f8aaee4b249ae3ce924a422a67: Time spent bootstrapping tablet: real 0.039s user 0.002s sys 0.000s
I20240503 02:00:26.735692 18933 tablet_bootstrap.cc:654] T 4782de9826d6460e9e993e01327034e4 P 6398c8993c3c45dcaab25b42142d5671: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:26.735859 18932 raft_consensus.cc:348] T ab536115ac7a4ba193f718d421f4000d P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:26.735957 18932 raft_consensus.cc:374] T ab536115ac7a4ba193f718d421f4000d P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:26.735977 18932 raft_consensus.cc:729] T ab536115ac7a4ba193f718d421f4000d P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16d437fe629643948c95d628926d5ffa, State: Initialized, Role: FOLLOWER
I20240503 02:00:26.736099 18932 consensus_queue.cc:260] T ab536115ac7a4ba193f718d421f4000d P 16d437fe629643948c95d628926d5ffa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:26.736140 18937 raft_consensus.cc:348] T 3f67591d6a3744ab90a3e4d2375a59db P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:26.736284 18937 raft_consensus.cc:374] T 3f67591d6a3744ab90a3e4d2375a59db P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:26.736321 18937 raft_consensus.cc:729] T 3f67591d6a3744ab90a3e4d2375a59db P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89dd09f8aaee4b249ae3ce924a422a67, State: Initialized, Role: FOLLOWER
I20240503 02:00:26.736440 18932 ts_tablet_manager.cc:1398] T ab536115ac7a4ba193f718d421f4000d P 16d437fe629643948c95d628926d5ffa: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:26.736464 18937 consensus_queue.cc:260] T 3f67591d6a3744ab90a3e4d2375a59db P 89dd09f8aaee4b249ae3ce924a422a67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:26.736824 18937 ts_tablet_manager.cc:1398] T 3f67591d6a3744ab90a3e4d2375a59db P 89dd09f8aaee4b249ae3ce924a422a67: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:26.737025 18937 tablet_bootstrap.cc:492] T 4782de9826d6460e9e993e01327034e4 P 89dd09f8aaee4b249ae3ce924a422a67: Bootstrap starting.
I20240503 02:00:26.737514 18933 tablet_bootstrap.cc:492] T 4782de9826d6460e9e993e01327034e4 P 6398c8993c3c45dcaab25b42142d5671: No bootstrap required, opened a new log
I20240503 02:00:26.737555 18936 tablet_bootstrap.cc:492] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0: No bootstrap required, opened a new log
I20240503 02:00:26.737603 18933 ts_tablet_manager.cc:1367] T 4782de9826d6460e9e993e01327034e4 P 6398c8993c3c45dcaab25b42142d5671: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:26.737637 18936 ts_tablet_manager.cc:1367] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240503 02:00:26.737800 18937 tablet_bootstrap.cc:654] T 4782de9826d6460e9e993e01327034e4 P 89dd09f8aaee4b249ae3ce924a422a67: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:26.738099 18936 raft_consensus.cc:348] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:26.738092 18933 raft_consensus.cc:348] T 4782de9826d6460e9e993e01327034e4 P 6398c8993c3c45dcaab25b42142d5671 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:26.738225 18936 raft_consensus.cc:374] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:26.738233 18933 raft_consensus.cc:374] T 4782de9826d6460e9e993e01327034e4 P 6398c8993c3c45dcaab25b42142d5671 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:26.738265 18936 raft_consensus.cc:729] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b8291b26a7f448b8636187d1caf86d0, State: Initialized, Role: FOLLOWER
I20240503 02:00:26.738268 18933 raft_consensus.cc:729] T 4782de9826d6460e9e993e01327034e4 P 6398c8993c3c45dcaab25b42142d5671 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6398c8993c3c45dcaab25b42142d5671, State: Initialized, Role: FOLLOWER
I20240503 02:00:26.738399 18933 consensus_queue.cc:260] T 4782de9826d6460e9e993e01327034e4 P 6398c8993c3c45dcaab25b42142d5671 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:26.738415 18936 consensus_queue.cc:260] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:26.738703 18933 ts_tablet_manager.cc:1398] T 4782de9826d6460e9e993e01327034e4 P 6398c8993c3c45dcaab25b42142d5671: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:26.738873 18933 tablet_bootstrap.cc:492] T ab536115ac7a4ba193f718d421f4000d P 6398c8993c3c45dcaab25b42142d5671: Bootstrap starting.
I20240503 02:00:26.739194 18936 ts_tablet_manager.cc:1398] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:26.739557 18934 tablet_bootstrap.cc:492] T ab536115ac7a4ba193f718d421f4000d P c843193b801b4f1a88f630ed3ed0f375: No bootstrap required, opened a new log
I20240503 02:00:26.739667 18934 ts_tablet_manager.cc:1367] T ab536115ac7a4ba193f718d421f4000d P c843193b801b4f1a88f630ed3ed0f375: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240503 02:00:26.739781 18933 tablet_bootstrap.cc:654] T ab536115ac7a4ba193f718d421f4000d P 6398c8993c3c45dcaab25b42142d5671: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:26.740113 18873 raft_consensus.cc:482] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:26.740188 18934 raft_consensus.cc:348] T ab536115ac7a4ba193f718d421f4000d P c843193b801b4f1a88f630ed3ed0f375 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:26.740209 18873 raft_consensus.cc:504] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:26.740314 18934 raft_consensus.cc:374] T ab536115ac7a4ba193f718d421f4000d P c843193b801b4f1a88f630ed3ed0f375 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:26.740348 18934 raft_consensus.cc:729] T ab536115ac7a4ba193f718d421f4000d P c843193b801b4f1a88f630ed3ed0f375 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c843193b801b4f1a88f630ed3ed0f375, State: Initialized, Role: FOLLOWER
I20240503 02:00:26.740458 18934 consensus_queue.cc:260] T ab536115ac7a4ba193f718d421f4000d P c843193b801b4f1a88f630ed3ed0f375 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:26.740691 18934 ts_tablet_manager.cc:1398] T ab536115ac7a4ba193f718d421f4000d P c843193b801b4f1a88f630ed3ed0f375: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 02:00:26.740864 18934 tablet_bootstrap.cc:492] T 4782de9826d6460e9e993e01327034e4 P c843193b801b4f1a88f630ed3ed0f375: Bootstrap starting.
I20240503 02:00:26.740952 18873 leader_election.cc:288] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134:44343), d43ed6b8663947d99a72cd922cc323aa (127.0.99.132:40023), 002cc3411acb4fdfa0b67c6f57f687a6 (127.0.99.135:38989), 6398c8993c3c45dcaab25b42142d5671 (127.0.99.129:36983), 89dd09f8aaee4b249ae3ce924a422a67 (127.0.99.130:38889), 5b8291b26a7f448b8636187d1caf86d0 (127.0.99.131:44479)
I20240503 02:00:26.741299 17751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f67591d6a3744ab90a3e4d2375a59db" candidate_uuid: "16d437fe629643948c95d628926d5ffa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" is_pre_election: true
I20240503 02:00:26.741434 17751 raft_consensus.cc:2454] T 3f67591d6a3744ab90a3e4d2375a59db P 002cc3411acb4fdfa0b67c6f57f687a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16d437fe629643948c95d628926d5ffa in term 0.
I20240503 02:00:26.741554 17356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f67591d6a3744ab90a3e4d2375a59db" candidate_uuid: "16d437fe629643948c95d628926d5ffa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d43ed6b8663947d99a72cd922cc323aa" is_pre_election: true
I20240503 02:00:26.741672 17356 raft_consensus.cc:2454] T 3f67591d6a3744ab90a3e4d2375a59db P d43ed6b8663947d99a72cd922cc323aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16d437fe629643948c95d628926d5ffa in term 0.
I20240503 02:00:26.741783 18934 tablet_bootstrap.cc:654] T 4782de9826d6460e9e993e01327034e4 P c843193b801b4f1a88f630ed3ed0f375: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:26.741999 17222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f67591d6a3744ab90a3e4d2375a59db" candidate_uuid: "16d437fe629643948c95d628926d5ffa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b8291b26a7f448b8636187d1caf86d0" is_pre_election: true
I20240503 02:00:26.742126 17222 raft_consensus.cc:2454] T 3f67591d6a3744ab90a3e4d2375a59db P 5b8291b26a7f448b8636187d1caf86d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16d437fe629643948c95d628926d5ffa in term 0.
I20240503 02:00:26.742393 16957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f67591d6a3744ab90a3e4d2375a59db" candidate_uuid: "16d437fe629643948c95d628926d5ffa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6398c8993c3c45dcaab25b42142d5671" is_pre_election: true
I20240503 02:00:26.742516 16957 raft_consensus.cc:2454] T 3f67591d6a3744ab90a3e4d2375a59db P 6398c8993c3c45dcaab25b42142d5671 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16d437fe629643948c95d628926d5ffa in term 0.
I20240503 02:00:26.743041 17091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f67591d6a3744ab90a3e4d2375a59db" candidate_uuid: "16d437fe629643948c95d628926d5ffa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89dd09f8aaee4b249ae3ce924a422a67" is_pre_election: true
I20240503 02:00:26.743155 17091 raft_consensus.cc:2454] T 3f67591d6a3744ab90a3e4d2375a59db P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16d437fe629643948c95d628926d5ffa in term 0.
I20240503 02:00:26.743795 17423 leader_election.cc:302] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 16d437fe629643948c95d628926d5ffa, 5b8291b26a7f448b8636187d1caf86d0, 6398c8993c3c45dcaab25b42142d5671, d43ed6b8663947d99a72cd922cc323aa; no voters:
I20240503 02:00:26.743988 18873 raft_consensus.cc:2781] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:26.744048 18873 raft_consensus.cc:482] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:26.744086 18873 raft_consensus.cc:3037] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:26.744633 17618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f67591d6a3744ab90a3e4d2375a59db" candidate_uuid: "16d437fe629643948c95d628926d5ffa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c843193b801b4f1a88f630ed3ed0f375" is_pre_election: true
I20240503 02:00:26.744839 17618 raft_consensus.cc:2454] T 3f67591d6a3744ab90a3e4d2375a59db P c843193b801b4f1a88f630ed3ed0f375 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16d437fe629643948c95d628926d5ffa in term 0.
I20240503 02:00:26.745332 18873 raft_consensus.cc:504] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:26.746035 18873 leader_election.cc:288] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa [CANDIDATE]: Term 1 election: Requested vote from peers c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134:44343), d43ed6b8663947d99a72cd922cc323aa (127.0.99.132:40023), 002cc3411acb4fdfa0b67c6f57f687a6 (127.0.99.135:38989), 6398c8993c3c45dcaab25b42142d5671 (127.0.99.129:36983), 89dd09f8aaee4b249ae3ce924a422a67 (127.0.99.130:38889), 5b8291b26a7f448b8636187d1caf86d0 (127.0.99.131:44479)
I20240503 02:00:26.746344 17618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f67591d6a3744ab90a3e4d2375a59db" candidate_uuid: "16d437fe629643948c95d628926d5ffa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c843193b801b4f1a88f630ed3ed0f375"
I20240503 02:00:26.746443 17618 raft_consensus.cc:3037] T 3f67591d6a3744ab90a3e4d2375a59db P c843193b801b4f1a88f630ed3ed0f375 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:26.747094 18934 tablet_bootstrap.cc:492] T 4782de9826d6460e9e993e01327034e4 P c843193b801b4f1a88f630ed3ed0f375: No bootstrap required, opened a new log
I20240503 02:00:26.747193 18934 ts_tablet_manager.cc:1367] T 4782de9826d6460e9e993e01327034e4 P c843193b801b4f1a88f630ed3ed0f375: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 02:00:26.747556 17618 raft_consensus.cc:2454] T 3f67591d6a3744ab90a3e4d2375a59db P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16d437fe629643948c95d628926d5ffa in term 1.
I20240503 02:00:26.747534 17356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f67591d6a3744ab90a3e4d2375a59db" candidate_uuid: "16d437fe629643948c95d628926d5ffa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d43ed6b8663947d99a72cd922cc323aa"
I20240503 02:00:26.747665 17356 raft_consensus.cc:3037] T 3f67591d6a3744ab90a3e4d2375a59db P d43ed6b8663947d99a72cd922cc323aa [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:26.747646 18934 raft_consensus.cc:348] T 4782de9826d6460e9e993e01327034e4 P c843193b801b4f1a88f630ed3ed0f375 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:26.747771 18934 raft_consensus.cc:374] T 4782de9826d6460e9e993e01327034e4 P c843193b801b4f1a88f630ed3ed0f375 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:26.747802 18934 raft_consensus.cc:729] T 4782de9826d6460e9e993e01327034e4 P c843193b801b4f1a88f630ed3ed0f375 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c843193b801b4f1a88f630ed3ed0f375, State: Initialized, Role: FOLLOWER
I20240503 02:00:26.747915 18934 consensus_queue.cc:260] T 4782de9826d6460e9e993e01327034e4 P c843193b801b4f1a88f630ed3ed0f375 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:26.748186 18934 ts_tablet_manager.cc:1398] T 4782de9826d6460e9e993e01327034e4 P c843193b801b4f1a88f630ed3ed0f375: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:26.748733 17356 raft_consensus.cc:2454] T 3f67591d6a3744ab90a3e4d2375a59db P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16d437fe629643948c95d628926d5ffa in term 1.
I20240503 02:00:26.748786 18937 tablet_bootstrap.cc:492] T 4782de9826d6460e9e993e01327034e4 P 89dd09f8aaee4b249ae3ce924a422a67: No bootstrap required, opened a new log
I20240503 02:00:26.748875 18937 ts_tablet_manager.cc:1367] T 4782de9826d6460e9e993e01327034e4 P 89dd09f8aaee4b249ae3ce924a422a67: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.000s
I20240503 02:00:26.749329 18937 raft_consensus.cc:348] T 4782de9826d6460e9e993e01327034e4 P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:26.749459 18937 raft_consensus.cc:374] T 4782de9826d6460e9e993e01327034e4 P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:26.749486 18937 raft_consensus.cc:729] T 4782de9826d6460e9e993e01327034e4 P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89dd09f8aaee4b249ae3ce924a422a67, State: Initialized, Role: FOLLOWER
I20240503 02:00:26.749620 18937 consensus_queue.cc:260] T 4782de9826d6460e9e993e01327034e4 P 89dd09f8aaee4b249ae3ce924a422a67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:26.749907 18937 ts_tablet_manager.cc:1398] T 4782de9826d6460e9e993e01327034e4 P 89dd09f8aaee4b249ae3ce924a422a67: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:26.750200 17091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f67591d6a3744ab90a3e4d2375a59db" candidate_uuid: "16d437fe629643948c95d628926d5ffa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89dd09f8aaee4b249ae3ce924a422a67"
I20240503 02:00:26.750317 17091 raft_consensus.cc:3037] T 3f67591d6a3744ab90a3e4d2375a59db P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:26.751399 17091 raft_consensus.cc:2454] T 3f67591d6a3744ab90a3e4d2375a59db P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16d437fe629643948c95d628926d5ffa in term 1.
I20240503 02:00:26.751876 17222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f67591d6a3744ab90a3e4d2375a59db" candidate_uuid: "16d437fe629643948c95d628926d5ffa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b8291b26a7f448b8636187d1caf86d0"
I20240503 02:00:26.752048 17222 raft_consensus.cc:3037] T 3f67591d6a3744ab90a3e4d2375a59db P 5b8291b26a7f448b8636187d1caf86d0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:26.753144 17222 raft_consensus.cc:2454] T 3f67591d6a3744ab90a3e4d2375a59db P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16d437fe629643948c95d628926d5ffa in term 1.
I20240503 02:00:26.753371 18690 raft_consensus.cc:482] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:26.753468 18690 raft_consensus.cc:504] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:26.754187 18690 leader_election.cc:288] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89dd09f8aaee4b249ae3ce924a422a67 (127.0.99.130:38889), 6398c8993c3c45dcaab25b42142d5671 (127.0.99.129:36983), d43ed6b8663947d99a72cd922cc323aa (127.0.99.132:40023), 16d437fe629643948c95d628926d5ffa (127.0.99.133:46581), c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134:44343), 002cc3411acb4fdfa0b67c6f57f687a6 (127.0.99.135:38989)
I20240503 02:00:26.754561 18692 raft_consensus.cc:482] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:26.754676 18692 raft_consensus.cc:504] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:26.755362 18692 leader_election.cc:288] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6398c8993c3c45dcaab25b42142d5671 (127.0.99.129:36983), 89dd09f8aaee4b249ae3ce924a422a67 (127.0.99.130:38889), 5b8291b26a7f448b8636187d1caf86d0 (127.0.99.131:44479), 16d437fe629643948c95d628926d5ffa (127.0.99.133:46581), c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134:44343), 002cc3411acb4fdfa0b67c6f57f687a6 (127.0.99.135:38989)
I20240503 02:00:26.756328 16959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab536115ac7a4ba193f718d421f4000d" candidate_uuid: "d43ed6b8663947d99a72cd922cc323aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6398c8993c3c45dcaab25b42142d5671" is_pre_election: true
I20240503 02:00:26.756428 16958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4782de9826d6460e9e993e01327034e4" candidate_uuid: "5b8291b26a7f448b8636187d1caf86d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6398c8993c3c45dcaab25b42142d5671" is_pre_election: true
I20240503 02:00:26.756561 16958 raft_consensus.cc:2454] T 4782de9826d6460e9e993e01327034e4 P 6398c8993c3c45dcaab25b42142d5671 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b8291b26a7f448b8636187d1caf86d0 in term 0.
I20240503 02:00:26.756922 17751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab536115ac7a4ba193f718d421f4000d" candidate_uuid: "d43ed6b8663947d99a72cd922cc323aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" is_pre_election: true
I20240503 02:00:26.757068 17751 raft_consensus.cc:2454] T ab536115ac7a4ba193f718d421f4000d P 002cc3411acb4fdfa0b67c6f57f687a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d43ed6b8663947d99a72cd922cc323aa in term 0.
I20240503 02:00:26.757210 16957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f67591d6a3744ab90a3e4d2375a59db" candidate_uuid: "16d437fe629643948c95d628926d5ffa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6398c8993c3c45dcaab25b42142d5671"
W20240503 02:00:26.757241 17291 leader_election.cc:341] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6398c8993c3c45dcaab25b42142d5671 (127.0.99.129:36983): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:00:26.757314 16957 raft_consensus.cc:3037] T 3f67591d6a3744ab90a3e4d2375a59db P 6398c8993c3c45dcaab25b42142d5671 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:26.757725 17091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab536115ac7a4ba193f718d421f4000d" candidate_uuid: "d43ed6b8663947d99a72cd922cc323aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89dd09f8aaee4b249ae3ce924a422a67" is_pre_election: true
I20240503 02:00:26.757833 17091 raft_consensus.cc:2454] T ab536115ac7a4ba193f718d421f4000d P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d43ed6b8663947d99a72cd922cc323aa in term 0.
I20240503 02:00:26.758498 16957 raft_consensus.cc:2454] T 3f67591d6a3744ab90a3e4d2375a59db P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16d437fe629643948c95d628926d5ffa in term 1.
I20240503 02:00:26.758572 17618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab536115ac7a4ba193f718d421f4000d" candidate_uuid: "d43ed6b8663947d99a72cd922cc323aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c843193b801b4f1a88f630ed3ed0f375" is_pre_election: true
I20240503 02:00:26.758706 17618 raft_consensus.cc:2454] T ab536115ac7a4ba193f718d421f4000d P c843193b801b4f1a88f630ed3ed0f375 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d43ed6b8663947d99a72cd922cc323aa in term 0.
I20240503 02:00:26.758822 17422 leader_election.cc:302] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 16d437fe629643948c95d628926d5ffa, 5b8291b26a7f448b8636187d1caf86d0, 89dd09f8aaee4b249ae3ce924a422a67, d43ed6b8663947d99a72cd922cc323aa; no voters:
I20240503 02:00:26.759060 18664 raft_consensus.cc:2781] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:26.759150 18664 raft_consensus.cc:686] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa [term 1 LEADER]: Becoming Leader. State: Replica: 16d437fe629643948c95d628926d5ffa, State: Running, Role: LEADER
I20240503 02:00:26.759135 17289 leader_election.cc:302] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 5 responses out of 7 voters: 4 yes votes; 1 no votes. yes voters: 002cc3411acb4fdfa0b67c6f57f687a6, 89dd09f8aaee4b249ae3ce924a422a67, c843193b801b4f1a88f630ed3ed0f375, d43ed6b8663947d99a72cd922cc323aa; no voters: 6398c8993c3c45dcaab25b42142d5671
I20240503 02:00:26.759303 18664 consensus_queue.cc:237] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:26.759423 18692 raft_consensus.cc:2781] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:26.759469 18692 raft_consensus.cc:482] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:26.759502 18692 raft_consensus.cc:3037] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:26.759663 17486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4782de9826d6460e9e993e01327034e4" candidate_uuid: "5b8291b26a7f448b8636187d1caf86d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16d437fe629643948c95d628926d5ffa" is_pre_election: true
I20240503 02:00:26.759776 17486 raft_consensus.cc:2454] T 4782de9826d6460e9e993e01327034e4 P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b8291b26a7f448b8636187d1caf86d0 in term 0.
I20240503 02:00:26.760515 18692 raft_consensus.cc:504] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:26.760933 17486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab536115ac7a4ba193f718d421f4000d" candidate_uuid: "d43ed6b8663947d99a72cd922cc323aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16d437fe629643948c95d628926d5ffa" is_pre_election: true
I20240503 02:00:26.761039 17486 raft_consensus.cc:2454] T ab536115ac7a4ba193f718d421f4000d P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d43ed6b8663947d99a72cd922cc323aa in term 0.
I20240503 02:00:26.761173 18692 leader_election.cc:288] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa [CANDIDATE]: Term 1 election: Requested vote from peers 6398c8993c3c45dcaab25b42142d5671 (127.0.99.129:36983), 89dd09f8aaee4b249ae3ce924a422a67 (127.0.99.130:38889), 5b8291b26a7f448b8636187d1caf86d0 (127.0.99.131:44479), 16d437fe629643948c95d628926d5ffa (127.0.99.133:46581), c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134:44343), 002cc3411acb4fdfa0b67c6f57f687a6 (127.0.99.135:38989)
I20240503 02:00:26.761327 17356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4782de9826d6460e9e993e01327034e4" candidate_uuid: "5b8291b26a7f448b8636187d1caf86d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d43ed6b8663947d99a72cd922cc323aa" is_pre_election: true
I20240503 02:00:26.761360 17750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4782de9826d6460e9e993e01327034e4" candidate_uuid: "5b8291b26a7f448b8636187d1caf86d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" is_pre_election: true
I20240503 02:00:26.761471 17356 raft_consensus.cc:2454] T 4782de9826d6460e9e993e01327034e4 P d43ed6b8663947d99a72cd922cc323aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b8291b26a7f448b8636187d1caf86d0 in term 0.
I20240503 02:00:26.761482 17750 raft_consensus.cc:2454] T 4782de9826d6460e9e993e01327034e4 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b8291b26a7f448b8636187d1caf86d0 in term 0.
I20240503 02:00:26.761643 17751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f67591d6a3744ab90a3e4d2375a59db" candidate_uuid: "16d437fe629643948c95d628926d5ffa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "002cc3411acb4fdfa0b67c6f57f687a6"
I20240503 02:00:26.761736 17751 raft_consensus.cc:3037] T 3f67591d6a3744ab90a3e4d2375a59db P 002cc3411acb4fdfa0b67c6f57f687a6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:26.761950 17158 leader_election.cc:302] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 16d437fe629643948c95d628926d5ffa, 5b8291b26a7f448b8636187d1caf86d0, 6398c8993c3c45dcaab25b42142d5671, d43ed6b8663947d99a72cd922cc323aa; no voters:
I20240503 02:00:26.762923 17751 raft_consensus.cc:2454] T 3f67591d6a3744ab90a3e4d2375a59db P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16d437fe629643948c95d628926d5ffa in term 1.
I20240503 02:00:26.763142 17223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab536115ac7a4ba193f718d421f4000d" candidate_uuid: "d43ed6b8663947d99a72cd922cc323aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b8291b26a7f448b8636187d1caf86d0"
I20240503 02:00:26.763267 17223 raft_consensus.cc:3037] T ab536115ac7a4ba193f718d421f4000d P 5b8291b26a7f448b8636187d1caf86d0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:26.763561 17620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab536115ac7a4ba193f718d421f4000d" candidate_uuid: "d43ed6b8663947d99a72cd922cc323aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c843193b801b4f1a88f630ed3ed0f375"
I20240503 02:00:26.763669 17620 raft_consensus.cc:3037] T ab536115ac7a4ba193f718d421f4000d P c843193b801b4f1a88f630ed3ed0f375 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:26.763880 18933 tablet_bootstrap.cc:492] T ab536115ac7a4ba193f718d421f4000d P 6398c8993c3c45dcaab25b42142d5671: No bootstrap required, opened a new log
I20240503 02:00:26.763984 18933 ts_tablet_manager.cc:1367] T ab536115ac7a4ba193f718d421f4000d P 6398c8993c3c45dcaab25b42142d5671: Time spent bootstrapping tablet: real 0.025s user 0.001s sys 0.000s
I20240503 02:00:26.764277 17223 raft_consensus.cc:2454] T ab536115ac7a4ba193f718d421f4000d P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d43ed6b8663947d99a72cd922cc323aa in term 1.
I20240503 02:00:26.764422 18933 raft_consensus.cc:348] T ab536115ac7a4ba193f718d421f4000d P 6398c8993c3c45dcaab25b42142d5671 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:26.764554 18933 raft_consensus.cc:374] T ab536115ac7a4ba193f718d421f4000d P 6398c8993c3c45dcaab25b42142d5671 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:26.764585 17620 raft_consensus.cc:2454] T ab536115ac7a4ba193f718d421f4000d P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d43ed6b8663947d99a72cd922cc323aa in term 1.
I20240503 02:00:26.764602 18933 raft_consensus.cc:729] T ab536115ac7a4ba193f718d421f4000d P 6398c8993c3c45dcaab25b42142d5671 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6398c8993c3c45dcaab25b42142d5671, State: Initialized, Role: FOLLOWER
I20240503 02:00:26.764792 18933 consensus_queue.cc:260] T ab536115ac7a4ba193f718d421f4000d P 6398c8993c3c45dcaab25b42142d5671 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:26.765148 18933 ts_tablet_manager.cc:1398] T ab536115ac7a4ba193f718d421f4000d P 6398c8993c3c45dcaab25b42142d5671: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 02:00:26.765318 18690 raft_consensus.cc:2781] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:26.765301 17751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab536115ac7a4ba193f718d421f4000d" candidate_uuid: "d43ed6b8663947d99a72cd922cc323aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "002cc3411acb4fdfa0b67c6f57f687a6"
I20240503 02:00:26.765375 18690 raft_consensus.cc:482] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:26.765416 18690 raft_consensus.cc:3037] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:26.765422 17751 raft_consensus.cc:3037] T ab536115ac7a4ba193f718d421f4000d P 002cc3411acb4fdfa0b67c6f57f687a6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:26.766045 17091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4782de9826d6460e9e993e01327034e4" candidate_uuid: "5b8291b26a7f448b8636187d1caf86d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89dd09f8aaee4b249ae3ce924a422a67" is_pre_election: true
I20240503 02:00:26.766167 17091 raft_consensus.cc:2454] T 4782de9826d6460e9e993e01327034e4 P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b8291b26a7f448b8636187d1caf86d0 in term 0.
I20240503 02:00:26.766402 18690 raft_consensus.cc:504] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:26.766564 17222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab536115ac7a4ba193f718d421f4000d" candidate_uuid: "d43ed6b8663947d99a72cd922cc323aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b8291b26a7f448b8636187d1caf86d0" is_pre_election: true
I20240503 02:00:26.766610 17751 raft_consensus.cc:2454] T ab536115ac7a4ba193f718d421f4000d P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d43ed6b8663947d99a72cd922cc323aa in term 1.
I20240503 02:00:26.766700 17222 raft_consensus.cc:2362] T ab536115ac7a4ba193f718d421f4000d P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate d43ed6b8663947d99a72cd922cc323aa in term 1. Re-sending same reply.
I20240503 02:00:26.767045 18690 leader_election.cc:288] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 [CANDIDATE]: Term 1 election: Requested vote from peers 89dd09f8aaee4b249ae3ce924a422a67 (127.0.99.130:38889), 6398c8993c3c45dcaab25b42142d5671 (127.0.99.129:36983), d43ed6b8663947d99a72cd922cc323aa (127.0.99.132:40023), 16d437fe629643948c95d628926d5ffa (127.0.99.133:46581), c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134:44343), 002cc3411acb4fdfa0b67c6f57f687a6 (127.0.99.135:38989)
I20240503 02:00:26.767194 17091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4782de9826d6460e9e993e01327034e4" candidate_uuid: "5b8291b26a7f448b8636187d1caf86d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89dd09f8aaee4b249ae3ce924a422a67"
I20240503 02:00:26.767313 17091 raft_consensus.cc:3037] T 4782de9826d6460e9e993e01327034e4 P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:26.768458 16958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4782de9826d6460e9e993e01327034e4" candidate_uuid: "5b8291b26a7f448b8636187d1caf86d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6398c8993c3c45dcaab25b42142d5671"
I20240503 02:00:26.768566 16958 raft_consensus.cc:3037] T 4782de9826d6460e9e993e01327034e4 P 6398c8993c3c45dcaab25b42142d5671 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:26.768610 17356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4782de9826d6460e9e993e01327034e4" candidate_uuid: "5b8291b26a7f448b8636187d1caf86d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d43ed6b8663947d99a72cd922cc323aa"
I20240503 02:00:26.768769 17356 raft_consensus.cc:3037] T 4782de9826d6460e9e993e01327034e4 P d43ed6b8663947d99a72cd922cc323aa [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:26.769227 16810 catalog_manager.cc:5484] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa reported cstate change: term changed from 0 to 1, leader changed from <none> to 16d437fe629643948c95d628926d5ffa (127.0.99.133). New cstate: current_term: 1 leader_uuid: "16d437fe629643948c95d628926d5ffa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:26.769555 16958 raft_consensus.cc:2454] T 4782de9826d6460e9e993e01327034e4 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b8291b26a7f448b8636187d1caf86d0 in term 1.
I20240503 02:00:26.769687 17356 raft_consensus.cc:2454] T 4782de9826d6460e9e993e01327034e4 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b8291b26a7f448b8636187d1caf86d0 in term 1.
I20240503 02:00:26.770638 17620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4782de9826d6460e9e993e01327034e4" candidate_uuid: "5b8291b26a7f448b8636187d1caf86d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c843193b801b4f1a88f630ed3ed0f375"
I20240503 02:00:26.770778 17620 raft_consensus.cc:3037] T 4782de9826d6460e9e993e01327034e4 P c843193b801b4f1a88f630ed3ed0f375 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:26.771008 17486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4782de9826d6460e9e993e01327034e4" candidate_uuid: "5b8291b26a7f448b8636187d1caf86d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16d437fe629643948c95d628926d5ffa"
I20240503 02:00:26.771121 17486 raft_consensus.cc:3037] T 4782de9826d6460e9e993e01327034e4 P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:26.771474 17289 leader_election.cc:302] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 002cc3411acb4fdfa0b67c6f57f687a6, 5b8291b26a7f448b8636187d1caf86d0, c843193b801b4f1a88f630ed3ed0f375, d43ed6b8663947d99a72cd922cc323aa; no voters:
I20240503 02:00:26.771832 17620 raft_consensus.cc:2454] T 4782de9826d6460e9e993e01327034e4 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b8291b26a7f448b8636187d1caf86d0 in term 1.
I20240503 02:00:26.771971 17487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab536115ac7a4ba193f718d421f4000d" candidate_uuid: "d43ed6b8663947d99a72cd922cc323aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16d437fe629643948c95d628926d5ffa"
I20240503 02:00:26.772075 17486 raft_consensus.cc:2454] T 4782de9826d6460e9e993e01327034e4 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b8291b26a7f448b8636187d1caf86d0 in term 1.
I20240503 02:00:26.772086 17487 raft_consensus.cc:3037] T ab536115ac7a4ba193f718d421f4000d P 16d437fe629643948c95d628926d5ffa [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:26.772274 17157 leader_election.cc:302] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 5b8291b26a7f448b8636187d1caf86d0, 6398c8993c3c45dcaab25b42142d5671, c843193b801b4f1a88f630ed3ed0f375, d43ed6b8663947d99a72cd922cc323aa; no voters:
I20240503 02:00:26.772989 16957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab536115ac7a4ba193f718d421f4000d" candidate_uuid: "d43ed6b8663947d99a72cd922cc323aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6398c8993c3c45dcaab25b42142d5671"
I20240503 02:00:26.773099 16957 raft_consensus.cc:3037] T ab536115ac7a4ba193f718d421f4000d P 6398c8993c3c45dcaab25b42142d5671 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:26.773258 17487 raft_consensus.cc:2454] T ab536115ac7a4ba193f718d421f4000d P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d43ed6b8663947d99a72cd922cc323aa in term 1.
I20240503 02:00:26.773355 17090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab536115ac7a4ba193f718d421f4000d" candidate_uuid: "d43ed6b8663947d99a72cd922cc323aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89dd09f8aaee4b249ae3ce924a422a67"
I20240503 02:00:26.773465 17090 raft_consensus.cc:3037] T ab536115ac7a4ba193f718d421f4000d P 89dd09f8aaee4b249ae3ce924a422a67 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:26.773998 16957 raft_consensus.cc:2454] T ab536115ac7a4ba193f718d421f4000d P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d43ed6b8663947d99a72cd922cc323aa in term 1.
I20240503 02:00:26.774330 17618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4782de9826d6460e9e993e01327034e4" candidate_uuid: "5b8291b26a7f448b8636187d1caf86d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c843193b801b4f1a88f630ed3ed0f375" is_pre_election: true
I20240503 02:00:26.774436 17618 raft_consensus.cc:2362] T 4782de9826d6460e9e993e01327034e4 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 5b8291b26a7f448b8636187d1caf86d0 in term 1. Re-sending same reply.
I20240503 02:00:26.774917 18692 raft_consensus.cc:2781] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:26.775121 18690 raft_consensus.cc:2781] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:26.775400 18690 raft_consensus.cc:686] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 LEADER]: Becoming Leader. State: Replica: 5b8291b26a7f448b8636187d1caf86d0, State: Running, Role: LEADER
I20240503 02:00:26.775504 17090 raft_consensus.cc:2454] T ab536115ac7a4ba193f718d421f4000d P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d43ed6b8663947d99a72cd922cc323aa in term 1.
I20240503 02:00:26.775810 18690 consensus_queue.cc:237] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:26.776847 18692 raft_consensus.cc:686] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa [term 1 LEADER]: Becoming Leader. State: Replica: d43ed6b8663947d99a72cd922cc323aa, State: Running, Role: LEADER
I20240503 02:00:26.777045 17091 raft_consensus.cc:2454] T 4782de9826d6460e9e993e01327034e4 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b8291b26a7f448b8636187d1caf86d0 in term 1.
I20240503 02:00:26.777037 18692 consensus_queue.cc:237] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:26.778287 17751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4782de9826d6460e9e993e01327034e4" candidate_uuid: "5b8291b26a7f448b8636187d1caf86d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "002cc3411acb4fdfa0b67c6f57f687a6"
I20240503 02:00:26.778450 17751 raft_consensus.cc:3037] T 4782de9826d6460e9e993e01327034e4 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:26.779465 17751 raft_consensus.cc:2454] T 4782de9826d6460e9e993e01327034e4 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b8291b26a7f448b8636187d1caf86d0 in term 1.
I20240503 02:00:26.779414 16809 catalog_manager.cc:5484] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa reported cstate change: term changed from 0 to 1, leader changed from <none> to d43ed6b8663947d99a72cd922cc323aa (127.0.99.132). New cstate: current_term: 1 leader_uuid: "d43ed6b8663947d99a72cd922cc323aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:26.782845 16809 catalog_manager.cc:5484] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5b8291b26a7f448b8636187d1caf86d0 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "5b8291b26a7f448b8636187d1caf86d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } health_report { overall_health: UNKNOWN } } }
W20240503 02:00:26.907433 18964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:00:26.907716 18964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:00:26.910789 18964 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_6'
I20240503 02:00:27.002791 17750 raft_consensus.cc:1261] T 4782de9826d6460e9e993e01327034e4 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer 5b8291b26a7f448b8636187d1caf86d0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:27.003302 17618 raft_consensus.cc:1261] T 4782de9826d6460e9e993e01327034e4 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer 5b8291b26a7f448b8636187d1caf86d0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:27.003420 17751 raft_consensus.cc:1261] T ab536115ac7a4ba193f718d421f4000d P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:27.003463 17487 raft_consensus.cc:1261] T 4782de9826d6460e9e993e01327034e4 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Refusing update from remote peer 5b8291b26a7f448b8636187d1caf86d0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:27.003501 18690 consensus_queue.cc:1035] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:27.003959 18961 consensus_queue.cc:1035] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:27.004741 18692 consensus_queue.cc:1035] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:27.005257 18690 consensus_queue.cc:1035] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:27.006155 17356 raft_consensus.cc:1261] T 4782de9826d6460e9e993e01327034e4 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Refusing update from remote peer 5b8291b26a7f448b8636187d1caf86d0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:27.006366 16957 raft_consensus.cc:1261] T 4782de9826d6460e9e993e01327034e4 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 5b8291b26a7f448b8636187d1caf86d0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:27.006698 17091 raft_consensus.cc:1261] T 4782de9826d6460e9e993e01327034e4 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer 5b8291b26a7f448b8636187d1caf86d0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:27.007282 18690 consensus_queue.cc:1035] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:27.007565 18690 consensus_queue.cc:1035] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:27.007766 18690 consensus_queue.cc:1035] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:27.019594 16957 raft_consensus.cc:1261] T ab536115ac7a4ba193f718d421f4000d P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:27.020615 17620 raft_consensus.cc:1261] T ab536115ac7a4ba193f718d421f4000d P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:27.020769 17356 raft_consensus.cc:1261] T 3f67591d6a3744ab90a3e4d2375a59db P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:27.021075 17222 raft_consensus.cc:1261] T 3f67591d6a3744ab90a3e4d2375a59db P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:27.021215 18664 consensus_queue.cc:1035] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:27.021282 17090 raft_consensus.cc:1261] T ab536115ac7a4ba193f718d421f4000d P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:27.021785 18664 consensus_queue.cc:1035] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:27.021847 17222 raft_consensus.cc:1261] T ab536115ac7a4ba193f718d421f4000d P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:27.023253 17090 raft_consensus.cc:1261] T 3f67591d6a3744ab90a3e4d2375a59db P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:27.023974 17486 raft_consensus.cc:1261] T ab536115ac7a4ba193f718d421f4000d P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:27.024173 18873 consensus_queue.cc:1035] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:27.024295 17620 raft_consensus.cc:1261] T 3f67591d6a3744ab90a3e4d2375a59db P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:27.029100 16957 raft_consensus.cc:1261] T 3f67591d6a3744ab90a3e4d2375a59db P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:27.029560 18873 consensus_queue.cc:1035] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:27.031330 17749 raft_consensus.cc:1261] T 3f67591d6a3744ab90a3e4d2375a59db P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:27.032126 18809 consensus_queue.cc:1035] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:27.032469 18878 consensus_queue.cc:1035] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:27.020157 19004 consensus_queue.cc:1035] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:27.042538 18631 consensus_queue.cc:1035] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:27.050554 18692 consensus_queue.cc:1035] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:27.071156 19010 consensus_queue.cc:1035] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:27.071727 19008 consensus_queue.cc:1035] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 210.786 ms
time per row: 0.105393 ms
I20240503 02:00:27.164785 18870 mvcc.cc:205] Tried to move back new op lower bound from 7023417864356876288 to 7023417863281315840. Current Snapshot: MvccSnapshot[applied={T|T < 7023417864356876288 or (T in {7023417864356876288})}]
I20240503 02:00:27.172279 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task42rE7p/build/debug/bin/kudu
/tmp/dist-test-task42rE7p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-8/wal
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-8/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-8/logs
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-8/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.137:41887
--local_ip_for_outbound_sockets=127.0.99.137
--tserver_master_addrs=127.0.99.190:41395
--webserver_port=40699
--webserver_interface=127.0.99.137
--builtin_ntp_servers=127.0.99.148:44551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 02:00:27.312486 19026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:00:27.312767 19026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:00:27.312839 19026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:00:27.315929 19026 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:00:27.316008 19026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:00:27.316118 19026 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.137
I20240503 02:00:27.319795 19026 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:44551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-8/data
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-8/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.137:41887
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-8/data/info.pb
--webserver_interface=127.0.99.137
--webserver_port=40699
--tserver_master_addrs=127.0.99.190:41395
--never_fsync=true
--heap_profile_path=/tmp/kudu.19026
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.137
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-8/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 02:00:27.321017 19026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:00:27.322584 19026 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:00:27.333665 19031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:27.334475 19034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:00:27.335809 19026 server_base.cc:1034] running on GCE node
W20240503 02:00:27.336778 19032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:00:27.337280 19026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:00:27.337996 19026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:00:27.339162 19026 hybrid_clock.cc:648] HybridClock initialized: now 1714701627339141 us; error 43 us; skew 500 ppm
I20240503 02:00:27.341713 19026 webserver.cc:460] Webserver started at http://127.0.99.137:40699/ using document root <none> and password file <none>
I20240503 02:00:27.342393 19026 fs_manager.cc:359] Metadata directory not provided
I20240503 02:00:27.342499 19026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:00:27.347091 19026 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 02:00:27.349547 19040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:27.350896 19026 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:27.351025 19026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-8/data,/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-8/wal
uuid: "838ef968bb45484bb27f5af6f3989bae"
format_stamp: "Formatted at 2024-05-03 02:00:19 on dist-test-slave-dtz3"
I20240503 02:00:27.351612 19026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-8/wal
metadata directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-8/wal
1 data directories: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-8/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:00:27.374383 19026 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:00:27.375073 19026 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:00:27.375250 19026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:00:27.375895 19026 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:00:27.377089 19026 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:00:27.377151 19026 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:27.377208 19026 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:00:27.377233 19026 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:27.388028 19026 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.137:41887
I20240503 02:00:27.388094 19153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.137:41887 every 8 connection(s)
I20240503 02:00:27.389237 19026 server_base.cc:1166] Dumped server information to /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-8/data/info.pb
I20240503 02:00:27.396358 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task42rE7p/build/debug/bin/kudu as pid 19026
I20240503 02:00:27.396956 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task42rE7p/build/debug/bin/kudu
/tmp/dist-test-task42rE7p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-9/wal
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-9/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-9/logs
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-9/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.138:33181
--local_ip_for_outbound_sockets=127.0.99.138
--tserver_master_addrs=127.0.99.190:41395
--webserver_port=35613
--webserver_interface=127.0.99.138
--builtin_ntp_servers=127.0.99.148:44551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:00:27.400678 19154 heartbeater.cc:347] Connected to a master server at 127.0.99.190:41395
I20240503 02:00:27.400993 19154 heartbeater.cc:464] Registering TS with master...
I20240503 02:00:27.401865 19154 heartbeater.cc:511] Master 127.0.99.190:41395 requested a full tablet report, sending...
I20240503 02:00:27.403448 16810 ts_manager.cc:194] Re-registered known tserver with Master: 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887)
I20240503 02:00:27.404083 16810 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.137:53085
W20240503 02:00:27.531677 19157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:00:27.531981 19157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:00:27.532053 19157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:00:27.535522 19157 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:00:27.535590 19157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:00:27.535666 19157 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.138
I20240503 02:00:27.540357 19157 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:44551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-9/data
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-9/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.138:33181
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-9/data/info.pb
--webserver_interface=127.0.99.138
--webserver_port=35613
--tserver_master_addrs=127.0.99.190:41395
--never_fsync=true
--heap_profile_path=/tmp/kudu.19157
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.138
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-9/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 02:00:27.541625 19157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:00:27.543248 19157 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:00:27.555526 19162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:27.556695 19163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:27.557689 19165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:00:27.563323 19157 server_base.cc:1034] running on GCE node
I20240503 02:00:27.563779 19157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:00:27.564426 19157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:00:27.565587 19157 hybrid_clock.cc:648] HybridClock initialized: now 1714701627565555 us; error 56 us; skew 500 ppm
I20240503 02:00:27.567867 19157 webserver.cc:460] Webserver started at http://127.0.99.138:35613/ using document root <none> and password file <none>
I20240503 02:00:27.568601 19157 fs_manager.cc:359] Metadata directory not provided
I20240503 02:00:27.568709 19157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:00:27.572990 19157 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20240503 02:00:27.575109 19171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:27.576365 19157 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:27.576503 19157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-9/data,/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-9/wal
uuid: "9564311af29a4493831a8db658a085c4"
format_stamp: "Formatted at 2024-05-03 02:00:19 on dist-test-slave-dtz3"
I20240503 02:00:27.576979 19157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-9/wal
metadata directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-9/wal
1 data directories: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-9/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:00:27.595770 19157 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:00:27.596400 19157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:00:27.596557 19157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:00:27.597193 19157 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:00:27.598245 19157 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:00:27.598304 19157 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:27.598356 19157 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:00:27.598384 19157 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:27.608955 19157 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.138:33181
I20240503 02:00:27.609052 19284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.138:33181 every 8 connection(s)
I20240503 02:00:27.610194 19157 server_base.cc:1166] Dumped server information to /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-9/data/info.pb
I20240503 02:00:27.613337 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task42rE7p/build/debug/bin/kudu as pid 19157
I20240503 02:00:27.613817 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task42rE7p/build/debug/bin/kudu
/tmp/dist-test-task42rE7p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.136:40167
--local_ip_for_outbound_sockets=127.0.99.136
--tserver_master_addrs=127.0.99.190:41395
--webserver_port=45657
--webserver_interface=127.0.99.136
--builtin_ntp_servers=127.0.99.148:44551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:00:27.621508 19285 heartbeater.cc:347] Connected to a master server at 127.0.99.190:41395
I20240503 02:00:27.621771 19285 heartbeater.cc:464] Registering TS with master...
I20240503 02:00:27.622574 19285 heartbeater.cc:511] Master 127.0.99.190:41395 requested a full tablet report, sending...
I20240503 02:00:27.623925 16810 ts_manager.cc:194] Re-registered known tserver with Master: 9564311af29a4493831a8db658a085c4 (127.0.99.138:33181)
I20240503 02:00:27.624904 16810 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.138:59445
W20240503 02:00:27.753712 19288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:00:27.754034 19288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:00:27.754097 19288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:00:27.758076 19288 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:00:27.758157 19288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:00:27.758251 19288 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.136
I20240503 02:00:27.763058 19288 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:44551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.136:40167
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.0.99.136
--webserver_port=45657
--tserver_master_addrs=127.0.99.190:41395
--never_fsync=true
--heap_profile_path=/tmp/kudu.19288
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.136
--log_dir=/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 02:00:27.764290 19288 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:00:27.765962 19288 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:00:27.777128 19296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:27.777972 19293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:27.779261 19294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:00:27.780295 19288 server_base.cc:1034] running on GCE node
I20240503 02:00:27.780872 19288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:00:27.781664 19288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:00:27.782886 19288 hybrid_clock.cc:648] HybridClock initialized: now 1714701627782849 us; error 42 us; skew 500 ppm
I20240503 02:00:27.785354 19288 webserver.cc:460] Webserver started at http://127.0.99.136:45657/ using document root <none> and password file <none>
I20240503 02:00:27.786073 19288 fs_manager.cc:359] Metadata directory not provided
I20240503 02:00:27.786180 19288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:00:27.790004 19288 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 02:00:27.792369 19302 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:27.793864 19288 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 02:00:27.794020 19288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "375d0f2ec11d4e25bd9d093bb6a5572d"
format_stamp: "Formatted at 2024-05-03 02:00:18 on dist-test-slave-dtz3"
I20240503 02:00:27.794589 19288 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:00:27.823194 19288 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:00:27.823961 19288 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:00:27.824143 19288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:00:27.824815 19288 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:00:27.825883 19288 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:00:27.825937 19288 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:27.826004 19288 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:00:27.826033 19288 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:27.837590 19288 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.136:40167
I20240503 02:00:27.837692 19415 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.136:40167 every 8 connection(s)
I20240503 02:00:27.839028 19288 server_base.cc:1166] Dumped server information to /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 02:00:27.842010 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task42rE7p/build/debug/bin/kudu as pid 19288
I20240503 02:00:27.850360 19416 heartbeater.cc:347] Connected to a master server at 127.0.99.190:41395
I20240503 02:00:27.850646 19416 heartbeater.cc:464] Registering TS with master...
I20240503 02:00:27.851357 19416 heartbeater.cc:511] Master 127.0.99.190:41395 requested a full tablet report, sending...
I20240503 02:00:27.852747 16810 ts_manager.cc:194] Re-registered known tserver with Master: 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167)
I20240503 02:00:27.853641 16810 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.136:34707
I20240503 02:00:28.085682 19350 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:28.088439 17467 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:00:28.091250 17202 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:28.109794 16938 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:28.112650 19219 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:28.116525 17070 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:00:28.119906 19088 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:28.132512 17730 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:00:28.149899 17335 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:00:28.151067 17598 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:28.282094 17202 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:28.288228 19088 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:28.289664 19350 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:28.294981 16938 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:28.296172 17467 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:00:28.296973 19219 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:28.297542 17730 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:00:28.319058 17335 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:00:28.319729 17070 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:00:28.347332 17598 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:28.407002 19154 heartbeater.cc:503] Master 127.0.99.190:41395 was elected leader, sending a full tablet report...
I20240503 02:00:28.514878 19350 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:28.515120 17730 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:00:28.517263 16938 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:28.525067 17467 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:00:28.532528 19088 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:28.535105 17070 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:00:28.538918 17202 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:28.545634 17335 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:00:28.556243 17598 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:28.587419 19219 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:28.626892 17487 consensus_queue.cc:237] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:28.628448 19285 heartbeater.cc:503] Master 127.0.99.190:41395 was elected leader, sending a full tablet report...
I20240503 02:00:28.629282 17356 raft_consensus.cc:1261] T d52afe1281064d6480427554ca4bb2f6 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:28.629307 17618 raft_consensus.cc:1261] T d52afe1281064d6480427554ca4bb2f6 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:28.629725 19567 consensus_queue.cc:1035] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:28.629909 17223 raft_consensus.cc:1261] T d52afe1281064d6480427554ca4bb2f6 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:28.630303 17750 raft_consensus.cc:1261] T d52afe1281064d6480427554ca4bb2f6 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:28.630676 19567 consensus_queue.cc:1035] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:28.631012 19567 consensus_queue.cc:1035] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:28.631353 19567 consensus_queue.cc:1035] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:28.631494 16959 raft_consensus.cc:1261] T d52afe1281064d6480427554ca4bb2f6 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:28.631927 19567 consensus_queue.cc:1035] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:28.633026 17090 raft_consensus.cc:1261] T d52afe1281064d6480427554ca4bb2f6 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:28.633775 18873 consensus_queue.cc:1035] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:28.634456 18873 raft_consensus.cc:2932] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } }
I20240503 02:00:28.635602 17223 raft_consensus.cc:2932] T d52afe1281064d6480427554ca4bb2f6 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } }
I20240503 02:00:28.637858 17356 raft_consensus.cc:2932] T d52afe1281064d6480427554ca4bb2f6 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } }
I20240503 02:00:28.638772 17750 raft_consensus.cc:2932] T d52afe1281064d6480427554ca4bb2f6 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } }
I20240503 02:00:28.640113 17090 raft_consensus.cc:2932] T d52afe1281064d6480427554ca4bb2f6 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } }
I20240503 02:00:28.640992 16817 catalog_manager.cc:5484] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "16d437fe629643948c95d628926d5ffa" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } health_report { overall_health: HEALTHY } } }
I20240503 02:00:28.642810 16959 raft_consensus.cc:2932] T d52afe1281064d6480427554ca4bb2f6 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } }
I20240503 02:00:28.643352 17618 raft_consensus.cc:2932] T d52afe1281064d6480427554ca4bb2f6 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } }
I20240503 02:00:28.650524 17487 consensus_queue.cc:237] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: true } }
I20240503 02:00:28.652436 17222 raft_consensus.cc:1261] T d52afe1281064d6480427554ca4bb2f6 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:28.652926 16959 raft_consensus.cc:1261] T d52afe1281064d6480427554ca4bb2f6 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:28.653249 17750 raft_consensus.cc:1261] T d52afe1281064d6480427554ca4bb2f6 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:28.653244 17619 raft_consensus.cc:1261] T d52afe1281064d6480427554ca4bb2f6 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:28.653687 17090 raft_consensus.cc:1261] T d52afe1281064d6480427554ca4bb2f6 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:28.653698 18873 consensus_queue.cc:1035] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:28.653903 17356 raft_consensus.cc:1261] T d52afe1281064d6480427554ca4bb2f6 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:28.654351 19567 consensus_queue.cc:1035] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:28.654538 19569 consensus_queue.cc:1035] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:28.654686 18878 consensus_queue.cc:1035] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:28.656303 19569 consensus_queue.cc:1035] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:28.656594 18878 consensus_queue.cc:1035] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:28.657691 19569 raft_consensus.cc:2932] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9564311af29a4493831a8db658a085c4 (127.0.99.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: true } } }
I20240503 02:00:28.658428 17356 raft_consensus.cc:2932] T d52afe1281064d6480427554ca4bb2f6 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9564311af29a4493831a8db658a085c4 (127.0.99.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: true } } }
I20240503 02:00:28.658859 17619 raft_consensus.cc:2932] T d52afe1281064d6480427554ca4bb2f6 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9564311af29a4493831a8db658a085c4 (127.0.99.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: true } } }
I20240503 02:00:28.660513 16802 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet d52afe1281064d6480427554ca4bb2f6 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 02:00:28.660429 16959 raft_consensus.cc:2932] T d52afe1281064d6480427554ca4bb2f6 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9564311af29a4493831a8db658a085c4 (127.0.99.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: true } } }
I20240503 02:00:28.661279 17090 raft_consensus.cc:2932] T d52afe1281064d6480427554ca4bb2f6 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9564311af29a4493831a8db658a085c4 (127.0.99.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: true } } }
I20240503 02:00:28.661737 17750 raft_consensus.cc:2932] T d52afe1281064d6480427554ca4bb2f6 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9564311af29a4493831a8db658a085c4 (127.0.99.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: true } } }
I20240503 02:00:28.663476 16813 catalog_manager.cc:5484] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa reported cstate change: config changed from index 4 to 5, NON_VOTER 9564311af29a4493831a8db658a085c4 (127.0.99.138) added. New cstate: current_term: 1 leader_uuid: "16d437fe629643948c95d628926d5ffa" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:28.665367 17222 raft_consensus.cc:2932] T d52afe1281064d6480427554ca4bb2f6 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9564311af29a4493831a8db658a085c4 (127.0.99.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: true } } }
I20240503 02:00:28.668807 17353 consensus_queue.cc:237] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
W20240503 02:00:28.669929 17424 consensus_peers.cc:485] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa -> Peer 9564311af29a4493831a8db658a085c4 (127.0.99.138:33181): Couldn't send request to peer 9564311af29a4493831a8db658a085c4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d52afe1281064d6480427554ca4bb2f6. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:28.670614 17619 raft_consensus.cc:1261] T ab536115ac7a4ba193f718d421f4000d P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:28.670730 17487 raft_consensus.cc:1261] T ab536115ac7a4ba193f718d421f4000d P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:28.670934 17750 raft_consensus.cc:1261] T ab536115ac7a4ba193f718d421f4000d P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:28.671030 17090 raft_consensus.cc:1261] T ab536115ac7a4ba193f718d421f4000d P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:28.671092 19518 consensus_queue.cc:1035] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:28.671355 19590 consensus_queue.cc:1035] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:28.671483 17222 raft_consensus.cc:1261] T ab536115ac7a4ba193f718d421f4000d P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:28.671840 19518 consensus_queue.cc:1035] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:28.672744 16959 raft_consensus.cc:1261] T ab536115ac7a4ba193f718d421f4000d P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:28.673197 19590 consensus_queue.cc:1035] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:28.674926 19589 raft_consensus.cc:2932] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } }
I20240503 02:00:28.675333 19534 consensus_queue.cc:1035] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:28.675482 16959 raft_consensus.cc:2932] T ab536115ac7a4ba193f718d421f4000d P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } }
I20240503 02:00:28.675680 19534 consensus_queue.cc:1035] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:28.676826 17222 raft_consensus.cc:2932] T ab536115ac7a4ba193f718d421f4000d P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } }
I20240503 02:00:28.677783 17619 raft_consensus.cc:2932] T ab536115ac7a4ba193f718d421f4000d P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } }
I20240503 02:00:28.677733 16812 catalog_manager.cc:5484] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "d43ed6b8663947d99a72cd922cc323aa" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:28.678752 17750 raft_consensus.cc:2932] T ab536115ac7a4ba193f718d421f4000d P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } }
I20240503 02:00:28.681887 17487 raft_consensus.cc:2932] T ab536115ac7a4ba193f718d421f4000d P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } }
I20240503 02:00:28.683979 17353 consensus_queue.cc:237] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: true } }
I20240503 02:00:28.687122 17090 raft_consensus.cc:2932] T ab536115ac7a4ba193f718d421f4000d P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } }
I20240503 02:00:28.688164 16959 raft_consensus.cc:1261] T ab536115ac7a4ba193f718d421f4000d P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:28.688545 19589 consensus_queue.cc:1035] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:28.688942 17092 raft_consensus.cc:1261] T ab536115ac7a4ba193f718d421f4000d P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:28.690018 17487 raft_consensus.cc:1261] T ab536115ac7a4ba193f718d421f4000d P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:28.690438 19589 consensus_queue.cc:1035] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:28.690483 17222 raft_consensus.cc:1261] T ab536115ac7a4ba193f718d421f4000d P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:28.690718 19589 consensus_queue.cc:1035] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:28.693050 19589 consensus_queue.cc:1035] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:28.694517 19012 raft_consensus.cc:2932] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9564311af29a4493831a8db658a085c4 (127.0.99.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: true } } }
I20240503 02:00:28.695104 17223 raft_consensus.cc:2932] T ab536115ac7a4ba193f718d421f4000d P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9564311af29a4493831a8db658a085c4 (127.0.99.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: true } } }
I20240503 02:00:28.695327 16959 raft_consensus.cc:2932] T ab536115ac7a4ba193f718d421f4000d P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9564311af29a4493831a8db658a085c4 (127.0.99.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: true } } }
I20240503 02:00:28.698168 17618 raft_consensus.cc:1261] T ab536115ac7a4ba193f718d421f4000d P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:28.698448 16812 catalog_manager.cc:5484] T ab536115ac7a4ba193f718d421f4000d P 6398c8993c3c45dcaab25b42142d5671 reported cstate change: config changed from index 4 to 5, NON_VOTER 9564311af29a4493831a8db658a085c4 (127.0.99.138) added. New cstate: current_term: 1 leader_uuid: "d43ed6b8663947d99a72cd922cc323aa" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: true } } }
I20240503 02:00:28.699323 19598 consensus_queue.cc:1035] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:28.699548 17487 raft_consensus.cc:2932] T ab536115ac7a4ba193f718d421f4000d P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9564311af29a4493831a8db658a085c4 (127.0.99.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: true } } }
I20240503 02:00:28.702091 16803 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet ab536115ac7a4ba193f718d421f4000d with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 02:00:28.702178 17486 consensus_queue.cc:237] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } }
I20240503 02:00:28.702558 17092 raft_consensus.cc:2932] T ab536115ac7a4ba193f718d421f4000d P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9564311af29a4493831a8db658a085c4 (127.0.99.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: true } } }
I20240503 02:00:28.702620 17618 raft_consensus.cc:2932] T ab536115ac7a4ba193f718d421f4000d P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9564311af29a4493831a8db658a085c4 (127.0.99.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: true } } }
I20240503 02:00:28.705900 17223 raft_consensus.cc:1261] T fbec07350ad04c49afd42e5d88dda6ad P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:28.706432 17749 raft_consensus.cc:1261] T fbec07350ad04c49afd42e5d88dda6ad P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:28.706540 17618 raft_consensus.cc:1261] T fbec07350ad04c49afd42e5d88dda6ad P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:28.706974 17751 raft_consensus.cc:1261] T ab536115ac7a4ba193f718d421f4000d P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:28.707106 19586 consensus_queue.cc:1035] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:28.707931 17353 raft_consensus.cc:1261] T fbec07350ad04c49afd42e5d88dda6ad P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:28.708144 19568 consensus_queue.cc:1035] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:28.708482 19568 consensus_queue.cc:1035] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:28.708600 19567 consensus_queue.cc:1035] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:28.708859 17092 raft_consensus.cc:1261] T fbec07350ad04c49afd42e5d88dda6ad P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:28.709295 19568 consensus_queue.cc:1035] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:28.709792 16959 raft_consensus.cc:1261] T fbec07350ad04c49afd42e5d88dda6ad P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:28.710141 19586 consensus_queue.cc:1035] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:28.711349 19597 consensus_queue.cc:1035] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:28.712303 17751 raft_consensus.cc:2932] T ab536115ac7a4ba193f718d421f4000d P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9564311af29a4493831a8db658a085c4 (127.0.99.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: true } } }
W20240503 02:00:28.712849 17292 consensus_peers.cc:485] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa -> Peer 9564311af29a4493831a8db658a085c4 (127.0.99.138:33181): Couldn't send request to peer 9564311af29a4493831a8db658a085c4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ab536115ac7a4ba193f718d421f4000d. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:28.712956 18878 raft_consensus.cc:2932] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } }
I20240503 02:00:28.713310 16959 raft_consensus.cc:2932] T fbec07350ad04c49afd42e5d88dda6ad P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } }
I20240503 02:00:28.713621 17353 raft_consensus.cc:2932] T fbec07350ad04c49afd42e5d88dda6ad P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } }
I20240503 02:00:28.715626 17751 raft_consensus.cc:2932] T fbec07350ad04c49afd42e5d88dda6ad P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } }
I20240503 02:00:28.715759 16812 catalog_manager.cc:5484] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "16d437fe629643948c95d628926d5ffa" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } health_report { overall_health: HEALTHY } } }
I20240503 02:00:28.716701 17092 raft_consensus.cc:2932] T fbec07350ad04c49afd42e5d88dda6ad P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } }
I20240503 02:00:28.718750 17223 raft_consensus.cc:2932] T fbec07350ad04c49afd42e5d88dda6ad P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } }
I20240503 02:00:28.721012 17619 raft_consensus.cc:2932] T fbec07350ad04c49afd42e5d88dda6ad P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } }
I20240503 02:00:28.724283 17486 consensus_queue.cc:237] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: true } }
I20240503 02:00:28.726442 17619 raft_consensus.cc:1261] T fbec07350ad04c49afd42e5d88dda6ad P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:28.726455 17092 raft_consensus.cc:1261] T fbec07350ad04c49afd42e5d88dda6ad P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:28.726586 17751 raft_consensus.cc:1261] T fbec07350ad04c49afd42e5d88dda6ad P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:28.726843 19567 consensus_queue.cc:1035] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:28.726909 17223 raft_consensus.cc:1261] T fbec07350ad04c49afd42e5d88dda6ad P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:28.727046 18878 consensus_queue.cc:1035] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:28.727182 19570 consensus_queue.cc:1035] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:28.727366 17353 raft_consensus.cc:1261] T fbec07350ad04c49afd42e5d88dda6ad P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:28.728080 16959 raft_consensus.cc:1261] T fbec07350ad04c49afd42e5d88dda6ad P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:28.728888 19569 consensus_queue.cc:1035] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:28.729035 19570 consensus_queue.cc:1035] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 02:00:28.729275 17424 consensus_peers.cc:485] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa -> Peer 9564311af29a4493831a8db658a085c4 (127.0.99.138:33181): Couldn't send request to peer 9564311af29a4493831a8db658a085c4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fbec07350ad04c49afd42e5d88dda6ad. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:28.729358 19586 consensus_queue.cc:1035] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:28.730607 19586 raft_consensus.cc:2932] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9564311af29a4493831a8db658a085c4 (127.0.99.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: true } } }
I20240503 02:00:28.730809 17487 consensus_queue.cc:237] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:28.730901 17092 raft_consensus.cc:2932] T fbec07350ad04c49afd42e5d88dda6ad P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9564311af29a4493831a8db658a085c4 (127.0.99.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: true } } }
I20240503 02:00:28.731173 16959 raft_consensus.cc:2932] T fbec07350ad04c49afd42e5d88dda6ad P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9564311af29a4493831a8db658a085c4 (127.0.99.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: true } } }
I20240503 02:00:28.733446 16802 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet fbec07350ad04c49afd42e5d88dda6ad with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 02:00:28.734232 16959 raft_consensus.cc:1261] T acfd773c1f384c219d776878407d1144 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:28.734171 16811 catalog_manager.cc:5484] T fbec07350ad04c49afd42e5d88dda6ad P 6398c8993c3c45dcaab25b42142d5671 reported cstate change: config changed from index 4 to 5, NON_VOTER 9564311af29a4493831a8db658a085c4 (127.0.99.138) added. New cstate: current_term: 1 leader_uuid: "16d437fe629643948c95d628926d5ffa" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: true } } }
I20240503 02:00:28.734376 17353 raft_consensus.cc:1261] T acfd773c1f384c219d776878407d1144 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:28.734640 19570 consensus_queue.cc:1035] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:28.735365 19568 consensus_queue.cc:1035] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:28.735291 17750 raft_consensus.cc:2932] T fbec07350ad04c49afd42e5d88dda6ad P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9564311af29a4493831a8db658a085c4 (127.0.99.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: true } } }
I20240503 02:00:28.735536 17618 raft_consensus.cc:2932] T fbec07350ad04c49afd42e5d88dda6ad P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9564311af29a4493831a8db658a085c4 (127.0.99.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: true } } }
I20240503 02:00:28.735574 17353 raft_consensus.cc:2932] T fbec07350ad04c49afd42e5d88dda6ad P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9564311af29a4493831a8db658a085c4 (127.0.99.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: true } } }
I20240503 02:00:28.736375 17619 raft_consensus.cc:1261] T acfd773c1f384c219d776878407d1144 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:28.736781 19568 consensus_queue.cc:1035] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:28.738179 17751 raft_consensus.cc:1261] T acfd773c1f384c219d776878407d1144 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:28.738654 19619 consensus_queue.cc:1035] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:28.739920 17222 raft_consensus.cc:2932] T fbec07350ad04c49afd42e5d88dda6ad P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9564311af29a4493831a8db658a085c4 (127.0.99.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: true } } }
I20240503 02:00:28.741006 17092 raft_consensus.cc:1261] T acfd773c1f384c219d776878407d1144 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:28.741197 17223 raft_consensus.cc:1261] T acfd773c1f384c219d776878407d1144 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:28.741631 19619 consensus_queue.cc:1035] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:28.741938 19619 consensus_queue.cc:1035] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:28.742931 19570 raft_consensus.cc:2932] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } }
I20240503 02:00:28.744983 17751 raft_consensus.cc:2932] T acfd773c1f384c219d776878407d1144 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } }
I20240503 02:00:28.746428 17618 raft_consensus.cc:2932] T acfd773c1f384c219d776878407d1144 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } }
I20240503 02:00:28.747293 17223 raft_consensus.cc:2932] T acfd773c1f384c219d776878407d1144 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } }
I20240503 02:00:28.748368 17092 raft_consensus.cc:2932] T acfd773c1f384c219d776878407d1144 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } }
I20240503 02:00:28.749768 17356 raft_consensus.cc:2932] T acfd773c1f384c219d776878407d1144 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } }
I20240503 02:00:28.750706 16809 catalog_manager.cc:5484] T acfd773c1f384c219d776878407d1144 P c843193b801b4f1a88f630ed3ed0f375 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "16d437fe629643948c95d628926d5ffa" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } }
I20240503 02:00:28.753612 16959 raft_consensus.cc:2932] T acfd773c1f384c219d776878407d1144 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } }
I20240503 02:00:28.760133 17487 consensus_queue.cc:237] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } }
I20240503 02:00:28.762046 16959 raft_consensus.cc:1261] T acfd773c1f384c219d776878407d1144 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:28.762125 17092 raft_consensus.cc:1261] T acfd773c1f384c219d776878407d1144 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:28.762490 17223 raft_consensus.cc:1261] T acfd773c1f384c219d776878407d1144 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:28.762495 18873 consensus_queue.cc:1035] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:28.762868 18878 consensus_queue.cc:1035] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:28.763597 18878 consensus_queue.cc:1035] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:28.765141 17618 raft_consensus.cc:1261] T acfd773c1f384c219d776878407d1144 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:28.765398 17750 raft_consensus.cc:1261] T acfd773c1f384c219d776878407d1144 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:28.765779 17356 raft_consensus.cc:1261] T acfd773c1f384c219d776878407d1144 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:28.765995 17751 consensus_queue.cc:237] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:28.766161 19567 consensus_queue.cc:1035] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:28.766404 18873 consensus_queue.cc:1035] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:28.766549 19567 consensus_queue.cc:1035] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:28.767042 19568 raft_consensus.cc:2932] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:28.767763 17092 raft_consensus.cc:1261] T 5d2104698d5f452c8350cce10afd3898 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:28.768151 16959 raft_consensus.cc:1261] T 5d2104698d5f452c8350cce10afd3898 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:28.768201 18884 consensus_queue.cc:1035] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:28.768520 19421 consensus_queue.cc:1035] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:28.769441 17222 raft_consensus.cc:1261] T 5d2104698d5f452c8350cce10afd3898 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:28.769644 17487 raft_consensus.cc:1261] T 5d2104698d5f452c8350cce10afd3898 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:28.769948 17356 raft_consensus.cc:1261] T 5d2104698d5f452c8350cce10afd3898 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:28.770128 16802 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet acfd773c1f384c219d776878407d1144 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 02:00:28.770421 16958 raft_consensus.cc:2932] T acfd773c1f384c219d776878407d1144 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:28.770884 17618 raft_consensus.cc:1261] T 5d2104698d5f452c8350cce10afd3898 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 02:00:28.771157 17422 consensus_peers.cc:485] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: acfd773c1f384c219d776878407d1144. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:28.771625 17222 raft_consensus.cc:2932] T acfd773c1f384c219d776878407d1144 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:28.773279 17751 raft_consensus.cc:2932] T acfd773c1f384c219d776878407d1144 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:28.773743 17092 raft_consensus.cc:2932] T acfd773c1f384c219d776878407d1144 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:28.774282 16817 catalog_manager.cc:5484] T acfd773c1f384c219d776878407d1144 P 6398c8993c3c45dcaab25b42142d5671 reported cstate change: config changed from index 4 to 5, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "16d437fe629643948c95d628926d5ffa" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:28.776197 17356 raft_consensus.cc:2932] T acfd773c1f384c219d776878407d1144 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:28.769778 19634 consensus_queue.cc:1035] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:28.777917 19421 consensus_queue.cc:1035] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:28.778405 19636 consensus_queue.cc:1035] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:28.779457 18884 consensus_queue.cc:1035] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:28.779747 17619 raft_consensus.cc:2932] T acfd773c1f384c219d776878407d1144 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:28.786067 17222 raft_consensus.cc:2932] T 5d2104698d5f452c8350cce10afd3898 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } }
I20240503 02:00:28.787320 19633 raft_consensus.cc:2932] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } }
I20240503 02:00:28.787781 17618 raft_consensus.cc:2932] T 5d2104698d5f452c8350cce10afd3898 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } }
I20240503 02:00:28.787914 17486 raft_consensus.cc:2932] T 5d2104698d5f452c8350cce10afd3898 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } }
I20240503 02:00:28.789717 17356 raft_consensus.cc:2932] T 5d2104698d5f452c8350cce10afd3898 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } }
I20240503 02:00:28.789717 17090 raft_consensus.cc:2932] T 5d2104698d5f452c8350cce10afd3898 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } }
I20240503 02:00:28.789727 16811 catalog_manager.cc:5484] T 5d2104698d5f452c8350cce10afd3898 P 5b8291b26a7f448b8636187d1caf86d0 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } }
I20240503 02:00:28.793663 16957 raft_consensus.cc:2932] T 5d2104698d5f452c8350cce10afd3898 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } }
I20240503 02:00:28.798398 17751 consensus_queue.cc:237] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: true } }
I20240503 02:00:28.800791 17486 raft_consensus.cc:1261] T 5d2104698d5f452c8350cce10afd3898 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:28.800859 17356 raft_consensus.cc:1261] T 5d2104698d5f452c8350cce10afd3898 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:28.801087 16957 raft_consensus.cc:1261] T 5d2104698d5f452c8350cce10afd3898 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:28.801263 17222 raft_consensus.cc:1261] T 5d2104698d5f452c8350cce10afd3898 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:28.801508 19634 consensus_queue.cc:1035] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:28.801613 19421 consensus_queue.cc:1035] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:28.801846 19634 consensus_queue.cc:1035] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:28.802031 17618 raft_consensus.cc:1261] T 5d2104698d5f452c8350cce10afd3898 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:28.802903 19634 consensus_queue.cc:1035] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:28.803030 17090 raft_consensus.cc:1261] T 5d2104698d5f452c8350cce10afd3898 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:28.804924 19636 consensus_queue.cc:1035] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:28.805431 19636 raft_consensus.cc:2932] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9564311af29a4493831a8db658a085c4 (127.0.99.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: true } } }
I20240503 02:00:28.806023 19648 consensus_queue.cc:1035] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:28.807119 17090 raft_consensus.cc:2932] T 5d2104698d5f452c8350cce10afd3898 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9564311af29a4493831a8db658a085c4 (127.0.99.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: true } } }
I20240503 02:00:28.807550 16802 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5d2104698d5f452c8350cce10afd3898 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 02:00:28.807606 17356 raft_consensus.cc:2932] T 5d2104698d5f452c8350cce10afd3898 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9564311af29a4493831a8db658a085c4 (127.0.99.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: true } } }
I20240503 02:00:28.808034 16957 raft_consensus.cc:2932] T 5d2104698d5f452c8350cce10afd3898 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9564311af29a4493831a8db658a085c4 (127.0.99.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: true } } }
W20240503 02:00:28.808210 17688 consensus_peers.cc:485] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 9564311af29a4493831a8db658a085c4 (127.0.99.138:33181): Couldn't send request to peer 9564311af29a4493831a8db658a085c4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5d2104698d5f452c8350cce10afd3898. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:28.809137 17222 raft_consensus.cc:2932] T 5d2104698d5f452c8350cce10afd3898 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9564311af29a4493831a8db658a085c4 (127.0.99.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: true } } }
I20240503 02:00:28.809434 17618 raft_consensus.cc:2932] T 5d2104698d5f452c8350cce10afd3898 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9564311af29a4493831a8db658a085c4 (127.0.99.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: true } } }
I20240503 02:00:28.810869 16809 catalog_manager.cc:5484] T 5d2104698d5f452c8350cce10afd3898 P 89dd09f8aaee4b249ae3ce924a422a67 reported cstate change: config changed from index 4 to 5, NON_VOTER 9564311af29a4493831a8db658a085c4 (127.0.99.138) added. New cstate: current_term: 1 leader_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: true } } }
I20240503 02:00:28.812269 17486 raft_consensus.cc:2932] T 5d2104698d5f452c8350cce10afd3898 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9564311af29a4493831a8db658a085c4 (127.0.99.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: true } } }
I20240503 02:00:28.816231 17356 consensus_queue.cc:237] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:28.817885 17090 raft_consensus.cc:1261] T 63962b169caf4f14888871e246c00075 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:28.818149 16957 raft_consensus.cc:1261] T 63962b169caf4f14888871e246c00075 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:28.818240 17486 raft_consensus.cc:1261] T 63962b169caf4f14888871e246c00075 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:28.818298 17222 raft_consensus.cc:1261] T 63962b169caf4f14888871e246c00075 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:28.818389 19599 consensus_queue.cc:1035] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:28.818578 19597 consensus_queue.cc:1035] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:28.818714 19589 consensus_queue.cc:1035] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:28.818974 19598 consensus_queue.cc:1035] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:28.819476 17751 raft_consensus.cc:1261] T 63962b169caf4f14888871e246c00075 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:28.820099 19597 consensus_queue.cc:1035] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:28.820266 17618 raft_consensus.cc:1261] T 63962b169caf4f14888871e246c00075 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:28.820647 19597 consensus_queue.cc:1035] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:28.822456 19598 raft_consensus.cc:2932] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } }
I20240503 02:00:28.823385 17618 raft_consensus.cc:2932] T 63962b169caf4f14888871e246c00075 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } }
I20240503 02:00:28.823582 17222 raft_consensus.cc:2932] T 63962b169caf4f14888871e246c00075 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } }
I20240503 02:00:28.823808 17751 raft_consensus.cc:2932] T 63962b169caf4f14888871e246c00075 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } }
I20240503 02:00:28.824827 17090 raft_consensus.cc:2932] T 63962b169caf4f14888871e246c00075 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } }
I20240503 02:00:28.825732 17486 raft_consensus.cc:2932] T 63962b169caf4f14888871e246c00075 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } }
I20240503 02:00:28.825959 16817 catalog_manager.cc:5484] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "d43ed6b8663947d99a72cd922cc323aa" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } health_report { overall_health: HEALTHY } } }
I20240503 02:00:28.827226 16957 raft_consensus.cc:2932] T 63962b169caf4f14888871e246c00075 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } }
I20240503 02:00:28.831097 17356 consensus_queue.cc:237] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: true } }
I20240503 02:00:28.833150 17751 raft_consensus.cc:1261] T 63962b169caf4f14888871e246c00075 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:28.833578 19518 consensus_queue.cc:1035] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:28.833778 17222 raft_consensus.cc:1261] T 63962b169caf4f14888871e246c00075 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240503 02:00:28.833976 17292 consensus_peers.cc:485] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa -> Peer 9564311af29a4493831a8db658a085c4 (127.0.99.138:33181): Couldn't send request to peer 9564311af29a4493831a8db658a085c4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 63962b169caf4f14888871e246c00075. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:28.833998 17090 raft_consensus.cc:1261] T 63962b169caf4f14888871e246c00075 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:28.834323 17618 raft_consensus.cc:1261] T 63962b169caf4f14888871e246c00075 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:28.834328 19599 consensus_queue.cc:1035] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:28.834641 19599 consensus_queue.cc:1035] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:28.834734 16957 raft_consensus.cc:1261] T 63962b169caf4f14888871e246c00075 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:28.834775 19518 consensus_queue.cc:1035] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:28.835060 19518 consensus_queue.cc:1035] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:28.835242 17486 raft_consensus.cc:1261] T 63962b169caf4f14888871e246c00075 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:28.835556 19597 consensus_queue.cc:1035] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:28.836972 19518 raft_consensus.cc:2932] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9564311af29a4493831a8db658a085c4 (127.0.99.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: true } } }
I20240503 02:00:28.837401 16958 raft_consensus.cc:2932] T 63962b169caf4f14888871e246c00075 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9564311af29a4493831a8db658a085c4 (127.0.99.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: true } } }
I20240503 02:00:28.839848 16803 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 63962b169caf4f14888871e246c00075 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 02:00:28.840481 17751 raft_consensus.cc:2932] T 63962b169caf4f14888871e246c00075 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9564311af29a4493831a8db658a085c4 (127.0.99.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: true } } }
I20240503 02:00:28.840842 16810 catalog_manager.cc:5484] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa reported cstate change: config changed from index 4 to 5, NON_VOTER 9564311af29a4493831a8db658a085c4 (127.0.99.138) added. New cstate: current_term: 1 leader_uuid: "d43ed6b8663947d99a72cd922cc323aa" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:28.841096 17222 raft_consensus.cc:2932] T 63962b169caf4f14888871e246c00075 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9564311af29a4493831a8db658a085c4 (127.0.99.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: true } } }
I20240503 02:00:28.842124 17618 raft_consensus.cc:2932] T 63962b169caf4f14888871e246c00075 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9564311af29a4493831a8db658a085c4 (127.0.99.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: true } } }
I20240503 02:00:28.844244 17486 raft_consensus.cc:2932] T 63962b169caf4f14888871e246c00075 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9564311af29a4493831a8db658a085c4 (127.0.99.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: true } } }
I20240503 02:00:28.850410 17751 consensus_queue.cc:237] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:28.850831 17092 raft_consensus.cc:2932] T 63962b169caf4f14888871e246c00075 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9564311af29a4493831a8db658a085c4 (127.0.99.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: true } } }
I20240503 02:00:28.852133 17222 raft_consensus.cc:1261] T fdf0a4b9e17642849a4094d2f0b9a255 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:28.852578 19636 consensus_queue.cc:1035] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:28.852933 16957 raft_consensus.cc:1261] T fdf0a4b9e17642849a4094d2f0b9a255 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:28.853387 17092 raft_consensus.cc:1261] T fdf0a4b9e17642849a4094d2f0b9a255 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:28.853533 19648 consensus_queue.cc:1035] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:28.853952 19648 consensus_queue.cc:1035] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:28.854179 17356 raft_consensus.cc:1261] T fdf0a4b9e17642849a4094d2f0b9a255 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:28.854740 19421 consensus_queue.cc:1035] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:28.856176 19416 heartbeater.cc:503] Master 127.0.99.190:41395 was elected leader, sending a full tablet report...
I20240503 02:00:28.856590 17486 raft_consensus.cc:1261] T fdf0a4b9e17642849a4094d2f0b9a255 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:28.856690 19648 raft_consensus.cc:2932] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } }
I20240503 02:00:28.857069 17092 raft_consensus.cc:2932] T fdf0a4b9e17642849a4094d2f0b9a255 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } }
I20240503 02:00:28.858762 17618 raft_consensus.cc:1261] T fdf0a4b9e17642849a4094d2f0b9a255 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:28.858855 16957 raft_consensus.cc:2932] T fdf0a4b9e17642849a4094d2f0b9a255 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } }
I20240503 02:00:28.860714 16810 catalog_manager.cc:5484] T fdf0a4b9e17642849a4094d2f0b9a255 P 89dd09f8aaee4b249ae3ce924a422a67 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } }
I20240503 02:00:28.860770 17222 raft_consensus.cc:2932] T fdf0a4b9e17642849a4094d2f0b9a255 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } }
I20240503 02:00:28.863567 17356 raft_consensus.cc:2932] T fdf0a4b9e17642849a4094d2f0b9a255 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } }
I20240503 02:00:28.864163 19636 consensus_queue.cc:1035] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:28.864459 19636 consensus_queue.cc:1035] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:28.864876 17751 consensus_queue.cc:237] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } }
I20240503 02:00:28.865984 17618 raft_consensus.cc:2932] T fdf0a4b9e17642849a4094d2f0b9a255 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } }
I20240503 02:00:28.867447 16957 raft_consensus.cc:1261] T fdf0a4b9e17642849a4094d2f0b9a255 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:28.868196 17487 raft_consensus.cc:1261] T fdf0a4b9e17642849a4094d2f0b9a255 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:28.868441 17222 raft_consensus.cc:1261] T fdf0a4b9e17642849a4094d2f0b9a255 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:28.868661 19633 consensus_queue.cc:1035] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:28.869165 19636 consensus_queue.cc:1035] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:28.869309 17092 raft_consensus.cc:1261] T fdf0a4b9e17642849a4094d2f0b9a255 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:28.869657 19636 consensus_queue.cc:1035] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:28.870456 19633 consensus_queue.cc:1035] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 02:00:28.872671 17686 consensus_peers.cc:485] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fdf0a4b9e17642849a4094d2f0b9a255. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:28.872737 17487 raft_consensus.cc:858] T fdf0a4b9e17642849a4094d2f0b9a255 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }.
Pending config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }:
New config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } }
I20240503 02:00:28.872560 19633 raft_consensus.cc:2932] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
W20240503 02:00:28.872834 17487 raft_consensus.cc:1565] T fdf0a4b9e17642849a4094d2f0b9a255 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Could not prepare op 'term: 1 index: 5' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }.
Pending config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }:
New config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } }
I20240503 02:00:28.873001 17092 raft_consensus.cc:2932] T fdf0a4b9e17642849a4094d2f0b9a255 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:28.873257 17487 raft_consensus.cc:2932] T fdf0a4b9e17642849a4094d2f0b9a255 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } }
I20240503 02:00:28.874574 16802 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet fdf0a4b9e17642849a4094d2f0b9a255 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 02:00:28.875047 16957 raft_consensus.cc:2932] T fdf0a4b9e17642849a4094d2f0b9a255 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:28.875547 17222 raft_consensus.cc:2932] T fdf0a4b9e17642849a4094d2f0b9a255 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:28.875867 16809 catalog_manager.cc:5484] T fdf0a4b9e17642849a4094d2f0b9a255 P 89dd09f8aaee4b249ae3ce924a422a67 reported cstate change: config changed from index 4 to 5, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:28.879155 17487 raft_consensus.cc:2932] T fdf0a4b9e17642849a4094d2f0b9a255 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:28.881024 17356 raft_consensus.cc:1261] T fdf0a4b9e17642849a4094d2f0b9a255 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:28.881469 19636 consensus_queue.cc:1035] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:28.882309 17619 raft_consensus.cc:1261] T fdf0a4b9e17642849a4094d2f0b9a255 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:28.882812 19636 consensus_queue.cc:1035] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:28.883822 17619 raft_consensus.cc:2932] T fdf0a4b9e17642849a4094d2f0b9a255 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:28.883821 17356 raft_consensus.cc:2932] T fdf0a4b9e17642849a4094d2f0b9a255 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:28.886178 17092 consensus_queue.cc:237] T ae4259dd3dc745869256269632886f8a P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:28.886708 17486 raft_consensus.cc:1203] T fdf0a4b9e17642849a4094d2f0b9a255 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:00:28.888757 16957 raft_consensus.cc:1261] T ae4259dd3dc745869256269632886f8a P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:28.888870 17356 raft_consensus.cc:1261] T ae4259dd3dc745869256269632886f8a P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:28.888972 17751 raft_consensus.cc:1261] T ae4259dd3dc745869256269632886f8a P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:28.889196 19018 consensus_queue.cc:1035] T ae4259dd3dc745869256269632886f8a P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:28.889299 17222 raft_consensus.cc:1261] T ae4259dd3dc745869256269632886f8a P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:28.889405 17486 raft_consensus.cc:1261] T ae4259dd3dc745869256269632886f8a P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:28.889402 19420 consensus_queue.cc:1035] T ae4259dd3dc745869256269632886f8a P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:28.889669 17619 raft_consensus.cc:1261] T ae4259dd3dc745869256269632886f8a P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:28.889747 19420 consensus_queue.cc:1035] T ae4259dd3dc745869256269632886f8a P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:28.889874 19018 consensus_queue.cc:1035] T ae4259dd3dc745869256269632886f8a P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:28.891064 19018 consensus_queue.cc:1035] T ae4259dd3dc745869256269632886f8a P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:28.891383 19018 consensus_queue.cc:1035] T ae4259dd3dc745869256269632886f8a P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:28.893033 19675 raft_consensus.cc:2932] T ae4259dd3dc745869256269632886f8a P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } }
I20240503 02:00:28.894941 17222 raft_consensus.cc:2932] T ae4259dd3dc745869256269632886f8a P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } }
I20240503 02:00:28.895041 17353 raft_consensus.cc:2932] T ae4259dd3dc745869256269632886f8a P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } }
I20240503 02:00:28.895418 17486 raft_consensus.cc:2932] T ae4259dd3dc745869256269632886f8a P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } }
I20240503 02:00:28.898013 16811 catalog_manager.cc:5484] T ae4259dd3dc745869256269632886f8a P 5b8291b26a7f448b8636187d1caf86d0 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "89dd09f8aaee4b249ae3ce924a422a67" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } }
I20240503 02:00:28.898221 16957 raft_consensus.cc:2932] T ae4259dd3dc745869256269632886f8a P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } }
I20240503 02:00:28.901695 17750 raft_consensus.cc:2932] T ae4259dd3dc745869256269632886f8a P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } }
I20240503 02:00:28.904760 17619 raft_consensus.cc:2932] T ae4259dd3dc745869256269632886f8a P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } }
I20240503 02:00:28.907243 17092 consensus_queue.cc:237] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:28.907341 17090 consensus_queue.cc:237] T ae4259dd3dc745869256269632886f8a P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } }
I20240503 02:00:28.909529 16958 raft_consensus.cc:1261] T ae4259dd3dc745869256269632886f8a P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:28.909899 19018 consensus_queue.cc:1035] T ae4259dd3dc745869256269632886f8a P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:28.910117 16958 raft_consensus.cc:1261] T 5833b4f9a5434734817f154d660aa9f7 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:28.910148 17223 raft_consensus.cc:1261] T 5833b4f9a5434734817f154d660aa9f7 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:28.910498 19677 consensus_queue.cc:1035] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:28.911312 17222 raft_consensus.cc:1261] T ae4259dd3dc745869256269632886f8a P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:28.912050 17353 raft_consensus.cc:1261] T 5833b4f9a5434734817f154d660aa9f7 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:28.912050 17356 raft_consensus.cc:1261] T ae4259dd3dc745869256269632886f8a P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:28.912321 19677 consensus_queue.cc:1035] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:28.912433 19676 consensus_queue.cc:1035] T ae4259dd3dc745869256269632886f8a P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:28.912478 19018 consensus_queue.cc:1035] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:28.913125 17486 raft_consensus.cc:1261] T ae4259dd3dc745869256269632886f8a P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:28.913153 17750 raft_consensus.cc:1261] T ae4259dd3dc745869256269632886f8a P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:28.913434 17618 raft_consensus.cc:1261] T ae4259dd3dc745869256269632886f8a P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:28.913590 17751 raft_consensus.cc:1261] T 5833b4f9a5434734817f154d660aa9f7 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:28.913604 17486 raft_consensus.cc:1261] T 5833b4f9a5434734817f154d660aa9f7 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:28.913682 17619 raft_consensus.cc:1261] T 5833b4f9a5434734817f154d660aa9f7 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:28.913971 19420 consensus_queue.cc:1035] T ae4259dd3dc745869256269632886f8a P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:28.914127 19018 consensus_queue.cc:1035] T ae4259dd3dc745869256269632886f8a P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:28.914217 19675 consensus_queue.cc:1035] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:28.914273 19676 consensus_queue.cc:1035] T ae4259dd3dc745869256269632886f8a P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:28.914532 19675 consensus_queue.cc:1035] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:28.914541 19676 consensus_queue.cc:1035] T ae4259dd3dc745869256269632886f8a P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:28.914662 19018 consensus_queue.cc:1035] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:28.916867 19676 raft_consensus.cc:2932] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } }
I20240503 02:00:28.917493 17222 raft_consensus.cc:2932] T 5833b4f9a5434734817f154d660aa9f7 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } }
I20240503 02:00:28.918848 19018 raft_consensus.cc:2932] T ae4259dd3dc745869256269632886f8a P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:28.920214 16812 catalog_manager.cc:5484] T 5833b4f9a5434734817f154d660aa9f7 P 5b8291b26a7f448b8636187d1caf86d0 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "89dd09f8aaee4b249ae3ce924a422a67" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } }
I20240503 02:00:28.920797 17353 raft_consensus.cc:2932] T 5833b4f9a5434734817f154d660aa9f7 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } }
I20240503 02:00:28.920892 16957 raft_consensus.cc:2932] T ae4259dd3dc745869256269632886f8a P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:28.921059 16805 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet ae4259dd3dc745869256269632886f8a with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 02:00:28.923009 17487 raft_consensus.cc:2932] T ae4259dd3dc745869256269632886f8a P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:28.923154 17356 raft_consensus.cc:2932] T ae4259dd3dc745869256269632886f8a P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:28.923492 16809 catalog_manager.cc:5484] T ae4259dd3dc745869256269632886f8a P 6398c8993c3c45dcaab25b42142d5671 reported cstate change: config changed from index 4 to 5, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "89dd09f8aaee4b249ae3ce924a422a67" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
W20240503 02:00:28.925333 17026 consensus_peers.cc:485] T ae4259dd3dc745869256269632886f8a P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ae4259dd3dc745869256269632886f8a. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:28.925590 17222 raft_consensus.cc:2932] T ae4259dd3dc745869256269632886f8a P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:28.925717 17750 raft_consensus.cc:2932] T 5833b4f9a5434734817f154d660aa9f7 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } }
I20240503 02:00:28.927238 17751 raft_consensus.cc:2932] T ae4259dd3dc745869256269632886f8a P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:28.929033 17487 raft_consensus.cc:2932] T 5833b4f9a5434734817f154d660aa9f7 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } }
I20240503 02:00:28.933432 17619 raft_consensus.cc:2932] T ae4259dd3dc745869256269632886f8a P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:28.936934 16957 raft_consensus.cc:2932] T 5833b4f9a5434734817f154d660aa9f7 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } }
I20240503 02:00:28.940798 17619 raft_consensus.cc:2932] T 5833b4f9a5434734817f154d660aa9f7 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } }
I20240503 02:00:28.942417 17092 consensus_queue.cc:237] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } }
I20240503 02:00:28.944304 16957 raft_consensus.cc:1261] T 5833b4f9a5434734817f154d660aa9f7 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:28.944761 17619 raft_consensus.cc:1261] T 5833b4f9a5434734817f154d660aa9f7 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:28.944813 17222 raft_consensus.cc:1261] T 5833b4f9a5434734817f154d660aa9f7 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:28.944793 19676 consensus_queue.cc:1035] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:28.945323 17356 raft_consensus.cc:1261] T 5833b4f9a5434734817f154d660aa9f7 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:28.945348 17487 raft_consensus.cc:1261] T 5833b4f9a5434734817f154d660aa9f7 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:28.945982 19675 consensus_queue.cc:1035] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:28.946208 19677 consensus_queue.cc:1035] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:28.946368 19018 consensus_queue.cc:1035] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:28.946413 17751 raft_consensus.cc:1261] T 5833b4f9a5434734817f154d660aa9f7 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:28.946838 19018 consensus_queue.cc:1035] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:28.949597 19677 consensus_queue.cc:1035] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 02:00:28.949759 17026 consensus_peers.cc:485] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5833b4f9a5434734817f154d660aa9f7. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:28.950554 19677 raft_consensus.cc:2932] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } }
I20240503 02:00:28.951117 17619 raft_consensus.cc:2932] T 5833b4f9a5434734817f154d660aa9f7 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } }
I20240503 02:00:28.952150 16805 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5833b4f9a5434734817f154d660aa9f7 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 02:00:28.952616 17222 raft_consensus.cc:2932] T 5833b4f9a5434734817f154d660aa9f7 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } }
I20240503 02:00:28.952843 16957 raft_consensus.cc:2932] T 5833b4f9a5434734817f154d660aa9f7 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } }
I20240503 02:00:28.953133 17356 raft_consensus.cc:2932] T 5833b4f9a5434734817f154d660aa9f7 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } }
I20240503 02:00:28.955724 16812 catalog_manager.cc:5484] T 5833b4f9a5434734817f154d660aa9f7 P 5b8291b26a7f448b8636187d1caf86d0 reported cstate change: config changed from index 4 to 5, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New cstate: current_term: 1 leader_uuid: "89dd09f8aaee4b249ae3ce924a422a67" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } }
I20240503 02:00:28.956532 17750 raft_consensus.cc:2932] T 5833b4f9a5434734817f154d660aa9f7 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } }
I20240503 02:00:28.962895 17487 raft_consensus.cc:2932] T 5833b4f9a5434734817f154d660aa9f7 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } }
I20240503 02:00:29.029898 19745 ts_tablet_manager.cc:902] T d52afe1281064d6480427554ca4bb2f6 P 9564311af29a4493831a8db658a085c4: Initiating tablet copy from peer 16d437fe629643948c95d628926d5ffa (127.0.99.133:46581)
I20240503 02:00:29.030624 19745 tablet_copy_client.cc:308] T d52afe1281064d6480427554ca4bb2f6 P 9564311af29a4493831a8db658a085c4: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.133:46581
I20240503 02:00:29.034744 17508 tablet_copy_service.cc:140] P 16d437fe629643948c95d628926d5ffa: Received BeginTabletCopySession request for tablet d52afe1281064d6480427554ca4bb2f6 from peer 9564311af29a4493831a8db658a085c4 ({username='slave'} at 127.0.99.138:53357)
I20240503 02:00:29.034879 17508 tablet_copy_service.cc:161] P 16d437fe629643948c95d628926d5ffa: Beginning new tablet copy session on tablet d52afe1281064d6480427554ca4bb2f6 from peer 9564311af29a4493831a8db658a085c4 at {username='slave'} at 127.0.99.138:53357: session id = 9564311af29a4493831a8db658a085c4-d52afe1281064d6480427554ca4bb2f6
I20240503 02:00:29.036054 17508 tablet_copy_source_session.cc:205] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:29.039417 19745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d52afe1281064d6480427554ca4bb2f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:29.044749 19745 tablet_copy_client.cc:783] T d52afe1281064d6480427554ca4bb2f6 P 9564311af29a4493831a8db658a085c4: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:29.044962 19745 tablet_copy_client.cc:647] T d52afe1281064d6480427554ca4bb2f6 P 9564311af29a4493831a8db658a085c4: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:29.046586 19745 tablet_copy_client.cc:523] T d52afe1281064d6480427554ca4bb2f6 P 9564311af29a4493831a8db658a085c4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:29.048087 19745 tablet_bootstrap.cc:492] T d52afe1281064d6480427554ca4bb2f6 P 9564311af29a4493831a8db658a085c4: Bootstrap starting.
I20240503 02:00:29.053920 19745 log.cc:824] T d52afe1281064d6480427554ca4bb2f6 P 9564311af29a4493831a8db658a085c4: Log is configured to *not* fsync() on all Append() calls
I20240503 02:00:29.076889 19745 tablet_bootstrap.cc:492] T d52afe1281064d6480427554ca4bb2f6 P 9564311af29a4493831a8db658a085c4: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:29.077220 19745 tablet_bootstrap.cc:492] T d52afe1281064d6480427554ca4bb2f6 P 9564311af29a4493831a8db658a085c4: Bootstrap complete.
I20240503 02:00:29.077498 19745 ts_tablet_manager.cc:1367] T d52afe1281064d6480427554ca4bb2f6 P 9564311af29a4493831a8db658a085c4: Time spent bootstrapping tablet: real 0.030s user 0.030s sys 0.000s
I20240503 02:00:29.079674 19745 raft_consensus.cc:348] T d52afe1281064d6480427554ca4bb2f6 P 9564311af29a4493831a8db658a085c4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: true } }
I20240503 02:00:29.079892 19745 raft_consensus.cc:729] T d52afe1281064d6480427554ca4bb2f6 P 9564311af29a4493831a8db658a085c4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9564311af29a4493831a8db658a085c4, State: Initialized, Role: LEARNER
I20240503 02:00:29.080238 19745 consensus_queue.cc:260] T d52afe1281064d6480427554ca4bb2f6 P 9564311af29a4493831a8db658a085c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: true } }
I20240503 02:00:29.081137 19745 ts_tablet_manager.cc:1398] T d52afe1281064d6480427554ca4bb2f6 P 9564311af29a4493831a8db658a085c4: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240503 02:00:29.081837 17508 tablet_copy_service.cc:342] P 16d437fe629643948c95d628926d5ffa: Request end of tablet copy session 9564311af29a4493831a8db658a085c4-d52afe1281064d6480427554ca4bb2f6 received from {username='slave'} at 127.0.99.138:53357
I20240503 02:00:29.081945 17508 tablet_copy_service.cc:433] P 16d437fe629643948c95d628926d5ffa: ending tablet copy session 9564311af29a4493831a8db658a085c4-d52afe1281064d6480427554ca4bb2f6 on tablet d52afe1281064d6480427554ca4bb2f6 with peer 9564311af29a4493831a8db658a085c4
I20240503 02:00:29.187706 19745 ts_tablet_manager.cc:902] T 5d2104698d5f452c8350cce10afd3898 P 9564311af29a4493831a8db658a085c4: Initiating tablet copy from peer 002cc3411acb4fdfa0b67c6f57f687a6 (127.0.99.135:38989)
I20240503 02:00:29.188060 19753 ts_tablet_manager.cc:902] T ab536115ac7a4ba193f718d421f4000d P 9564311af29a4493831a8db658a085c4: Initiating tablet copy from peer d43ed6b8663947d99a72cd922cc323aa (127.0.99.132:40023)
I20240503 02:00:29.188247 19745 tablet_copy_client.cc:308] T 5d2104698d5f452c8350cce10afd3898 P 9564311af29a4493831a8db658a085c4: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.135:38989
I20240503 02:00:29.188591 19753 tablet_copy_client.cc:308] T ab536115ac7a4ba193f718d421f4000d P 9564311af29a4493831a8db658a085c4: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:40023
I20240503 02:00:29.192237 17772 tablet_copy_service.cc:140] P 002cc3411acb4fdfa0b67c6f57f687a6: Received BeginTabletCopySession request for tablet 5d2104698d5f452c8350cce10afd3898 from peer 9564311af29a4493831a8db658a085c4 ({username='slave'} at 127.0.99.138:57403)
I20240503 02:00:29.192386 17772 tablet_copy_service.cc:161] P 002cc3411acb4fdfa0b67c6f57f687a6: Beginning new tablet copy session on tablet 5d2104698d5f452c8350cce10afd3898 from peer 9564311af29a4493831a8db658a085c4 at {username='slave'} at 127.0.99.138:57403: session id = 9564311af29a4493831a8db658a085c4-5d2104698d5f452c8350cce10afd3898
I20240503 02:00:29.192482 17376 tablet_copy_service.cc:140] P d43ed6b8663947d99a72cd922cc323aa: Received BeginTabletCopySession request for tablet ab536115ac7a4ba193f718d421f4000d from peer 9564311af29a4493831a8db658a085c4 ({username='slave'} at 127.0.99.138:42723)
I20240503 02:00:29.192615 17376 tablet_copy_service.cc:161] P d43ed6b8663947d99a72cd922cc323aa: Beginning new tablet copy session on tablet ab536115ac7a4ba193f718d421f4000d from peer 9564311af29a4493831a8db658a085c4 at {username='slave'} at 127.0.99.138:42723: session id = 9564311af29a4493831a8db658a085c4-ab536115ac7a4ba193f718d421f4000d
I20240503 02:00:29.193905 17376 tablet_copy_source_session.cc:205] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:29.193905 17772 tablet_copy_source_session.cc:205] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:29.195111 19745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d2104698d5f452c8350cce10afd3898. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:29.196938 19753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab536115ac7a4ba193f718d421f4000d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:29.196936 19745 tablet_copy_client.cc:783] T 5d2104698d5f452c8350cce10afd3898 P 9564311af29a4493831a8db658a085c4: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:29.197486 19745 tablet_copy_client.cc:647] T 5d2104698d5f452c8350cce10afd3898 P 9564311af29a4493831a8db658a085c4: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:29.199293 19745 tablet_copy_client.cc:523] T 5d2104698d5f452c8350cce10afd3898 P 9564311af29a4493831a8db658a085c4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:29.200258 19753 tablet_copy_client.cc:783] T ab536115ac7a4ba193f718d421f4000d P 9564311af29a4493831a8db658a085c4: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:29.200418 19753 tablet_copy_client.cc:647] T ab536115ac7a4ba193f718d421f4000d P 9564311af29a4493831a8db658a085c4: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:29.200575 19745 tablet_bootstrap.cc:492] T 5d2104698d5f452c8350cce10afd3898 P 9564311af29a4493831a8db658a085c4: Bootstrap starting.
I20240503 02:00:29.201915 19753 tablet_copy_client.cc:523] T ab536115ac7a4ba193f718d421f4000d P 9564311af29a4493831a8db658a085c4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:29.202829 19753 tablet_bootstrap.cc:492] T ab536115ac7a4ba193f718d421f4000d P 9564311af29a4493831a8db658a085c4: Bootstrap starting.
I20240503 02:00:29.211793 19771 ts_tablet_manager.cc:902] T acfd773c1f384c219d776878407d1144 P 375d0f2ec11d4e25bd9d093bb6a5572d: Initiating tablet copy from peer 16d437fe629643948c95d628926d5ffa (127.0.99.133:46581)
I20240503 02:00:29.212636 19771 tablet_copy_client.cc:308] T acfd773c1f384c219d776878407d1144 P 375d0f2ec11d4e25bd9d093bb6a5572d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.133:46581
I20240503 02:00:29.213512 19770 ts_tablet_manager.cc:902] T 63962b169caf4f14888871e246c00075 P 9564311af29a4493831a8db658a085c4: Initiating tablet copy from peer d43ed6b8663947d99a72cd922cc323aa (127.0.99.132:40023)
I20240503 02:00:29.214076 19770 tablet_copy_client.cc:308] T 63962b169caf4f14888871e246c00075 P 9564311af29a4493831a8db658a085c4: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:40023
I20240503 02:00:29.214493 17376 tablet_copy_service.cc:140] P d43ed6b8663947d99a72cd922cc323aa: Received BeginTabletCopySession request for tablet 63962b169caf4f14888871e246c00075 from peer 9564311af29a4493831a8db658a085c4 ({username='slave'} at 127.0.99.138:42723)
I20240503 02:00:29.214587 17376 tablet_copy_service.cc:161] P d43ed6b8663947d99a72cd922cc323aa: Beginning new tablet copy session on tablet 63962b169caf4f14888871e246c00075 from peer 9564311af29a4493831a8db658a085c4 at {username='slave'} at 127.0.99.138:42723: session id = 9564311af29a4493831a8db658a085c4-63962b169caf4f14888871e246c00075
I20240503 02:00:29.215449 17376 tablet_copy_source_session.cc:205] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:29.216537 19770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63962b169caf4f14888871e246c00075. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:29.220721 19770 tablet_copy_client.cc:783] T 63962b169caf4f14888871e246c00075 P 9564311af29a4493831a8db658a085c4: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:29.220979 19770 tablet_copy_client.cc:647] T 63962b169caf4f14888871e246c00075 P 9564311af29a4493831a8db658a085c4: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:29.222765 19770 tablet_copy_client.cc:523] T 63962b169caf4f14888871e246c00075 P 9564311af29a4493831a8db658a085c4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:29.224027 19770 tablet_bootstrap.cc:492] T 63962b169caf4f14888871e246c00075 P 9564311af29a4493831a8db658a085c4: Bootstrap starting.
I20240503 02:00:29.227118 17508 tablet_copy_service.cc:140] P 16d437fe629643948c95d628926d5ffa: Received BeginTabletCopySession request for tablet acfd773c1f384c219d776878407d1144 from peer 375d0f2ec11d4e25bd9d093bb6a5572d ({username='slave'} at 127.0.99.136:36855)
I20240503 02:00:29.227233 17508 tablet_copy_service.cc:161] P 16d437fe629643948c95d628926d5ffa: Beginning new tablet copy session on tablet acfd773c1f384c219d776878407d1144 from peer 375d0f2ec11d4e25bd9d093bb6a5572d at {username='slave'} at 127.0.99.136:36855: session id = 375d0f2ec11d4e25bd9d093bb6a5572d-acfd773c1f384c219d776878407d1144
I20240503 02:00:29.228039 17508 tablet_copy_source_session.cc:205] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:29.229423 19745 tablet_bootstrap.cc:492] T 5d2104698d5f452c8350cce10afd3898 P 9564311af29a4493831a8db658a085c4: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:29.229729 19745 tablet_bootstrap.cc:492] T 5d2104698d5f452c8350cce10afd3898 P 9564311af29a4493831a8db658a085c4: Bootstrap complete.
I20240503 02:00:29.229920 19745 ts_tablet_manager.cc:1367] T 5d2104698d5f452c8350cce10afd3898 P 9564311af29a4493831a8db658a085c4: Time spent bootstrapping tablet: real 0.029s user 0.022s sys 0.004s
I20240503 02:00:29.230530 19745 raft_consensus.cc:348] T 5d2104698d5f452c8350cce10afd3898 P 9564311af29a4493831a8db658a085c4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: true } }
I20240503 02:00:29.230749 19745 raft_consensus.cc:729] T 5d2104698d5f452c8350cce10afd3898 P 9564311af29a4493831a8db658a085c4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9564311af29a4493831a8db658a085c4, State: Initialized, Role: LEARNER
I20240503 02:00:29.230906 19745 consensus_queue.cc:260] T 5d2104698d5f452c8350cce10afd3898 P 9564311af29a4493831a8db658a085c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: true } }
I20240503 02:00:29.231247 19745 ts_tablet_manager.cc:1398] T 5d2104698d5f452c8350cce10afd3898 P 9564311af29a4493831a8db658a085c4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:29.231434 19771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acfd773c1f384c219d776878407d1144. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:29.232421 19753 tablet_bootstrap.cc:492] T ab536115ac7a4ba193f718d421f4000d P 9564311af29a4493831a8db658a085c4: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:29.232718 19753 tablet_bootstrap.cc:492] T ab536115ac7a4ba193f718d421f4000d P 9564311af29a4493831a8db658a085c4: Bootstrap complete.
I20240503 02:00:29.232884 19753 ts_tablet_manager.cc:1367] T ab536115ac7a4ba193f718d421f4000d P 9564311af29a4493831a8db658a085c4: Time spent bootstrapping tablet: real 0.030s user 0.026s sys 0.001s
I20240503 02:00:29.233264 19753 raft_consensus.cc:348] T ab536115ac7a4ba193f718d421f4000d P 9564311af29a4493831a8db658a085c4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: true } }
I20240503 02:00:29.233403 19753 raft_consensus.cc:729] T ab536115ac7a4ba193f718d421f4000d P 9564311af29a4493831a8db658a085c4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9564311af29a4493831a8db658a085c4, State: Initialized, Role: LEARNER
I20240503 02:00:29.233491 19753 consensus_queue.cc:260] T ab536115ac7a4ba193f718d421f4000d P 9564311af29a4493831a8db658a085c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: true } }
I20240503 02:00:29.234127 19753 ts_tablet_manager.cc:1398] T ab536115ac7a4ba193f718d421f4000d P 9564311af29a4493831a8db658a085c4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:29.234172 17772 tablet_copy_service.cc:342] P 002cc3411acb4fdfa0b67c6f57f687a6: Request end of tablet copy session 9564311af29a4493831a8db658a085c4-5d2104698d5f452c8350cce10afd3898 received from {username='slave'} at 127.0.99.138:57403
I20240503 02:00:29.234246 17772 tablet_copy_service.cc:433] P 002cc3411acb4fdfa0b67c6f57f687a6: ending tablet copy session 9564311af29a4493831a8db658a085c4-5d2104698d5f452c8350cce10afd3898 on tablet 5d2104698d5f452c8350cce10afd3898 with peer 9564311af29a4493831a8db658a085c4
I20240503 02:00:29.234642 17376 tablet_copy_service.cc:342] P d43ed6b8663947d99a72cd922cc323aa: Request end of tablet copy session 9564311af29a4493831a8db658a085c4-ab536115ac7a4ba193f718d421f4000d received from {username='slave'} at 127.0.99.138:42723
I20240503 02:00:29.234738 17376 tablet_copy_service.cc:433] P d43ed6b8663947d99a72cd922cc323aa: ending tablet copy session 9564311af29a4493831a8db658a085c4-ab536115ac7a4ba193f718d421f4000d on tablet ab536115ac7a4ba193f718d421f4000d with peer 9564311af29a4493831a8db658a085c4
I20240503 02:00:29.241209 19771 tablet_copy_client.cc:783] T acfd773c1f384c219d776878407d1144 P 375d0f2ec11d4e25bd9d093bb6a5572d: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:29.241475 19771 tablet_copy_client.cc:647] T acfd773c1f384c219d776878407d1144 P 375d0f2ec11d4e25bd9d093bb6a5572d: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:29.244144 19771 tablet_copy_client.cc:523] T acfd773c1f384c219d776878407d1144 P 375d0f2ec11d4e25bd9d093bb6a5572d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:29.246248 19771 tablet_bootstrap.cc:492] T acfd773c1f384c219d776878407d1144 P 375d0f2ec11d4e25bd9d093bb6a5572d: Bootstrap starting.
I20240503 02:00:29.251439 19770 tablet_bootstrap.cc:492] T 63962b169caf4f14888871e246c00075 P 9564311af29a4493831a8db658a085c4: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:29.251741 19770 tablet_bootstrap.cc:492] T 63962b169caf4f14888871e246c00075 P 9564311af29a4493831a8db658a085c4: Bootstrap complete.
I20240503 02:00:29.251909 19770 ts_tablet_manager.cc:1367] T 63962b169caf4f14888871e246c00075 P 9564311af29a4493831a8db658a085c4: Time spent bootstrapping tablet: real 0.028s user 0.023s sys 0.000s
I20240503 02:00:29.252285 19770 raft_consensus.cc:348] T 63962b169caf4f14888871e246c00075 P 9564311af29a4493831a8db658a085c4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: true } }
I20240503 02:00:29.252491 19770 raft_consensus.cc:729] T 63962b169caf4f14888871e246c00075 P 9564311af29a4493831a8db658a085c4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9564311af29a4493831a8db658a085c4, State: Initialized, Role: LEARNER
I20240503 02:00:29.252635 19770 consensus_queue.cc:260] T 63962b169caf4f14888871e246c00075 P 9564311af29a4493831a8db658a085c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: true } }
I20240503 02:00:29.253057 19770 ts_tablet_manager.cc:1398] T 63962b169caf4f14888871e246c00075 P 9564311af29a4493831a8db658a085c4: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:29.253479 19771 log.cc:824] T acfd773c1f384c219d776878407d1144 P 375d0f2ec11d4e25bd9d093bb6a5572d: Log is configured to *not* fsync() on all Append() calls
I20240503 02:00:29.253597 17376 tablet_copy_service.cc:342] P d43ed6b8663947d99a72cd922cc323aa: Request end of tablet copy session 9564311af29a4493831a8db658a085c4-63962b169caf4f14888871e246c00075 received from {username='slave'} at 127.0.99.138:42723
I20240503 02:00:29.253674 17376 tablet_copy_service.cc:433] P d43ed6b8663947d99a72cd922cc323aa: ending tablet copy session 9564311af29a4493831a8db658a085c4-63962b169caf4f14888871e246c00075 on tablet 63962b169caf4f14888871e246c00075 with peer 9564311af29a4493831a8db658a085c4
I20240503 02:00:29.280925 19771 tablet_bootstrap.cc:492] T acfd773c1f384c219d776878407d1144 P 375d0f2ec11d4e25bd9d093bb6a5572d: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:29.281347 19771 tablet_bootstrap.cc:492] T acfd773c1f384c219d776878407d1144 P 375d0f2ec11d4e25bd9d093bb6a5572d: Bootstrap complete.
I20240503 02:00:29.281631 19771 ts_tablet_manager.cc:1367] T acfd773c1f384c219d776878407d1144 P 375d0f2ec11d4e25bd9d093bb6a5572d: Time spent bootstrapping tablet: real 0.036s user 0.033s sys 0.001s
I20240503 02:00:29.283962 19771 raft_consensus.cc:348] T acfd773c1f384c219d776878407d1144 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } }
I20240503 02:00:29.284170 19771 raft_consensus.cc:729] T acfd773c1f384c219d776878407d1144 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 375d0f2ec11d4e25bd9d093bb6a5572d, State: Initialized, Role: LEARNER
I20240503 02:00:29.284507 19771 consensus_queue.cc:260] T acfd773c1f384c219d776878407d1144 P 375d0f2ec11d4e25bd9d093bb6a5572d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } }
I20240503 02:00:29.285413 19771 ts_tablet_manager.cc:1398] T acfd773c1f384c219d776878407d1144 P 375d0f2ec11d4e25bd9d093bb6a5572d: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20240503 02:00:29.286319 17508 tablet_copy_service.cc:342] P 16d437fe629643948c95d628926d5ffa: Request end of tablet copy session 375d0f2ec11d4e25bd9d093bb6a5572d-acfd773c1f384c219d776878407d1144 received from {username='slave'} at 127.0.99.136:36855
I20240503 02:00:29.286398 17508 tablet_copy_service.cc:433] P 16d437fe629643948c95d628926d5ffa: ending tablet copy session 375d0f2ec11d4e25bd9d093bb6a5572d-acfd773c1f384c219d776878407d1144 on tablet acfd773c1f384c219d776878407d1144 with peer 375d0f2ec11d4e25bd9d093bb6a5572d
I20240503 02:00:29.312278 19771 ts_tablet_manager.cc:902] T ae4259dd3dc745869256269632886f8a P 375d0f2ec11d4e25bd9d093bb6a5572d: Initiating tablet copy from peer 89dd09f8aaee4b249ae3ce924a422a67 (127.0.99.130:38889)
I20240503 02:00:29.312821 19771 tablet_copy_client.cc:308] T ae4259dd3dc745869256269632886f8a P 375d0f2ec11d4e25bd9d093bb6a5572d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:38889
I20240503 02:00:29.316402 17112 tablet_copy_service.cc:140] P 89dd09f8aaee4b249ae3ce924a422a67: Received BeginTabletCopySession request for tablet ae4259dd3dc745869256269632886f8a from peer 375d0f2ec11d4e25bd9d093bb6a5572d ({username='slave'} at 127.0.99.136:56011)
I20240503 02:00:29.316558 17112 tablet_copy_service.cc:161] P 89dd09f8aaee4b249ae3ce924a422a67: Beginning new tablet copy session on tablet ae4259dd3dc745869256269632886f8a from peer 375d0f2ec11d4e25bd9d093bb6a5572d at {username='slave'} at 127.0.99.136:56011: session id = 375d0f2ec11d4e25bd9d093bb6a5572d-ae4259dd3dc745869256269632886f8a
I20240503 02:00:29.318027 17112 tablet_copy_source_session.cc:205] T ae4259dd3dc745869256269632886f8a P 89dd09f8aaee4b249ae3ce924a422a67: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:29.319227 19771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae4259dd3dc745869256269632886f8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:29.321388 19771 tablet_copy_client.cc:783] T ae4259dd3dc745869256269632886f8a P 375d0f2ec11d4e25bd9d093bb6a5572d: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:29.321532 19771 tablet_copy_client.cc:647] T ae4259dd3dc745869256269632886f8a P 375d0f2ec11d4e25bd9d093bb6a5572d: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:29.322877 19771 tablet_copy_client.cc:523] T ae4259dd3dc745869256269632886f8a P 375d0f2ec11d4e25bd9d093bb6a5572d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:29.323935 19771 tablet_bootstrap.cc:492] T ae4259dd3dc745869256269632886f8a P 375d0f2ec11d4e25bd9d093bb6a5572d: Bootstrap starting.
I20240503 02:00:29.330979 19770 ts_tablet_manager.cc:902] T fbec07350ad04c49afd42e5d88dda6ad P 9564311af29a4493831a8db658a085c4: Initiating tablet copy from peer 16d437fe629643948c95d628926d5ffa (127.0.99.133:46581)
I20240503 02:00:29.331440 19770 tablet_copy_client.cc:308] T fbec07350ad04c49afd42e5d88dda6ad P 9564311af29a4493831a8db658a085c4: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.133:46581
I20240503 02:00:29.331907 17508 tablet_copy_service.cc:140] P 16d437fe629643948c95d628926d5ffa: Received BeginTabletCopySession request for tablet fbec07350ad04c49afd42e5d88dda6ad from peer 9564311af29a4493831a8db658a085c4 ({username='slave'} at 127.0.99.138:53357)
I20240503 02:00:29.332013 17508 tablet_copy_service.cc:161] P 16d437fe629643948c95d628926d5ffa: Beginning new tablet copy session on tablet fbec07350ad04c49afd42e5d88dda6ad from peer 9564311af29a4493831a8db658a085c4 at {username='slave'} at 127.0.99.138:53357: session id = 9564311af29a4493831a8db658a085c4-fbec07350ad04c49afd42e5d88dda6ad
I20240503 02:00:29.332892 17508 tablet_copy_source_session.cc:205] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:29.333720 19770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbec07350ad04c49afd42e5d88dda6ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:29.335739 19770 tablet_copy_client.cc:783] T fbec07350ad04c49afd42e5d88dda6ad P 9564311af29a4493831a8db658a085c4: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:29.336169 19770 tablet_copy_client.cc:647] T fbec07350ad04c49afd42e5d88dda6ad P 9564311af29a4493831a8db658a085c4: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:29.337399 19770 tablet_copy_client.cc:523] T fbec07350ad04c49afd42e5d88dda6ad P 9564311af29a4493831a8db658a085c4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:29.338529 19770 tablet_bootstrap.cc:492] T fbec07350ad04c49afd42e5d88dda6ad P 9564311af29a4493831a8db658a085c4: Bootstrap starting.
I20240503 02:00:29.347318 19771 tablet_bootstrap.cc:492] T ae4259dd3dc745869256269632886f8a P 375d0f2ec11d4e25bd9d093bb6a5572d: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:29.347585 19771 tablet_bootstrap.cc:492] T ae4259dd3dc745869256269632886f8a P 375d0f2ec11d4e25bd9d093bb6a5572d: Bootstrap complete.
I20240503 02:00:29.347738 19771 ts_tablet_manager.cc:1367] T ae4259dd3dc745869256269632886f8a P 375d0f2ec11d4e25bd9d093bb6a5572d: Time spent bootstrapping tablet: real 0.024s user 0.021s sys 0.000s
I20240503 02:00:29.348279 19771 raft_consensus.cc:348] T ae4259dd3dc745869256269632886f8a P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } }
I20240503 02:00:29.348417 19771 raft_consensus.cc:729] T ae4259dd3dc745869256269632886f8a P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 375d0f2ec11d4e25bd9d093bb6a5572d, State: Initialized, Role: LEARNER
I20240503 02:00:29.348515 19771 consensus_queue.cc:260] T ae4259dd3dc745869256269632886f8a P 375d0f2ec11d4e25bd9d093bb6a5572d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } }
I20240503 02:00:29.348795 19771 ts_tablet_manager.cc:1398] T ae4259dd3dc745869256269632886f8a P 375d0f2ec11d4e25bd9d093bb6a5572d: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240503 02:00:29.349614 17112 tablet_copy_service.cc:342] P 89dd09f8aaee4b249ae3ce924a422a67: Request end of tablet copy session 375d0f2ec11d4e25bd9d093bb6a5572d-ae4259dd3dc745869256269632886f8a received from {username='slave'} at 127.0.99.136:56011
I20240503 02:00:29.349704 17112 tablet_copy_service.cc:433] P 89dd09f8aaee4b249ae3ce924a422a67: ending tablet copy session 375d0f2ec11d4e25bd9d093bb6a5572d-ae4259dd3dc745869256269632886f8a on tablet ae4259dd3dc745869256269632886f8a with peer 375d0f2ec11d4e25bd9d093bb6a5572d
I20240503 02:00:29.360877 19770 tablet_bootstrap.cc:492] T fbec07350ad04c49afd42e5d88dda6ad P 9564311af29a4493831a8db658a085c4: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:29.361099 19770 tablet_bootstrap.cc:492] T fbec07350ad04c49afd42e5d88dda6ad P 9564311af29a4493831a8db658a085c4: Bootstrap complete.
I20240503 02:00:29.361236 19770 ts_tablet_manager.cc:1367] T fbec07350ad04c49afd42e5d88dda6ad P 9564311af29a4493831a8db658a085c4: Time spent bootstrapping tablet: real 0.023s user 0.022s sys 0.001s
I20240503 02:00:29.361730 19770 raft_consensus.cc:348] T fbec07350ad04c49afd42e5d88dda6ad P 9564311af29a4493831a8db658a085c4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: true } }
I20240503 02:00:29.361869 19770 raft_consensus.cc:729] T fbec07350ad04c49afd42e5d88dda6ad P 9564311af29a4493831a8db658a085c4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9564311af29a4493831a8db658a085c4, State: Initialized, Role: LEARNER
I20240503 02:00:29.361984 19770 consensus_queue.cc:260] T fbec07350ad04c49afd42e5d88dda6ad P 9564311af29a4493831a8db658a085c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: true } }
I20240503 02:00:29.362295 19770 ts_tablet_manager.cc:1398] T fbec07350ad04c49afd42e5d88dda6ad P 9564311af29a4493831a8db658a085c4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:29.362761 17508 tablet_copy_service.cc:342] P 16d437fe629643948c95d628926d5ffa: Request end of tablet copy session 9564311af29a4493831a8db658a085c4-fbec07350ad04c49afd42e5d88dda6ad received from {username='slave'} at 127.0.99.138:53357
I20240503 02:00:29.362854 17508 tablet_copy_service.cc:433] P 16d437fe629643948c95d628926d5ffa: ending tablet copy session 9564311af29a4493831a8db658a085c4-fbec07350ad04c49afd42e5d88dda6ad on tablet fbec07350ad04c49afd42e5d88dda6ad with peer 9564311af29a4493831a8db658a085c4
I20240503 02:00:29.372138 19771 ts_tablet_manager.cc:902] T fdf0a4b9e17642849a4094d2f0b9a255 P 375d0f2ec11d4e25bd9d093bb6a5572d: Initiating tablet copy from peer 002cc3411acb4fdfa0b67c6f57f687a6 (127.0.99.135:38989)
I20240503 02:00:29.372597 19771 tablet_copy_client.cc:308] T fdf0a4b9e17642849a4094d2f0b9a255 P 375d0f2ec11d4e25bd9d093bb6a5572d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.135:38989
I20240503 02:00:29.375952 17772 tablet_copy_service.cc:140] P 002cc3411acb4fdfa0b67c6f57f687a6: Received BeginTabletCopySession request for tablet fdf0a4b9e17642849a4094d2f0b9a255 from peer 375d0f2ec11d4e25bd9d093bb6a5572d ({username='slave'} at 127.0.99.136:48307)
I20240503 02:00:29.376041 17772 tablet_copy_service.cc:161] P 002cc3411acb4fdfa0b67c6f57f687a6: Beginning new tablet copy session on tablet fdf0a4b9e17642849a4094d2f0b9a255 from peer 375d0f2ec11d4e25bd9d093bb6a5572d at {username='slave'} at 127.0.99.136:48307: session id = 375d0f2ec11d4e25bd9d093bb6a5572d-fdf0a4b9e17642849a4094d2f0b9a255
I20240503 02:00:29.376802 17772 tablet_copy_source_session.cc:205] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:29.377547 19771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdf0a4b9e17642849a4094d2f0b9a255. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:29.379189 19771 tablet_copy_client.cc:783] T fdf0a4b9e17642849a4094d2f0b9a255 P 375d0f2ec11d4e25bd9d093bb6a5572d: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:29.379328 19771 tablet_copy_client.cc:647] T fdf0a4b9e17642849a4094d2f0b9a255 P 375d0f2ec11d4e25bd9d093bb6a5572d: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:29.380275 19771 tablet_copy_client.cc:523] T fdf0a4b9e17642849a4094d2f0b9a255 P 375d0f2ec11d4e25bd9d093bb6a5572d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:29.381203 19771 tablet_bootstrap.cc:492] T fdf0a4b9e17642849a4094d2f0b9a255 P 375d0f2ec11d4e25bd9d093bb6a5572d: Bootstrap starting.
I20240503 02:00:29.399928 19771 tablet_bootstrap.cc:492] T fdf0a4b9e17642849a4094d2f0b9a255 P 375d0f2ec11d4e25bd9d093bb6a5572d: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:29.400205 19771 tablet_bootstrap.cc:492] T fdf0a4b9e17642849a4094d2f0b9a255 P 375d0f2ec11d4e25bd9d093bb6a5572d: Bootstrap complete.
I20240503 02:00:29.400362 19771 ts_tablet_manager.cc:1367] T fdf0a4b9e17642849a4094d2f0b9a255 P 375d0f2ec11d4e25bd9d093bb6a5572d: Time spent bootstrapping tablet: real 0.019s user 0.012s sys 0.004s
I20240503 02:00:29.400774 19771 raft_consensus.cc:348] T fdf0a4b9e17642849a4094d2f0b9a255 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } }
I20240503 02:00:29.400878 19771 raft_consensus.cc:729] T fdf0a4b9e17642849a4094d2f0b9a255 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 375d0f2ec11d4e25bd9d093bb6a5572d, State: Initialized, Role: LEARNER
I20240503 02:00:29.400981 19771 consensus_queue.cc:260] T fdf0a4b9e17642849a4094d2f0b9a255 P 375d0f2ec11d4e25bd9d093bb6a5572d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } }
I20240503 02:00:29.401216 19771 ts_tablet_manager.cc:1398] T fdf0a4b9e17642849a4094d2f0b9a255 P 375d0f2ec11d4e25bd9d093bb6a5572d: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240503 02:00:29.401647 17772 tablet_copy_service.cc:342] P 002cc3411acb4fdfa0b67c6f57f687a6: Request end of tablet copy session 375d0f2ec11d4e25bd9d093bb6a5572d-fdf0a4b9e17642849a4094d2f0b9a255 received from {username='slave'} at 127.0.99.136:48307
I20240503 02:00:29.401722 17772 tablet_copy_service.cc:433] P 002cc3411acb4fdfa0b67c6f57f687a6: ending tablet copy session 375d0f2ec11d4e25bd9d093bb6a5572d-fdf0a4b9e17642849a4094d2f0b9a255 on tablet fdf0a4b9e17642849a4094d2f0b9a255 with peer 375d0f2ec11d4e25bd9d093bb6a5572d
I20240503 02:00:29.530606 19239 raft_consensus.cc:1203] T d52afe1281064d6480427554ca4bb2f6 P 9564311af29a4493831a8db658a085c4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 02:00:29.555076 19886 ts_tablet_manager.cc:902] T 5833b4f9a5434734817f154d660aa9f7 P 838ef968bb45484bb27f5af6f3989bae: Initiating tablet copy from peer 89dd09f8aaee4b249ae3ce924a422a67 (127.0.99.130:38889)
I20240503 02:00:29.555784 19886 tablet_copy_client.cc:308] T 5833b4f9a5434734817f154d660aa9f7 P 838ef968bb45484bb27f5af6f3989bae: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:38889
I20240503 02:00:29.559652 17112 tablet_copy_service.cc:140] P 89dd09f8aaee4b249ae3ce924a422a67: Received BeginTabletCopySession request for tablet 5833b4f9a5434734817f154d660aa9f7 from peer 838ef968bb45484bb27f5af6f3989bae ({username='slave'} at 127.0.99.137:47267)
I20240503 02:00:29.559744 17112 tablet_copy_service.cc:161] P 89dd09f8aaee4b249ae3ce924a422a67: Beginning new tablet copy session on tablet 5833b4f9a5434734817f154d660aa9f7 from peer 838ef968bb45484bb27f5af6f3989bae at {username='slave'} at 127.0.99.137:47267: session id = 838ef968bb45484bb27f5af6f3989bae-5833b4f9a5434734817f154d660aa9f7
I20240503 02:00:29.560441 17112 tablet_copy_source_session.cc:205] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:29.563401 19886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5833b4f9a5434734817f154d660aa9f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:29.568295 19886 tablet_copy_client.cc:783] T 5833b4f9a5434734817f154d660aa9f7 P 838ef968bb45484bb27f5af6f3989bae: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:29.569221 19886 tablet_copy_client.cc:647] T 5833b4f9a5434734817f154d660aa9f7 P 838ef968bb45484bb27f5af6f3989bae: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:29.570580 19886 tablet_copy_client.cc:523] T 5833b4f9a5434734817f154d660aa9f7 P 838ef968bb45484bb27f5af6f3989bae: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:29.571933 19886 tablet_bootstrap.cc:492] T 5833b4f9a5434734817f154d660aa9f7 P 838ef968bb45484bb27f5af6f3989bae: Bootstrap starting.
I20240503 02:00:29.577100 19886 log.cc:824] T 5833b4f9a5434734817f154d660aa9f7 P 838ef968bb45484bb27f5af6f3989bae: Log is configured to *not* fsync() on all Append() calls
I20240503 02:00:29.596540 19886 tablet_bootstrap.cc:492] T 5833b4f9a5434734817f154d660aa9f7 P 838ef968bb45484bb27f5af6f3989bae: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:29.596891 19886 tablet_bootstrap.cc:492] T 5833b4f9a5434734817f154d660aa9f7 P 838ef968bb45484bb27f5af6f3989bae: Bootstrap complete.
I20240503 02:00:29.597148 19886 ts_tablet_manager.cc:1367] T 5833b4f9a5434734817f154d660aa9f7 P 838ef968bb45484bb27f5af6f3989bae: Time spent bootstrapping tablet: real 0.025s user 0.026s sys 0.000s
I20240503 02:00:29.599143 19886 raft_consensus.cc:348] T 5833b4f9a5434734817f154d660aa9f7 P 838ef968bb45484bb27f5af6f3989bae [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } }
I20240503 02:00:29.599359 19886 raft_consensus.cc:729] T 5833b4f9a5434734817f154d660aa9f7 P 838ef968bb45484bb27f5af6f3989bae [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 838ef968bb45484bb27f5af6f3989bae, State: Initialized, Role: LEARNER
I20240503 02:00:29.599625 19886 consensus_queue.cc:260] T 5833b4f9a5434734817f154d660aa9f7 P 838ef968bb45484bb27f5af6f3989bae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } }
I20240503 02:00:29.600433 19886 ts_tablet_manager.cc:1398] T 5833b4f9a5434734817f154d660aa9f7 P 838ef968bb45484bb27f5af6f3989bae: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 02:00:29.601132 17112 tablet_copy_service.cc:342] P 89dd09f8aaee4b249ae3ce924a422a67: Request end of tablet copy session 838ef968bb45484bb27f5af6f3989bae-5833b4f9a5434734817f154d660aa9f7 received from {username='slave'} at 127.0.99.137:47267
I20240503 02:00:29.601217 17112 tablet_copy_service.cc:433] P 89dd09f8aaee4b249ae3ce924a422a67: ending tablet copy session 838ef968bb45484bb27f5af6f3989bae-5833b4f9a5434734817f154d660aa9f7 on tablet 5833b4f9a5434734817f154d660aa9f7 with peer 838ef968bb45484bb27f5af6f3989bae
I20240503 02:00:29.642643 19239 raft_consensus.cc:1203] T 5d2104698d5f452c8350cce10afd3898 P 9564311af29a4493831a8db658a085c4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 02:00:29.712999 19239 raft_consensus.cc:1203] T 63962b169caf4f14888871e246c00075 P 9564311af29a4493831a8db658a085c4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 02:00:29.718559 19370 raft_consensus.cc:1203] T ae4259dd3dc745869256269632886f8a P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 02:00:29.775938 19370 raft_consensus.cc:1203] T acfd773c1f384c219d776878407d1144 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 02:00:29.790840 19239 raft_consensus.cc:1203] T ab536115ac7a4ba193f718d421f4000d P 9564311af29a4493831a8db658a085c4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 02:00:29.823992 19239 raft_consensus.cc:1203] T fbec07350ad04c49afd42e5d88dda6ad P 9564311af29a4493831a8db658a085c4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 02:00:29.940322 19370 raft_consensus.cc:1203] T fdf0a4b9e17642849a4094d2f0b9a255 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 02:00:30.015558 19620 raft_consensus.cc:1050] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa: attempting to promote NON_VOTER 9564311af29a4493831a8db658a085c4 to VOTER
I20240503 02:00:30.016301 19620 consensus_queue.cc:237] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } }
I20240503 02:00:30.018873 19239 raft_consensus.cc:1261] T d52afe1281064d6480427554ca4bb2f6 P 9564311af29a4493831a8db658a085c4 [term 1 LEARNER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:30.019318 17222 raft_consensus.cc:1261] T d52afe1281064d6480427554ca4bb2f6 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:30.019384 19947 consensus_queue.cc:1035] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:30.019856 17092 raft_consensus.cc:1261] T d52afe1281064d6480427554ca4bb2f6 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:30.020190 19948 consensus_queue.cc:1035] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:30.020547 17353 raft_consensus.cc:1261] T d52afe1281064d6480427554ca4bb2f6 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:30.021407 19949 consensus_queue.cc:1035] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:30.021845 19947 consensus_queue.cc:1035] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:30.022559 16958 raft_consensus.cc:1261] T d52afe1281064d6480427554ca4bb2f6 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:30.023013 19620 consensus_queue.cc:1035] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:30.024936 17750 raft_consensus.cc:1261] T d52afe1281064d6480427554ca4bb2f6 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:30.025350 17618 raft_consensus.cc:1261] T d52afe1281064d6480427554ca4bb2f6 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:30.028132 19947 consensus_queue.cc:1035] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:30.028548 19875 consensus_queue.cc:1035] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:30.028771 19948 raft_consensus.cc:2932] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9564311af29a4493831a8db658a085c4 (127.0.99.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } }
I20240503 02:00:30.030985 17750 raft_consensus.cc:2932] T d52afe1281064d6480427554ca4bb2f6 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9564311af29a4493831a8db658a085c4 (127.0.99.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } }
I20240503 02:00:30.031872 17353 raft_consensus.cc:2932] T d52afe1281064d6480427554ca4bb2f6 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9564311af29a4493831a8db658a085c4 (127.0.99.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } }
I20240503 02:00:30.031742 19239 raft_consensus.cc:2932] T d52afe1281064d6480427554ca4bb2f6 P 9564311af29a4493831a8db658a085c4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9564311af29a4493831a8db658a085c4 (127.0.99.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } }
I20240503 02:00:30.033612 17222 raft_consensus.cc:2932] T d52afe1281064d6480427554ca4bb2f6 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9564311af29a4493831a8db658a085c4 (127.0.99.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } }
I20240503 02:00:30.034946 17618 raft_consensus.cc:2932] T d52afe1281064d6480427554ca4bb2f6 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9564311af29a4493831a8db658a085c4 (127.0.99.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } }
I20240503 02:00:30.035424 16810 catalog_manager.cc:5484] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa reported cstate change: config changed from index 5 to 6, 9564311af29a4493831a8db658a085c4 (127.0.99.138) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "16d437fe629643948c95d628926d5ffa" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:00:30.036144 17092 raft_consensus.cc:2932] T d52afe1281064d6480427554ca4bb2f6 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9564311af29a4493831a8db658a085c4 (127.0.99.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } }
I20240503 02:00:30.038995 16958 raft_consensus.cc:2932] T d52afe1281064d6480427554ca4bb2f6 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9564311af29a4493831a8db658a085c4 (127.0.99.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } }
I20240503 02:00:30.134599 19636 raft_consensus.cc:1050] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6: attempting to promote NON_VOTER 9564311af29a4493831a8db658a085c4 to VOTER
I20240503 02:00:30.135210 19636 consensus_queue.cc:237] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } }
I20240503 02:00:30.137364 17222 raft_consensus.cc:1261] T 5d2104698d5f452c8350cce10afd3898 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:30.137513 19239 raft_consensus.cc:1261] T 5d2104698d5f452c8350cce10afd3898 P 9564311af29a4493831a8db658a085c4 [term 1 LEARNER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:30.137630 17092 raft_consensus.cc:1261] T 5d2104698d5f452c8350cce10afd3898 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:30.137933 16958 raft_consensus.cc:1261] T 5d2104698d5f452c8350cce10afd3898 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:30.138053 17618 raft_consensus.cc:1261] T 5d2104698d5f452c8350cce10afd3898 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:30.138149 17487 raft_consensus.cc:1261] T 5d2104698d5f452c8350cce10afd3898 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:30.138132 20007 consensus_queue.cc:1035] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:30.138381 20004 consensus_queue.cc:1035] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:30.138870 19636 consensus_queue.cc:1035] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:30.138983 17356 raft_consensus.cc:1261] T 5d2104698d5f452c8350cce10afd3898 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:30.138995 19421 consensus_queue.cc:1035] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:30.139098 19648 consensus_queue.cc:1035] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:30.139518 20005 consensus_queue.cc:1035] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:30.140003 19421 consensus_queue.cc:1035] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:30.143172 19421 raft_consensus.cc:2932] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9564311af29a4493831a8db658a085c4 (127.0.99.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } }
I20240503 02:00:30.143795 16958 raft_consensus.cc:2932] T 5d2104698d5f452c8350cce10afd3898 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9564311af29a4493831a8db658a085c4 (127.0.99.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } }
I20240503 02:00:30.143782 19239 raft_consensus.cc:2932] T 5d2104698d5f452c8350cce10afd3898 P 9564311af29a4493831a8db658a085c4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9564311af29a4493831a8db658a085c4 (127.0.99.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } }
I20240503 02:00:30.143972 17618 raft_consensus.cc:2932] T 5d2104698d5f452c8350cce10afd3898 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9564311af29a4493831a8db658a085c4 (127.0.99.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } }
I20240503 02:00:30.147104 16817 catalog_manager.cc:5484] T 5d2104698d5f452c8350cce10afd3898 P 6398c8993c3c45dcaab25b42142d5671 reported cstate change: config changed from index 5 to 6, 9564311af29a4493831a8db658a085c4 (127.0.99.138) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } }
I20240503 02:00:30.147186 17487 raft_consensus.cc:2932] T 5d2104698d5f452c8350cce10afd3898 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9564311af29a4493831a8db658a085c4 (127.0.99.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } }
I20240503 02:00:30.149187 19108 raft_consensus.cc:1203] T 5833b4f9a5434734817f154d660aa9f7 P 838ef968bb45484bb27f5af6f3989bae [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 02:00:30.151520 17356 raft_consensus.cc:2932] T 5d2104698d5f452c8350cce10afd3898 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9564311af29a4493831a8db658a085c4 (127.0.99.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } }
I20240503 02:00:30.153088 17092 raft_consensus.cc:2932] T 5d2104698d5f452c8350cce10afd3898 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9564311af29a4493831a8db658a085c4 (127.0.99.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } }
I20240503 02:00:30.156117 17222 raft_consensus.cc:2932] T 5d2104698d5f452c8350cce10afd3898 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9564311af29a4493831a8db658a085c4 (127.0.99.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } }
I20240503 02:00:30.157979 17750 consensus_queue.cc:237] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } }
I20240503 02:00:30.160082 19239 raft_consensus.cc:1261] T 5d2104698d5f452c8350cce10afd3898 P 9564311af29a4493831a8db658a085c4 [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:30.160182 17618 raft_consensus.cc:1261] T 5d2104698d5f452c8350cce10afd3898 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:30.160290 16957 raft_consensus.cc:1261] T 5d2104698d5f452c8350cce10afd3898 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:30.160528 20006 consensus_queue.cc:1035] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:30.160727 17356 raft_consensus.cc:1261] T 5d2104698d5f452c8350cce10afd3898 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:30.160743 20007 consensus_queue.cc:1035] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:30.160914 19636 consensus_queue.cc:1035] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:30.161129 17092 raft_consensus.cc:1261] T 5d2104698d5f452c8350cce10afd3898 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:30.161393 17222 raft_consensus.cc:1261] T 5d2104698d5f452c8350cce10afd3898 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:30.161490 19636 consensus_queue.cc:1035] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:30.161610 19648 consensus_queue.cc:1035] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:30.162194 20007 consensus_queue.cc:1035] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:30.163921 19648 raft_consensus.cc:2932] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 16d437fe629643948c95d628926d5ffa (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } }
I20240503 02:00:30.164835 19239 raft_consensus.cc:2932] T 5d2104698d5f452c8350cce10afd3898 P 9564311af29a4493831a8db658a085c4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 16d437fe629643948c95d628926d5ffa (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } }
I20240503 02:00:30.165083 17222 raft_consensus.cc:2932] T 5d2104698d5f452c8350cce10afd3898 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 16d437fe629643948c95d628926d5ffa (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } }
I20240503 02:00:30.165055 16958 raft_consensus.cc:2932] T 5d2104698d5f452c8350cce10afd3898 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 16d437fe629643948c95d628926d5ffa (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } }
I20240503 02:00:30.165717 16802 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5d2104698d5f452c8350cce10afd3898 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:00:30.167886 19590 raft_consensus.cc:1050] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa: attempting to promote NON_VOTER 9564311af29a4493831a8db658a085c4 to VOTER
I20240503 02:00:30.168277 16815 catalog_manager.cc:5484] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6 reported cstate change: config changed from index 6 to 7, VOTER 16d437fe629643948c95d628926d5ffa (127.0.99.133) evicted. New cstate: current_term: 1 leader_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:00:30.168501 19590 consensus_queue.cc:237] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } }
I20240503 02:00:30.169214 17618 raft_consensus.cc:2932] T 5d2104698d5f452c8350cce10afd3898 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 16d437fe629643948c95d628926d5ffa (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } }
I20240503 02:00:30.170811 19239 raft_consensus.cc:1261] T ab536115ac7a4ba193f718d421f4000d P 9564311af29a4493831a8db658a085c4 [term 1 LEARNER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:30.171027 17090 raft_consensus.cc:1261] T ab536115ac7a4ba193f718d421f4000d P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:30.171154 19590 consensus_queue.cc:1035] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:30.172601 17750 raft_consensus.cc:1261] T ab536115ac7a4ba193f718d421f4000d P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:30.172708 17487 raft_consensus.cc:1261] T ab536115ac7a4ba193f718d421f4000d P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:30.173143 17223 raft_consensus.cc:1261] T ab536115ac7a4ba193f718d421f4000d P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:30.173000 17356 raft_consensus.cc:2932] T 5d2104698d5f452c8350cce10afd3898 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 16d437fe629643948c95d628926d5ffa (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } }
I20240503 02:00:30.173261 17618 raft_consensus.cc:1261] T ab536115ac7a4ba193f718d421f4000d P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:30.173718 19894 consensus_queue.cc:1035] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:30.174046 19894 consensus_queue.cc:1035] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:30.173949 17092 raft_consensus.cc:2932] T 5d2104698d5f452c8350cce10afd3898 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 16d437fe629643948c95d628926d5ffa (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } }
I20240503 02:00:30.174345 19894 consensus_queue.cc:1035] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:30.176486 16958 raft_consensus.cc:1261] T ab536115ac7a4ba193f718d421f4000d P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:30.177775 17467 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d2104698d5f452c8350cce10afd3898 with delete_type TABLET_DATA_TOMBSTONED (TS 16d437fe629643948c95d628926d5ffa not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:33592
I20240503 02:00:30.177119 20018 consensus_queue.cc:1035] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240503 02:00:30.178629 19590 consensus_queue.cc:1035] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:30.180099 20017 consensus_queue.cc:1035] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:30.180577 20025 tablet_replica.cc:330] T 5d2104698d5f452c8350cce10afd3898 P 16d437fe629643948c95d628926d5ffa: stopping tablet replica
I20240503 02:00:30.181569 20025 raft_consensus.cc:2229] T 5d2104698d5f452c8350cce10afd3898 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:30.181917 20025 raft_consensus.cc:2258] T 5d2104698d5f452c8350cce10afd3898 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:30.181973 19894 raft_consensus.cc:2932] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9564311af29a4493831a8db658a085c4 (127.0.99.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } }
I20240503 02:00:30.182299 17750 raft_consensus.cc:2932] T ab536115ac7a4ba193f718d421f4000d P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9564311af29a4493831a8db658a085c4 (127.0.99.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } }
I20240503 02:00:30.184563 17487 raft_consensus.cc:2932] T ab536115ac7a4ba193f718d421f4000d P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9564311af29a4493831a8db658a085c4 (127.0.99.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } }
I20240503 02:00:30.184736 17092 raft_consensus.cc:2932] T ab536115ac7a4ba193f718d421f4000d P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9564311af29a4493831a8db658a085c4 (127.0.99.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } }
I20240503 02:00:30.185642 20025 ts_tablet_manager.cc:1875] T 5d2104698d5f452c8350cce10afd3898 P 16d437fe629643948c95d628926d5ffa: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:30.186391 19238 raft_consensus.cc:2932] T ab536115ac7a4ba193f718d421f4000d P 9564311af29a4493831a8db658a085c4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9564311af29a4493831a8db658a085c4 (127.0.99.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } }
I20240503 02:00:30.186941 16811 catalog_manager.cc:5484] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa reported cstate change: config changed from index 5 to 6, 9564311af29a4493831a8db658a085c4 (127.0.99.138) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d43ed6b8663947d99a72cd922cc323aa" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:00:30.188418 20025 ts_tablet_manager.cc:1888] T 5d2104698d5f452c8350cce10afd3898 P 16d437fe629643948c95d628926d5ffa: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 02:00:30.188511 20025 log.cc:1196] T 5d2104698d5f452c8350cce10afd3898 P 16d437fe629643948c95d628926d5ffa: Deleting WAL directory at /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/5d2104698d5f452c8350cce10afd3898
I20240503 02:00:30.188956 16958 raft_consensus.cc:2932] T ab536115ac7a4ba193f718d421f4000d P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9564311af29a4493831a8db658a085c4 (127.0.99.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } }
I20240503 02:00:30.189705 16802 catalog_manager.cc:4830] TS 16d437fe629643948c95d628926d5ffa (127.0.99.133:46581): tablet 5d2104698d5f452c8350cce10afd3898 (table rebalance_test_table_2 [id=9f4cafd5cdb041c1bbe1592f04370c0b]) successfully deleted
I20240503 02:00:30.189898 17618 raft_consensus.cc:2932] T ab536115ac7a4ba193f718d421f4000d P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9564311af29a4493831a8db658a085c4 (127.0.99.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } }
I20240503 02:00:30.194566 17222 raft_consensus.cc:2932] T ab536115ac7a4ba193f718d421f4000d P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9564311af29a4493831a8db658a085c4 (127.0.99.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } }
I20240503 02:00:30.197525 17356 consensus_queue.cc:237] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } }
I20240503 02:00:30.199155 17222 raft_consensus.cc:1261] T ab536115ac7a4ba193f718d421f4000d P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:30.199177 19238 raft_consensus.cc:1261] T ab536115ac7a4ba193f718d421f4000d P 9564311af29a4493831a8db658a085c4 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:30.199260 17750 raft_consensus.cc:1261] T ab536115ac7a4ba193f718d421f4000d P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:30.199666 20021 consensus_queue.cc:1035] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:30.199923 17618 raft_consensus.cc:1261] T ab536115ac7a4ba193f718d421f4000d P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:30.200104 17092 raft_consensus.cc:1261] T ab536115ac7a4ba193f718d421f4000d P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:30.200103 16958 raft_consensus.cc:1261] T ab536115ac7a4ba193f718d421f4000d P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:30.200484 20019 consensus_queue.cc:1035] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:30.200742 19895 consensus_queue.cc:1035] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:30.200963 19590 consensus_queue.cc:1035] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:30.201139 19894 consensus_queue.cc:1035] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:30.201891 20017 consensus_queue.cc:1035] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:30.203933 19895 raft_consensus.cc:2932] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 16d437fe629643948c95d628926d5ffa (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } }
I20240503 02:00:30.205809 19238 raft_consensus.cc:2932] T ab536115ac7a4ba193f718d421f4000d P 9564311af29a4493831a8db658a085c4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 16d437fe629643948c95d628926d5ffa (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } }
I20240503 02:00:30.206071 16803 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ab536115ac7a4ba193f718d421f4000d with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:00:30.206617 16958 raft_consensus.cc:2932] T ab536115ac7a4ba193f718d421f4000d P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 16d437fe629643948c95d628926d5ffa (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } }
I20240503 02:00:30.207080 17750 raft_consensus.cc:2932] T ab536115ac7a4ba193f718d421f4000d P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 16d437fe629643948c95d628926d5ffa (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } }
I20240503 02:00:30.209336 16817 catalog_manager.cc:5484] T ab536115ac7a4ba193f718d421f4000d P 9564311af29a4493831a8db658a085c4 reported cstate change: config changed from index 6 to 7, VOTER 16d437fe629643948c95d628926d5ffa (127.0.99.133) evicted. New cstate: current_term: 1 leader_uuid: "d43ed6b8663947d99a72cd922cc323aa" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } }
I20240503 02:00:30.209683 17092 raft_consensus.cc:2932] T ab536115ac7a4ba193f718d421f4000d P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 16d437fe629643948c95d628926d5ffa (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } }
I20240503 02:00:30.210378 17222 raft_consensus.cc:2932] T ab536115ac7a4ba193f718d421f4000d P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 16d437fe629643948c95d628926d5ffa (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } }
I20240503 02:00:30.212881 17618 raft_consensus.cc:2932] T ab536115ac7a4ba193f718d421f4000d P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 16d437fe629643948c95d628926d5ffa (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } }
I20240503 02:00:30.214067 17467 tablet_service.cc:1514] Processing DeleteTablet for tablet ab536115ac7a4ba193f718d421f4000d with delete_type TABLET_DATA_TOMBSTONED (TS 16d437fe629643948c95d628926d5ffa not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:33592
I20240503 02:00:30.215003 20025 tablet_replica.cc:330] T ab536115ac7a4ba193f718d421f4000d P 16d437fe629643948c95d628926d5ffa: stopping tablet replica
I20240503 02:00:30.215273 20025 raft_consensus.cc:2229] T ab536115ac7a4ba193f718d421f4000d P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:30.215602 20025 raft_consensus.cc:2258] T ab536115ac7a4ba193f718d421f4000d P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:30.216846 20025 ts_tablet_manager.cc:1875] T ab536115ac7a4ba193f718d421f4000d P 16d437fe629643948c95d628926d5ffa: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:30.218832 20025 ts_tablet_manager.cc:1888] T ab536115ac7a4ba193f718d421f4000d P 16d437fe629643948c95d628926d5ffa: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 02:00:30.218904 20025 log.cc:1196] T ab536115ac7a4ba193f718d421f4000d P 16d437fe629643948c95d628926d5ffa: Deleting WAL directory at /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/ab536115ac7a4ba193f718d421f4000d
I20240503 02:00:30.219398 16802 catalog_manager.cc:4830] TS 16d437fe629643948c95d628926d5ffa (127.0.99.133:46581): tablet ab536115ac7a4ba193f718d421f4000d (table rebalance_test_table_6 [id=fc19f05fb43b48068855575b19b79556]) successfully deleted
I20240503 02:00:30.246853 20017 raft_consensus.cc:1050] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa: attempting to promote NON_VOTER 9564311af29a4493831a8db658a085c4 to VOTER
I20240503 02:00:30.247520 20017 consensus_queue.cc:237] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } }
I20240503 02:00:30.249356 19238 raft_consensus.cc:1261] T 63962b169caf4f14888871e246c00075 P 9564311af29a4493831a8db658a085c4 [term 1 LEARNER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:30.249524 17222 raft_consensus.cc:1261] T 63962b169caf4f14888871e246c00075 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:30.249536 17487 raft_consensus.cc:1261] T 63962b169caf4f14888871e246c00075 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:30.249621 17750 raft_consensus.cc:1261] T 63962b169caf4f14888871e246c00075 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:30.250039 19894 consensus_queue.cc:1035] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:30.250250 20019 consensus_queue.cc:1035] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:30.250366 20021 consensus_queue.cc:1035] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:30.251510 17092 raft_consensus.cc:1261] T 63962b169caf4f14888871e246c00075 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:30.251498 16958 raft_consensus.cc:1261] T 63962b169caf4f14888871e246c00075 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:30.251899 19894 consensus_queue.cc:1035] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:30.252089 19895 consensus_queue.cc:1035] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:30.252357 20017 consensus_queue.cc:1035] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:30.252841 17618 raft_consensus.cc:1261] T 63962b169caf4f14888871e246c00075 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:30.254123 19895 consensus_queue.cc:1035] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:30.254951 19894 raft_consensus.cc:2932] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9564311af29a4493831a8db658a085c4 (127.0.99.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } }
I20240503 02:00:30.256134 17750 raft_consensus.cc:2932] T 63962b169caf4f14888871e246c00075 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9564311af29a4493831a8db658a085c4 (127.0.99.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } }
I20240503 02:00:30.256803 16958 raft_consensus.cc:2932] T 63962b169caf4f14888871e246c00075 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9564311af29a4493831a8db658a085c4 (127.0.99.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } }
I20240503 02:00:30.256990 17618 raft_consensus.cc:2932] T 63962b169caf4f14888871e246c00075 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9564311af29a4493831a8db658a085c4 (127.0.99.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } }
I20240503 02:00:30.258986 17222 raft_consensus.cc:2932] T 63962b169caf4f14888871e246c00075 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9564311af29a4493831a8db658a085c4 (127.0.99.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } }
I20240503 02:00:30.259003 16812 catalog_manager.cc:5484] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa reported cstate change: config changed from index 5 to 6, 9564311af29a4493831a8db658a085c4 (127.0.99.138) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d43ed6b8663947d99a72cd922cc323aa" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:00:30.260241 17487 raft_consensus.cc:2932] T 63962b169caf4f14888871e246c00075 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9564311af29a4493831a8db658a085c4 (127.0.99.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } }
I20240503 02:00:30.264325 19238 raft_consensus.cc:2932] T 63962b169caf4f14888871e246c00075 P 9564311af29a4493831a8db658a085c4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9564311af29a4493831a8db658a085c4 (127.0.99.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } }
I20240503 02:00:30.265892 17092 raft_consensus.cc:2932] T 63962b169caf4f14888871e246c00075 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9564311af29a4493831a8db658a085c4 (127.0.99.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } }
I20240503 02:00:30.268983 17356 consensus_queue.cc:237] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } }
I20240503 02:00:30.270829 16958 raft_consensus.cc:1261] T 63962b169caf4f14888871e246c00075 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:30.270823 17222 raft_consensus.cc:1261] T 63962b169caf4f14888871e246c00075 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:30.270848 17750 raft_consensus.cc:1261] T 63962b169caf4f14888871e246c00075 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:30.271209 20017 consensus_queue.cc:1035] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:30.271363 19238 raft_consensus.cc:1261] T 63962b169caf4f14888871e246c00075 P 9564311af29a4493831a8db658a085c4 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:30.271428 19895 consensus_queue.cc:1035] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:30.271492 17092 raft_consensus.cc:1261] T 63962b169caf4f14888871e246c00075 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:30.271701 19590 consensus_queue.cc:1035] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:30.271836 19895 consensus_queue.cc:1035] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:30.271975 20018 consensus_queue.cc:1035] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:30.273634 17487 raft_consensus.cc:1261] T 63962b169caf4f14888871e246c00075 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:30.273819 19947 raft_consensus.cc:1050] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa: attempting to promote NON_VOTER 9564311af29a4493831a8db658a085c4 to VOTER
I20240503 02:00:30.274262 19895 consensus_queue.cc:1035] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:30.274411 19947 consensus_queue.cc:237] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } }
I20240503 02:00:30.275689 20020 raft_consensus.cc:2932] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } }
I20240503 02:00:30.276624 17092 raft_consensus.cc:1261] T fbec07350ad04c49afd42e5d88dda6ad P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:30.276878 17356 raft_consensus.cc:1261] T fbec07350ad04c49afd42e5d88dda6ad P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:30.277406 17618 raft_consensus.cc:1261] T fbec07350ad04c49afd42e5d88dda6ad P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:30.277544 17750 raft_consensus.cc:1261] T fbec07350ad04c49afd42e5d88dda6ad P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:30.277707 19875 consensus_queue.cc:1035] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:30.278367 19620 consensus_queue.cc:1035] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:30.278504 16958 raft_consensus.cc:1261] T fbec07350ad04c49afd42e5d88dda6ad P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:30.279206 17223 raft_consensus.cc:1261] T fbec07350ad04c49afd42e5d88dda6ad P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:30.279141 17222 raft_consensus.cc:2932] T 63962b169caf4f14888871e246c00075 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } }
I20240503 02:00:30.279678 19620 consensus_queue.cc:1035] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:30.279851 19875 consensus_queue.cc:1035] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:30.280326 19875 consensus_queue.cc:1035] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:30.280493 19567 consensus_queue.cc:1035] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:30.280759 16803 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 63962b169caf4f14888871e246c00075 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:00:30.281425 17750 raft_consensus.cc:2932] T 63962b169caf4f14888871e246c00075 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } }
I20240503 02:00:30.281599 17091 raft_consensus.cc:2932] T 63962b169caf4f14888871e246c00075 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } }
I20240503 02:00:30.282068 16811 catalog_manager.cc:5484] T 63962b169caf4f14888871e246c00075 P 5b8291b26a7f448b8636187d1caf86d0 reported cstate change: config changed from index 6 to 7, VOTER c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134) evicted. New cstate: current_term: 1 leader_uuid: "d43ed6b8663947d99a72cd922cc323aa" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } }
I20240503 02:00:30.284263 16957 raft_consensus.cc:2932] T 63962b169caf4f14888871e246c00075 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } }
I20240503 02:00:30.284621 19239 raft_consensus.cc:1261] T fbec07350ad04c49afd42e5d88dda6ad P 9564311af29a4493831a8db658a085c4 [term 1 LEARNER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:30.288254 17598 tablet_service.cc:1514] Processing DeleteTablet for tablet 63962b169caf4f14888871e246c00075 with delete_type TABLET_DATA_TOMBSTONED (TS c843193b801b4f1a88f630ed3ed0f375 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:51030
I20240503 02:00:30.290582 19567 consensus_queue.cc:1035] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:30.293547 17487 raft_consensus.cc:2932] T 63962b169caf4f14888871e246c00075 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } }
I20240503 02:00:30.294433 19875 raft_consensus.cc:2932] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9564311af29a4493831a8db658a085c4 (127.0.99.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } }
I20240503 02:00:30.296590 19239 raft_consensus.cc:2932] T fbec07350ad04c49afd42e5d88dda6ad P 9564311af29a4493831a8db658a085c4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9564311af29a4493831a8db658a085c4 (127.0.99.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } }
I20240503 02:00:30.296775 17356 raft_consensus.cc:2932] T fbec07350ad04c49afd42e5d88dda6ad P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9564311af29a4493831a8db658a085c4 (127.0.99.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } }
I20240503 02:00:30.298478 17222 raft_consensus.cc:2932] T fbec07350ad04c49afd42e5d88dda6ad P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9564311af29a4493831a8db658a085c4 (127.0.99.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } }
I20240503 02:00:30.298556 16958 raft_consensus.cc:2932] T fbec07350ad04c49afd42e5d88dda6ad P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9564311af29a4493831a8db658a085c4 (127.0.99.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } }
I20240503 02:00:30.301457 19238 raft_consensus.cc:2932] T 63962b169caf4f14888871e246c00075 P 9564311af29a4493831a8db658a085c4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } }
I20240503 02:00:30.301649 16818 catalog_manager.cc:5484] T fbec07350ad04c49afd42e5d88dda6ad P 6398c8993c3c45dcaab25b42142d5671 reported cstate change: config changed from index 5 to 6, 9564311af29a4493831a8db658a085c4 (127.0.99.138) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "16d437fe629643948c95d628926d5ffa" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } }
I20240503 02:00:30.302086 17750 raft_consensus.cc:2932] T fbec07350ad04c49afd42e5d88dda6ad P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9564311af29a4493831a8db658a085c4 (127.0.99.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } }
I20240503 02:00:30.310338 17090 raft_consensus.cc:2932] T fbec07350ad04c49afd42e5d88dda6ad P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9564311af29a4493831a8db658a085c4 (127.0.99.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } }
I20240503 02:00:30.310880 17618 raft_consensus.cc:2932] T fbec07350ad04c49afd42e5d88dda6ad P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9564311af29a4493831a8db658a085c4 (127.0.99.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } }
I20240503 02:00:30.315308 19691 raft_consensus.cc:1050] T ae4259dd3dc745869256269632886f8a P 89dd09f8aaee4b249ae3ce924a422a67: attempting to promote NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d to VOTER
I20240503 02:00:30.315657 17487 consensus_queue.cc:237] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } }
I20240503 02:00:30.315903 19691 consensus_queue.cc:237] T ae4259dd3dc745869256269632886f8a P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } }
I20240503 02:00:30.315357 20045 tablet_replica.cc:330] T 63962b169caf4f14888871e246c00075 P c843193b801b4f1a88f630ed3ed0f375: stopping tablet replica
I20240503 02:00:30.316519 20045 raft_consensus.cc:2229] T 63962b169caf4f14888871e246c00075 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:30.316808 20045 raft_consensus.cc:2258] T 63962b169caf4f14888871e246c00075 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:30.318316 17356 raft_consensus.cc:1261] T fbec07350ad04c49afd42e5d88dda6ad P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:30.318661 16958 raft_consensus.cc:1261] T ae4259dd3dc745869256269632886f8a P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:30.318924 20045 ts_tablet_manager.cc:1875] T 63962b169caf4f14888871e246c00075 P c843193b801b4f1a88f630ed3ed0f375: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:30.319017 19370 raft_consensus.cc:1261] T ae4259dd3dc745869256269632886f8a P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 LEARNER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:30.319077 19896 consensus_queue.cc:1035] T ae4259dd3dc745869256269632886f8a P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:30.319116 17223 raft_consensus.cc:1261] T ae4259dd3dc745869256269632886f8a P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:30.319694 19896 consensus_queue.cc:1035] T ae4259dd3dc745869256269632886f8a P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:30.320005 19949 consensus_queue.cc:1035] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:30.321131 17751 raft_consensus.cc:1261] T ae4259dd3dc745869256269632886f8a P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:30.321893 19676 consensus_queue.cc:1035] T ae4259dd3dc745869256269632886f8a P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:30.322161 19691 consensus_queue.cc:1035] T ae4259dd3dc745869256269632886f8a P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:30.322877 17356 raft_consensus.cc:1261] T ae4259dd3dc745869256269632886f8a P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:30.323623 16958 raft_consensus.cc:1261] T fbec07350ad04c49afd42e5d88dda6ad P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:30.323864 17222 raft_consensus.cc:1261] T fbec07350ad04c49afd42e5d88dda6ad P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:30.324317 19238 raft_consensus.cc:1261] T fbec07350ad04c49afd42e5d88dda6ad P 9564311af29a4493831a8db658a085c4 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:30.319890 17618 raft_consensus.cc:1261] T ae4259dd3dc745869256269632886f8a P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:30.324609 19949 consensus_queue.cc:1035] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:30.324906 19896 consensus_queue.cc:1035] T ae4259dd3dc745869256269632886f8a P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:30.325071 19676 consensus_queue.cc:1035] T ae4259dd3dc745869256269632886f8a P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:30.325104 19949 consensus_queue.cc:1035] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:30.324628 17487 raft_consensus.cc:1261] T ae4259dd3dc745869256269632886f8a P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:30.325663 17750 raft_consensus.cc:1261] T fbec07350ad04c49afd42e5d88dda6ad P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:30.326027 19567 consensus_queue.cc:1035] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:30.327459 19948 consensus_queue.cc:1035] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:30.328135 20045 ts_tablet_manager.cc:1888] T 63962b169caf4f14888871e246c00075 P c843193b801b4f1a88f630ed3ed0f375: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 02:00:30.328423 17090 raft_consensus.cc:1261] T fbec07350ad04c49afd42e5d88dda6ad P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:30.328294 19875 raft_consensus.cc:2932] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } }
I20240503 02:00:30.329676 19567 consensus_queue.cc:1035] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:30.329993 17750 raft_consensus.cc:2932] T fbec07350ad04c49afd42e5d88dda6ad P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } }
I20240503 02:00:30.330724 20045 log.cc:1196] T 63962b169caf4f14888871e246c00075 P c843193b801b4f1a88f630ed3ed0f375: Deleting WAL directory at /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/63962b169caf4f14888871e246c00075
I20240503 02:00:30.330709 17090 raft_consensus.cc:2932] T fbec07350ad04c49afd42e5d88dda6ad P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } }
I20240503 02:00:30.331969 16802 catalog_manager.cc:4830] TS c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134:44343): tablet 63962b169caf4f14888871e246c00075 (table rebalance_test_table_3 [id=78777800a49e46109c1b3bb45ab99e28]) successfully deleted
I20240503 02:00:30.332500 19238 raft_consensus.cc:2932] T fbec07350ad04c49afd42e5d88dda6ad P 9564311af29a4493831a8db658a085c4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } }
I20240503 02:00:30.332813 16958 raft_consensus.cc:2932] T fbec07350ad04c49afd42e5d88dda6ad P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } }
I20240503 02:00:30.330027 20055 consensus_queue.cc:1035] T ae4259dd3dc745869256269632886f8a P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.003s
I20240503 02:00:30.335528 16815 catalog_manager.cc:5484] T fbec07350ad04c49afd42e5d88dda6ad P 9564311af29a4493831a8db658a085c4 reported cstate change: config changed from index 6 to 7, VOTER c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134) evicted. New cstate: current_term: 1 leader_uuid: "16d437fe629643948c95d628926d5ffa" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } }
I20240503 02:00:30.335908 16802 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet fbec07350ad04c49afd42e5d88dda6ad with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:00:30.336160 17224 raft_consensus.cc:2932] T fbec07350ad04c49afd42e5d88dda6ad P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } }
I20240503 02:00:30.336407 17356 raft_consensus.cc:2932] T fbec07350ad04c49afd42e5d88dda6ad P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } }
I20240503 02:00:30.338518 17224 raft_consensus.cc:2932] T ae4259dd3dc745869256269632886f8a P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:30.338615 17618 raft_consensus.cc:2932] T ae4259dd3dc745869256269632886f8a P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:30.340958 17598 tablet_service.cc:1514] Processing DeleteTablet for tablet fbec07350ad04c49afd42e5d88dda6ad with delete_type TABLET_DATA_TOMBSTONED (TS c843193b801b4f1a88f630ed3ed0f375 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:51030
I20240503 02:00:30.342043 20045 tablet_replica.cc:330] T fbec07350ad04c49afd42e5d88dda6ad P c843193b801b4f1a88f630ed3ed0f375: stopping tablet replica
I20240503 02:00:30.342319 20045 raft_consensus.cc:2229] T fbec07350ad04c49afd42e5d88dda6ad P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:30.342674 20045 raft_consensus.cc:2258] T fbec07350ad04c49afd42e5d88dda6ad P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:30.342556 16811 catalog_manager.cc:5484] T ae4259dd3dc745869256269632886f8a P c843193b801b4f1a88f630ed3ed0f375 reported cstate change: config changed from index 5 to 6, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "89dd09f8aaee4b249ae3ce924a422a67" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:30.344878 19567 raft_consensus.cc:1050] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa: attempting to promote NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d to VOTER
I20240503 02:00:30.345480 19567 consensus_queue.cc:237] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } }
I20240503 02:00:30.345996 20045 ts_tablet_manager.cc:1875] T fbec07350ad04c49afd42e5d88dda6ad P c843193b801b4f1a88f630ed3ed0f375: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:30.347646 16957 raft_consensus.cc:1261] T acfd773c1f384c219d776878407d1144 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:30.348088 19948 consensus_queue.cc:1035] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:30.348667 20045 ts_tablet_manager.cc:1888] T fbec07350ad04c49afd42e5d88dda6ad P c843193b801b4f1a88f630ed3ed0f375: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 02:00:30.348721 17090 raft_consensus.cc:1261] T acfd773c1f384c219d776878407d1144 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:30.348831 20045 log.cc:1196] T fbec07350ad04c49afd42e5d88dda6ad P c843193b801b4f1a88f630ed3ed0f375: Deleting WAL directory at /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/fbec07350ad04c49afd42e5d88dda6ad
I20240503 02:00:30.349246 17618 raft_consensus.cc:1261] T acfd773c1f384c219d776878407d1144 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:30.349347 19370 raft_consensus.cc:1261] T acfd773c1f384c219d776878407d1144 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 LEARNER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:30.349599 17356 raft_consensus.cc:1261] T acfd773c1f384c219d776878407d1144 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:30.349694 17750 raft_consensus.cc:1261] T acfd773c1f384c219d776878407d1144 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:30.350014 19949 consensus_queue.cc:1035] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:30.350216 20040 consensus_queue.cc:1035] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:30.350351 19875 consensus_queue.cc:1035] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:30.350646 16802 catalog_manager.cc:4830] TS c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134:44343): tablet fbec07350ad04c49afd42e5d88dda6ad (table rebalance_test_table_0 [id=ee1e0b20af0b4ecea04ae147701a0482]) successfully deleted
I20240503 02:00:30.351858 20040 consensus_queue.cc:1035] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:30.352020 20068 consensus_queue.cc:1035] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:30.352125 20059 raft_consensus.cc:2932] T ae4259dd3dc745869256269632886f8a P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:30.353345 17224 raft_consensus.cc:1261] T acfd773c1f384c219d776878407d1144 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:30.353695 19949 consensus_queue.cc:1035] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:30.355487 16957 raft_consensus.cc:2932] T ae4259dd3dc745869256269632886f8a P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:30.356487 17353 raft_consensus.cc:2932] T ae4259dd3dc745869256269632886f8a P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:30.357071 19369 raft_consensus.cc:2932] T ae4259dd3dc745869256269632886f8a P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:30.361090 19949 raft_consensus.cc:2932] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:30.363312 17356 raft_consensus.cc:2932] T acfd773c1f384c219d776878407d1144 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:30.363312 17751 raft_consensus.cc:2932] T acfd773c1f384c219d776878407d1144 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:30.363370 19369 raft_consensus.cc:2932] T acfd773c1f384c219d776878407d1144 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:30.363350 17618 raft_consensus.cc:2932] T acfd773c1f384c219d776878407d1144 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:30.366499 16815 catalog_manager.cc:5484] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa reported cstate change: config changed from index 5 to 6, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "16d437fe629643948c95d628926d5ffa" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:00:30.367446 17751 raft_consensus.cc:2932] T ae4259dd3dc745869256269632886f8a P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:30.368274 16957 raft_consensus.cc:2932] T acfd773c1f384c219d776878407d1144 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:30.372905 17224 raft_consensus.cc:2932] T acfd773c1f384c219d776878407d1144 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:30.374068 17091 consensus_queue.cc:237] T ae4259dd3dc745869256269632886f8a P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } }
I20240503 02:00:30.376242 17751 raft_consensus.cc:1261] T ae4259dd3dc745869256269632886f8a P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:30.376817 20056 consensus_queue.cc:1035] T ae4259dd3dc745869256269632886f8a P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:30.376910 17224 raft_consensus.cc:1261] T ae4259dd3dc745869256269632886f8a P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:30.377287 19369 raft_consensus.cc:1261] T ae4259dd3dc745869256269632886f8a P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:30.377362 20056 consensus_queue.cc:1035] T ae4259dd3dc745869256269632886f8a P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:30.377817 17356 raft_consensus.cc:1261] T ae4259dd3dc745869256269632886f8a P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:30.378253 19676 consensus_queue.cc:1035] T ae4259dd3dc745869256269632886f8a P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:30.378563 19676 consensus_queue.cc:1035] T ae4259dd3dc745869256269632886f8a P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:30.379359 20007 raft_consensus.cc:1050] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6: attempting to promote NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d to VOTER
I20240503 02:00:30.379704 16957 raft_consensus.cc:1261] T ae4259dd3dc745869256269632886f8a P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:30.379974 20007 consensus_queue.cc:237] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } }
I20240503 02:00:30.380147 20057 consensus_queue.cc:1035] T ae4259dd3dc745869256269632886f8a P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:30.382035 17090 raft_consensus.cc:1261] T fdf0a4b9e17642849a4094d2f0b9a255 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:30.382267 17223 raft_consensus.cc:1261] T fdf0a4b9e17642849a4094d2f0b9a255 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:30.382459 19369 raft_consensus.cc:1261] T fdf0a4b9e17642849a4094d2f0b9a255 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 LEARNER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:30.382594 20006 consensus_queue.cc:1035] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:30.382963 20006 consensus_queue.cc:1035] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:30.383128 20004 consensus_queue.cc:1035] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:30.384542 17092 raft_consensus.cc:2932] T acfd773c1f384c219d776878407d1144 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:30.387984 19691 raft_consensus.cc:2932] T ae4259dd3dc745869256269632886f8a P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 16d437fe629643948c95d628926d5ffa (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:30.389317 17356 raft_consensus.cc:2932] T ae4259dd3dc745869256269632886f8a P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 16d437fe629643948c95d628926d5ffa (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:30.390038 16805 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ae4259dd3dc745869256269632886f8a with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:00:30.390250 17751 raft_consensus.cc:2932] T ae4259dd3dc745869256269632886f8a P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 16d437fe629643948c95d628926d5ffa (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:30.390504 19370 raft_consensus.cc:2932] T ae4259dd3dc745869256269632886f8a P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 16d437fe629643948c95d628926d5ffa (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:30.392431 17618 raft_consensus.cc:1261] T ae4259dd3dc745869256269632886f8a P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:30.392266 16812 catalog_manager.cc:5484] T ae4259dd3dc745869256269632886f8a P d43ed6b8663947d99a72cd922cc323aa reported cstate change: config changed from index 6 to 7, VOTER 16d437fe629643948c95d628926d5ffa (127.0.99.133) evicted. New cstate: current_term: 1 leader_uuid: "89dd09f8aaee4b249ae3ce924a422a67" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:30.393154 20058 consensus_queue.cc:1035] T ae4259dd3dc745869256269632886f8a P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:30.394218 17618 raft_consensus.cc:2932] T ae4259dd3dc745869256269632886f8a P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 16d437fe629643948c95d628926d5ffa (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:30.395877 17487 raft_consensus.cc:1261] T fdf0a4b9e17642849a4094d2f0b9a255 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:30.395953 17619 raft_consensus.cc:1261] T fdf0a4b9e17642849a4094d2f0b9a255 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:30.396399 16958 raft_consensus.cc:1261] T fdf0a4b9e17642849a4094d2f0b9a255 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:30.396847 17356 raft_consensus.cc:1261] T fdf0a4b9e17642849a4094d2f0b9a255 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:30.397338 20004 consensus_queue.cc:1035] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:30.397505 19636 consensus_queue.cc:1035] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:30.397830 19636 consensus_queue.cc:1035] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:30.398185 19636 consensus_queue.cc:1035] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:30.398389 17487 consensus_queue.cc:237] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } }
I20240503 02:00:30.400534 17222 raft_consensus.cc:1261] T acfd773c1f384c219d776878407d1144 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:30.400599 17750 raft_consensus.cc:1261] T acfd773c1f384c219d776878407d1144 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:30.400949 17619 raft_consensus.cc:1261] T acfd773c1f384c219d776878407d1144 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:30.401290 19567 consensus_queue.cc:1035] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:30.401403 19370 raft_consensus.cc:1261] T acfd773c1f384c219d776878407d1144 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:30.401675 19567 consensus_queue.cc:1035] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:30.401818 20040 consensus_queue.cc:1035] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:30.402091 19948 consensus_queue.cc:1035] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:30.402678 16959 raft_consensus.cc:1261] T acfd773c1f384c219d776878407d1144 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:30.403049 17467 tablet_service.cc:1514] Processing DeleteTablet for tablet ae4259dd3dc745869256269632886f8a with delete_type TABLET_DATA_TOMBSTONED (TS 16d437fe629643948c95d628926d5ffa not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:33592
I20240503 02:00:30.403488 19567 consensus_queue.cc:1035] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:30.403574 20025 tablet_replica.cc:330] T ae4259dd3dc745869256269632886f8a P 16d437fe629643948c95d628926d5ffa: stopping tablet replica
I20240503 02:00:30.403702 20025 raft_consensus.cc:2229] T ae4259dd3dc745869256269632886f8a P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:30.403846 20025 pending_rounds.cc:62] T ae4259dd3dc745869256269632886f8a P 16d437fe629643948c95d628926d5ffa: Trying to abort 1 pending ops.
I20240503 02:00:30.403918 20025 pending_rounds.cc:68] T ae4259dd3dc745869256269632886f8a P 16d437fe629643948c95d628926d5ffa: Aborting op as it isn't in flight: id { term: 1 index: 6 } timestamp: 7023417877772034048 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "ae4259dd3dc745869256269632886f8a" old_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } } new_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } } }
I20240503 02:00:30.404115 20025 raft_consensus.cc:2894] T ae4259dd3dc745869256269632886f8a P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Aborting config change with OpId 1.6: Aborted: Op aborted
I20240503 02:00:30.404191 20025 raft_consensus.cc:2866] T ae4259dd3dc745869256269632886f8a P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240503 02:00:30.404242 20025 raft_consensus.cc:2258] T ae4259dd3dc745869256269632886f8a P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:30.405617 20025 ts_tablet_manager.cc:1875] T ae4259dd3dc745869256269632886f8a P 16d437fe629643948c95d628926d5ffa: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:30.406126 16957 raft_consensus.cc:2932] T ae4259dd3dc745869256269632886f8a P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 16d437fe629643948c95d628926d5ffa (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:30.408743 19948 raft_consensus.cc:2932] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:30.410480 17224 raft_consensus.cc:2932] T ae4259dd3dc745869256269632886f8a P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 16d437fe629643948c95d628926d5ffa (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:30.411226 16802 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet acfd773c1f384c219d776878407d1144 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:00:30.411362 17222 raft_consensus.cc:2932] T acfd773c1f384c219d776878407d1144 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:30.411892 17750 raft_consensus.cc:2932] T acfd773c1f384c219d776878407d1144 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:30.412768 17091 raft_consensus.cc:1261] T acfd773c1f384c219d776878407d1144 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:30.413331 19567 consensus_queue.cc:1035] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:30.413626 17619 raft_consensus.cc:2932] T acfd773c1f384c219d776878407d1144 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:30.414527 17091 raft_consensus.cc:2932] T acfd773c1f384c219d776878407d1144 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:30.414942 16815 catalog_manager.cc:5484] T acfd773c1f384c219d776878407d1144 P 002cc3411acb4fdfa0b67c6f57f687a6 reported cstate change: config changed from index 6 to 7, VOTER d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "16d437fe629643948c95d628926d5ffa" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:30.414856 20006 raft_consensus.cc:2932] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:30.416319 20025 ts_tablet_manager.cc:1888] T ae4259dd3dc745869256269632886f8a P 16d437fe629643948c95d628926d5ffa: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 02:00:30.416402 20025 log.cc:1196] T ae4259dd3dc745869256269632886f8a P 16d437fe629643948c95d628926d5ffa: Deleting WAL directory at /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/ae4259dd3dc745869256269632886f8a
I20240503 02:00:30.417522 17356 raft_consensus.cc:2932] T fdf0a4b9e17642849a4094d2f0b9a255 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:30.417729 16802 catalog_manager.cc:4830] TS 16d437fe629643948c95d628926d5ffa (127.0.99.133:46581): tablet ae4259dd3dc745869256269632886f8a (table rebalance_test_table_4 [id=421dd60555d242a78638e28180581a9c]) successfully deleted
I20240503 02:00:30.418025 16957 raft_consensus.cc:2932] T acfd773c1f384c219d776878407d1144 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:30.419409 16958 raft_consensus.cc:2932] T fdf0a4b9e17642849a4094d2f0b9a255 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:30.419832 17090 raft_consensus.cc:2932] T fdf0a4b9e17642849a4094d2f0b9a255 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:30.420702 16812 catalog_manager.cc:5484] T fdf0a4b9e17642849a4094d2f0b9a255 P d43ed6b8663947d99a72cd922cc323aa reported cstate change: config changed from index 5 to 6, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:30.422184 17335 tablet_service.cc:1514] Processing DeleteTablet for tablet acfd773c1f384c219d776878407d1144 with delete_type TABLET_DATA_TOMBSTONED (TS d43ed6b8663947d99a72cd922cc323aa not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:44362
I20240503 02:00:30.423538 17487 raft_consensus.cc:2932] T fdf0a4b9e17642849a4094d2f0b9a255 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:30.423938 19370 raft_consensus.cc:2932] T acfd773c1f384c219d776878407d1144 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:30.427783 17620 raft_consensus.cc:2932] T fdf0a4b9e17642849a4094d2f0b9a255 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:30.430625 20095 tablet_replica.cc:330] T acfd773c1f384c219d776878407d1144 P d43ed6b8663947d99a72cd922cc323aa: stopping tablet replica
I20240503 02:00:30.430780 19369 raft_consensus.cc:2932] T fdf0a4b9e17642849a4094d2f0b9a255 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:30.431015 20095 raft_consensus.cc:2229] T acfd773c1f384c219d776878407d1144 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:30.431725 20095 raft_consensus.cc:2258] T acfd773c1f384c219d776878407d1144 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:30.433934 20095 ts_tablet_manager.cc:1875] T acfd773c1f384c219d776878407d1144 P d43ed6b8663947d99a72cd922cc323aa: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:30.436529 20095 ts_tablet_manager.cc:1888] T acfd773c1f384c219d776878407d1144 P d43ed6b8663947d99a72cd922cc323aa: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 02:00:30.436636 20095 log.cc:1196] T acfd773c1f384c219d776878407d1144 P d43ed6b8663947d99a72cd922cc323aa: Deleting WAL directory at /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/acfd773c1f384c219d776878407d1144
I20240503 02:00:30.437345 16803 catalog_manager.cc:4830] TS d43ed6b8663947d99a72cd922cc323aa (127.0.99.132:40023): tablet acfd773c1f384c219d776878407d1144 (table rebalance_test_table_0 [id=ee1e0b20af0b4ecea04ae147701a0482]) successfully deleted
I20240503 02:00:30.438612 17223 raft_consensus.cc:2932] T fdf0a4b9e17642849a4094d2f0b9a255 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:30.442332 17751 consensus_queue.cc:237] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } }
I20240503 02:00:30.444339 19369 raft_consensus.cc:1261] T fdf0a4b9e17642849a4094d2f0b9a255 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:30.444381 17091 raft_consensus.cc:1261] T fdf0a4b9e17642849a4094d2f0b9a255 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:30.444873 19421 consensus_queue.cc:1035] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:30.445068 16957 raft_consensus.cc:1261] T fdf0a4b9e17642849a4094d2f0b9a255 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:30.445336 19421 consensus_queue.cc:1035] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:30.445631 20007 consensus_queue.cc:1035] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:30.447804 19421 raft_consensus.cc:2932] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:30.448741 16957 raft_consensus.cc:2932] T fdf0a4b9e17642849a4094d2f0b9a255 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:30.449363 17487 raft_consensus.cc:1261] T fdf0a4b9e17642849a4094d2f0b9a255 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:30.449928 17223 raft_consensus.cc:1261] T fdf0a4b9e17642849a4094d2f0b9a255 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:30.449959 19648 consensus_queue.cc:1035] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:30.450620 16802 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet fdf0a4b9e17642849a4094d2f0b9a255 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:00:30.452045 17487 raft_consensus.cc:2932] T fdf0a4b9e17642849a4094d2f0b9a255 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:30.452245 17620 raft_consensus.cc:1261] T fdf0a4b9e17642849a4094d2f0b9a255 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:30.452085 17091 raft_consensus.cc:2932] T fdf0a4b9e17642849a4094d2f0b9a255 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:30.452315 19369 raft_consensus.cc:2932] T fdf0a4b9e17642849a4094d2f0b9a255 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:30.455242 20005 consensus_queue.cc:1035] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:30.455502 16811 catalog_manager.cc:5484] T fdf0a4b9e17642849a4094d2f0b9a255 P 375d0f2ec11d4e25bd9d093bb6a5572d reported cstate change: config changed from index 6 to 7, VOTER d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:30.457633 20005 consensus_queue.cc:1035] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:30.458587 17223 raft_consensus.cc:2932] T fdf0a4b9e17642849a4094d2f0b9a255 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:30.460023 17619 raft_consensus.cc:2932] T fdf0a4b9e17642849a4094d2f0b9a255 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:30.460300 17335 tablet_service.cc:1514] Processing DeleteTablet for tablet fdf0a4b9e17642849a4094d2f0b9a255 with delete_type TABLET_DATA_TOMBSTONED (TS d43ed6b8663947d99a72cd922cc323aa not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:44362
I20240503 02:00:30.460572 20095 tablet_replica.cc:330] T fdf0a4b9e17642849a4094d2f0b9a255 P d43ed6b8663947d99a72cd922cc323aa: stopping tablet replica
I20240503 02:00:30.460781 20095 raft_consensus.cc:2229] T fdf0a4b9e17642849a4094d2f0b9a255 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:30.460989 20095 raft_consensus.cc:2258] T fdf0a4b9e17642849a4094d2f0b9a255 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:30.463162 20095 ts_tablet_manager.cc:1875] T fdf0a4b9e17642849a4094d2f0b9a255 P d43ed6b8663947d99a72cd922cc323aa: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:30.466130 20095 ts_tablet_manager.cc:1888] T fdf0a4b9e17642849a4094d2f0b9a255 P d43ed6b8663947d99a72cd922cc323aa: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 02:00:30.466270 20095 log.cc:1196] T fdf0a4b9e17642849a4094d2f0b9a255 P d43ed6b8663947d99a72cd922cc323aa: Deleting WAL directory at /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/fdf0a4b9e17642849a4094d2f0b9a255
I20240503 02:00:30.466861 16803 catalog_manager.cc:4830] TS d43ed6b8663947d99a72cd922cc323aa (127.0.99.132:40023): tablet fdf0a4b9e17642849a4094d2f0b9a255 (table rebalance_test_table_3 [id=78777800a49e46109c1b3bb45ab99e28]) successfully deleted
I20240503 02:00:30.486928 17487 consensus_queue.cc:237] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } }
I20240503 02:00:30.489327 17751 raft_consensus.cc:1261] T d52afe1281064d6480427554ca4bb2f6 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:30.489502 16957 raft_consensus.cc:1261] T d52afe1281064d6480427554ca4bb2f6 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:30.489755 17223 raft_consensus.cc:1261] T d52afe1281064d6480427554ca4bb2f6 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:30.490267 19238 raft_consensus.cc:1261] T d52afe1281064d6480427554ca4bb2f6 P 9564311af29a4493831a8db658a085c4 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:30.490694 19875 consensus_queue.cc:1035] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:30.491067 17619 raft_consensus.cc:1261] T d52afe1281064d6480427554ca4bb2f6 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:30.491081 19875 consensus_queue.cc:1035] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:30.491312 19953 consensus_queue.cc:1035] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:30.492967 19953 consensus_queue.cc:1035] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:30.493171 19875 consensus_queue.cc:1035] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:30.494735 17091 raft_consensus.cc:1261] T d52afe1281064d6480427554ca4bb2f6 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:30.495258 19567 consensus_queue.cc:1035] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:30.495760 20040 raft_consensus.cc:2932] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } }
I20240503 02:00:30.497359 17619 raft_consensus.cc:2932] T d52afe1281064d6480427554ca4bb2f6 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } }
I20240503 02:00:30.498117 16802 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d52afe1281064d6480427554ca4bb2f6 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:00:30.498795 16957 raft_consensus.cc:2932] T d52afe1281064d6480427554ca4bb2f6 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } }
I20240503 02:00:30.498797 17223 raft_consensus.cc:2932] T d52afe1281064d6480427554ca4bb2f6 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } }
I20240503 02:00:30.499442 17091 raft_consensus.cc:2932] T d52afe1281064d6480427554ca4bb2f6 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } }
I20240503 02:00:30.500562 16812 catalog_manager.cc:5484] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa reported cstate change: config changed from index 6 to 7, VOTER d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "16d437fe629643948c95d628926d5ffa" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:30.502317 17751 raft_consensus.cc:2932] T d52afe1281064d6480427554ca4bb2f6 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } }
I20240503 02:00:30.504307 17335 tablet_service.cc:1514] Processing DeleteTablet for tablet d52afe1281064d6480427554ca4bb2f6 with delete_type TABLET_DATA_TOMBSTONED (TS d43ed6b8663947d99a72cd922cc323aa not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:44362
I20240503 02:00:30.504559 20095 tablet_replica.cc:330] T d52afe1281064d6480427554ca4bb2f6 P d43ed6b8663947d99a72cd922cc323aa: stopping tablet replica
I20240503 02:00:30.504732 20095 raft_consensus.cc:2229] T d52afe1281064d6480427554ca4bb2f6 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:30.504904 20095 raft_consensus.cc:2258] T d52afe1281064d6480427554ca4bb2f6 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:30.505693 20095 ts_tablet_manager.cc:1875] T d52afe1281064d6480427554ca4bb2f6 P d43ed6b8663947d99a72cd922cc323aa: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:30.507865 20095 ts_tablet_manager.cc:1888] T d52afe1281064d6480427554ca4bb2f6 P d43ed6b8663947d99a72cd922cc323aa: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 02:00:30.507920 20095 log.cc:1196] T d52afe1281064d6480427554ca4bb2f6 P d43ed6b8663947d99a72cd922cc323aa: Deleting WAL directory at /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/d52afe1281064d6480427554ca4bb2f6
I20240503 02:00:30.507792 19238 raft_consensus.cc:2932] T d52afe1281064d6480427554ca4bb2f6 P 9564311af29a4493831a8db658a085c4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } }
I20240503 02:00:30.508853 16803 catalog_manager.cc:4830] TS d43ed6b8663947d99a72cd922cc323aa (127.0.99.132:40023): tablet d52afe1281064d6480427554ca4bb2f6 (table rebalance_test_table_3 [id=78777800a49e46109c1b3bb45ab99e28]) successfully deleted
I20240503 02:00:30.657939 20055 raft_consensus.cc:1050] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67: attempting to promote NON_VOTER 838ef968bb45484bb27f5af6f3989bae to VOTER
I20240503 02:00:30.658651 20055 consensus_queue.cc:237] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } }
I20240503 02:00:30.660830 17353 raft_consensus.cc:1261] T 5833b4f9a5434734817f154d660aa9f7 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:30.660884 19108 raft_consensus.cc:1261] T 5833b4f9a5434734817f154d660aa9f7 P 838ef968bb45484bb27f5af6f3989bae [term 1 LEARNER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:30.661553 17751 raft_consensus.cc:1261] T 5833b4f9a5434734817f154d660aa9f7 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:30.661648 20058 consensus_queue.cc:1035] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:30.661770 17222 raft_consensus.cc:1261] T 5833b4f9a5434734817f154d660aa9f7 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:30.662195 20058 consensus_queue.cc:1035] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:30.662243 17619 raft_consensus.cc:1261] T 5833b4f9a5434734817f154d660aa9f7 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:30.662334 19676 consensus_queue.cc:1035] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:30.662391 16957 raft_consensus.cc:1261] T 5833b4f9a5434734817f154d660aa9f7 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:30.662701 17487 raft_consensus.cc:1261] T 5833b4f9a5434734817f154d660aa9f7 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:30.662896 20057 consensus_queue.cc:1035] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:30.663041 19691 consensus_queue.cc:1035] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:30.663383 20055 consensus_queue.cc:1035] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:30.663520 19691 consensus_queue.cc:1035] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:30.667385 20058 raft_consensus.cc:2932] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:30.669075 17487 raft_consensus.cc:2932] T 5833b4f9a5434734817f154d660aa9f7 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:30.669116 17222 raft_consensus.cc:2932] T 5833b4f9a5434734817f154d660aa9f7 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:30.669175 17353 raft_consensus.cc:2932] T 5833b4f9a5434734817f154d660aa9f7 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:30.671015 16957 raft_consensus.cc:2932] T 5833b4f9a5434734817f154d660aa9f7 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:30.672240 17751 raft_consensus.cc:2932] T 5833b4f9a5434734817f154d660aa9f7 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:30.672637 16817 catalog_manager.cc:5484] T 5833b4f9a5434734817f154d660aa9f7 P 5b8291b26a7f448b8636187d1caf86d0 reported cstate change: config changed from index 5 to 6, 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "89dd09f8aaee4b249ae3ce924a422a67" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:30.673347 17619 raft_consensus.cc:2932] T 5833b4f9a5434734817f154d660aa9f7 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:30.674897 19108 raft_consensus.cc:2932] T 5833b4f9a5434734817f154d660aa9f7 P 838ef968bb45484bb27f5af6f3989bae [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:30.679708 17091 consensus_queue.cc:237] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } }
I20240503 02:00:30.681360 17751 raft_consensus.cc:1261] T 5833b4f9a5434734817f154d660aa9f7 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:30.681537 17487 raft_consensus.cc:1261] T 5833b4f9a5434734817f154d660aa9f7 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:30.682063 20057 consensus_queue.cc:1035] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:30.682163 19108 raft_consensus.cc:1261] T 5833b4f9a5434734817f154d660aa9f7 P 838ef968bb45484bb27f5af6f3989bae [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:30.682183 17222 raft_consensus.cc:1261] T 5833b4f9a5434734817f154d660aa9f7 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:30.682631 20057 consensus_queue.cc:1035] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:30.682973 19691 consensus_queue.cc:1035] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:30.683687 19896 consensus_queue.cc:1035] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:30.685655 17353 raft_consensus.cc:1261] T 5833b4f9a5434734817f154d660aa9f7 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:30.686192 19676 consensus_queue.cc:1035] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:30.686913 16957 raft_consensus.cc:1261] T 5833b4f9a5434734817f154d660aa9f7 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:30.686887 20057 raft_consensus.cc:2932] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:30.687407 17750 raft_consensus.cc:2932] T 5833b4f9a5434734817f154d660aa9f7 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:30.687650 20055 consensus_queue.cc:1035] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:30.690240 16805 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5833b4f9a5434734817f154d660aa9f7 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:00:30.690356 16812 catalog_manager.cc:5484] T 5833b4f9a5434734817f154d660aa9f7 P 002cc3411acb4fdfa0b67c6f57f687a6 reported cstate change: config changed from index 6 to 7, VOTER c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134) evicted. New cstate: current_term: 1 leader_uuid: "89dd09f8aaee4b249ae3ce924a422a67" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:30.690812 19108 raft_consensus.cc:2932] T 5833b4f9a5434734817f154d660aa9f7 P 838ef968bb45484bb27f5af6f3989bae [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:30.692078 16957 raft_consensus.cc:2932] T 5833b4f9a5434734817f154d660aa9f7 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:30.692207 17222 raft_consensus.cc:2932] T 5833b4f9a5434734817f154d660aa9f7 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:30.694599 17598 tablet_service.cc:1514] Processing DeleteTablet for tablet 5833b4f9a5434734817f154d660aa9f7 with delete_type TABLET_DATA_TOMBSTONED (TS c843193b801b4f1a88f630ed3ed0f375 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:51030
I20240503 02:00:30.694876 20045 tablet_replica.cc:330] T 5833b4f9a5434734817f154d660aa9f7 P c843193b801b4f1a88f630ed3ed0f375: stopping tablet replica
I20240503 02:00:30.695048 20045 raft_consensus.cc:2229] T 5833b4f9a5434734817f154d660aa9f7 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:30.695271 20045 raft_consensus.cc:2258] T 5833b4f9a5434734817f154d660aa9f7 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:30.696380 20045 ts_tablet_manager.cc:1875] T 5833b4f9a5434734817f154d660aa9f7 P c843193b801b4f1a88f630ed3ed0f375: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:30.697240 17353 raft_consensus.cc:2932] T 5833b4f9a5434734817f154d660aa9f7 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:30.699223 20045 ts_tablet_manager.cc:1888] T 5833b4f9a5434734817f154d660aa9f7 P c843193b801b4f1a88f630ed3ed0f375: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 02:00:30.699342 20045 log.cc:1196] T 5833b4f9a5434734817f154d660aa9f7 P c843193b801b4f1a88f630ed3ed0f375: Deleting WAL directory at /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/5833b4f9a5434734817f154d660aa9f7
I20240503 02:00:30.699643 17487 raft_consensus.cc:2932] T 5833b4f9a5434734817f154d660aa9f7 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:30.699924 16802 catalog_manager.cc:4830] TS c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134:44343): tablet 5833b4f9a5434734817f154d660aa9f7 (table rebalance_test_table_1 [id=38eeb06923e14f2ead405ffb039fd231]) successfully deleted
I20240503 02:00:30.726852 17353 consensus_queue.cc:237] T a39f284167e942188c181147e8a9e74e P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:30.728704 16957 raft_consensus.cc:1261] T a39f284167e942188c181147e8a9e74e P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:30.728724 17091 raft_consensus.cc:1261] T a39f284167e942188c181147e8a9e74e P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:30.728724 17751 raft_consensus.cc:1261] T a39f284167e942188c181147e8a9e74e P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:30.729188 19894 consensus_queue.cc:1035] T a39f284167e942188c181147e8a9e74e P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:30.729494 20017 consensus_queue.cc:1035] T a39f284167e942188c181147e8a9e74e P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:30.729763 17619 raft_consensus.cc:1261] T a39f284167e942188c181147e8a9e74e P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:30.730015 17222 raft_consensus.cc:1261] T a39f284167e942188c181147e8a9e74e P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:30.730122 17487 raft_consensus.cc:1261] T a39f284167e942188c181147e8a9e74e P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:30.730425 20017 consensus_queue.cc:1035] T a39f284167e942188c181147e8a9e74e P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:30.730576 19894 consensus_queue.cc:1035] T a39f284167e942188c181147e8a9e74e P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:30.730791 19895 consensus_queue.cc:1035] T a39f284167e942188c181147e8a9e74e P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:30.731559 20017 consensus_queue.cc:1035] T a39f284167e942188c181147e8a9e74e P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:30.733614 20018 raft_consensus.cc:2932] T a39f284167e942188c181147e8a9e74e P d43ed6b8663947d99a72cd922cc323aa [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } }
I20240503 02:00:30.734226 17751 raft_consensus.cc:2932] T a39f284167e942188c181147e8a9e74e P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } }
I20240503 02:00:30.734544 17487 raft_consensus.cc:2932] T a39f284167e942188c181147e8a9e74e P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } }
I20240503 02:00:30.736385 17619 raft_consensus.cc:2932] T a39f284167e942188c181147e8a9e74e P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } }
I20240503 02:00:30.737080 17091 raft_consensus.cc:2932] T a39f284167e942188c181147e8a9e74e P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } }
I20240503 02:00:30.737376 16815 catalog_manager.cc:5484] T a39f284167e942188c181147e8a9e74e P 002cc3411acb4fdfa0b67c6f57f687a6 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "d43ed6b8663947d99a72cd922cc323aa" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } }
I20240503 02:00:30.738301 16957 raft_consensus.cc:2932] T a39f284167e942188c181147e8a9e74e P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } }
I20240503 02:00:30.742745 17222 raft_consensus.cc:2932] T a39f284167e942188c181147e8a9e74e P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } }
I20240503 02:00:30.751782 17353 consensus_queue.cc:237] T a39f284167e942188c181147e8a9e74e P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } }
I20240503 02:00:30.754266 17619 raft_consensus.cc:1261] T a39f284167e942188c181147e8a9e74e P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:30.754755 20020 consensus_queue.cc:1035] T a39f284167e942188c181147e8a9e74e P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:30.755149 17487 raft_consensus.cc:1261] T a39f284167e942188c181147e8a9e74e P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:30.755203 17751 raft_consensus.cc:1261] T a39f284167e942188c181147e8a9e74e P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:30.755707 19894 consensus_queue.cc:1035] T a39f284167e942188c181147e8a9e74e P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:30.755782 16957 raft_consensus.cc:1261] T a39f284167e942188c181147e8a9e74e P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:30.755889 19895 consensus_queue.cc:1035] T a39f284167e942188c181147e8a9e74e P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:30.756551 19894 consensus_queue.cc:1035] T a39f284167e942188c181147e8a9e74e P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:30.756742 17222 raft_consensus.cc:1261] T a39f284167e942188c181147e8a9e74e P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:30.757069 19894 consensus_queue.cc:1035] T a39f284167e942188c181147e8a9e74e P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:30.758762 17091 raft_consensus.cc:1261] T a39f284167e942188c181147e8a9e74e P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:30.759260 19895 consensus_queue.cc:1035] T a39f284167e942188c181147e8a9e74e P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 02:00:30.761148 17290 consensus_peers.cc:485] T a39f284167e942188c181147e8a9e74e P d43ed6b8663947d99a72cd922cc323aa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a39f284167e942188c181147e8a9e74e. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:30.760995 20019 raft_consensus.cc:2932] T a39f284167e942188c181147e8a9e74e P d43ed6b8663947d99a72cd922cc323aa [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } }
I20240503 02:00:30.761693 17751 raft_consensus.cc:2932] T a39f284167e942188c181147e8a9e74e P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } }
I20240503 02:00:30.761694 17487 raft_consensus.cc:2932] T a39f284167e942188c181147e8a9e74e P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } }
I20240503 02:00:30.763051 17223 raft_consensus.cc:2932] T a39f284167e942188c181147e8a9e74e P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } }
I20240503 02:00:30.763973 17619 raft_consensus.cc:2932] T a39f284167e942188c181147e8a9e74e P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } }
I20240503 02:00:30.764333 16957 raft_consensus.cc:2932] T a39f284167e942188c181147e8a9e74e P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } }
I20240503 02:00:30.766474 16803 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet a39f284167e942188c181147e8a9e74e with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 02:00:30.767499 16811 catalog_manager.cc:5484] T a39f284167e942188c181147e8a9e74e P 6398c8993c3c45dcaab25b42142d5671 reported cstate change: config changed from index 4 to 5, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New cstate: current_term: 1 leader_uuid: "d43ed6b8663947d99a72cd922cc323aa" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } }
I20240503 02:00:30.768325 17091 raft_consensus.cc:2932] T a39f284167e942188c181147e8a9e74e P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } }
I20240503 02:00:30.777845 17487 consensus_queue.cc:237] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:30.779670 17223 raft_consensus.cc:1261] T 3f67591d6a3744ab90a3e4d2375a59db P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:30.780098 16957 raft_consensus.cc:1261] T 3f67591d6a3744ab90a3e4d2375a59db P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:30.780097 19949 consensus_queue.cc:1035] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:30.780274 17091 raft_consensus.cc:1261] T 3f67591d6a3744ab90a3e4d2375a59db P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:30.780417 17353 raft_consensus.cc:1261] T 3f67591d6a3744ab90a3e4d2375a59db P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:30.780555 19567 consensus_queue.cc:1035] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:30.780787 19949 consensus_queue.cc:1035] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:30.781010 19953 consensus_queue.cc:1035] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:30.782032 17619 raft_consensus.cc:1261] T 3f67591d6a3744ab90a3e4d2375a59db P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:30.783775 19949 consensus_queue.cc:1035] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:30.783938 17751 raft_consensus.cc:1261] T 3f67591d6a3744ab90a3e4d2375a59db P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:30.784570 19947 consensus_queue.cc:1035] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:30.784451 19567 raft_consensus.cc:2932] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } }
I20240503 02:00:30.785424 17353 raft_consensus.cc:2932] T 3f67591d6a3744ab90a3e4d2375a59db P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } }
I20240503 02:00:30.786130 17751 raft_consensus.cc:2932] T 3f67591d6a3744ab90a3e4d2375a59db P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } }
I20240503 02:00:30.787734 17223 raft_consensus.cc:2932] T 3f67591d6a3744ab90a3e4d2375a59db P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } }
I20240503 02:00:30.789618 16957 raft_consensus.cc:2932] T 3f67591d6a3744ab90a3e4d2375a59db P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } }
I20240503 02:00:30.789616 17091 raft_consensus.cc:2932] T 3f67591d6a3744ab90a3e4d2375a59db P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } }
I20240503 02:00:30.791612 16817 catalog_manager.cc:5484] T 3f67591d6a3744ab90a3e4d2375a59db P 002cc3411acb4fdfa0b67c6f57f687a6 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "16d437fe629643948c95d628926d5ffa" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } }
I20240503 02:00:30.796725 17619 raft_consensus.cc:2932] T 3f67591d6a3744ab90a3e4d2375a59db P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } }
I20240503 02:00:30.797246 17487 consensus_queue.cc:237] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } }
I20240503 02:00:30.801133 17353 raft_consensus.cc:1261] T 3f67591d6a3744ab90a3e4d2375a59db P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:30.801514 17091 raft_consensus.cc:1261] T 3f67591d6a3744ab90a3e4d2375a59db P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:30.802145 19875 consensus_queue.cc:1035] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:30.803014 16957 raft_consensus.cc:1261] T 3f67591d6a3744ab90a3e4d2375a59db P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:30.803388 19949 consensus_queue.cc:1035] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:30.803676 17223 raft_consensus.cc:1261] T 3f67591d6a3744ab90a3e4d2375a59db P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:30.803781 19947 consensus_queue.cc:1035] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:30.804806 20040 consensus_queue.cc:1035] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:30.805282 17751 raft_consensus.cc:1261] T 3f67591d6a3744ab90a3e4d2375a59db P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:30.805815 17619 raft_consensus.cc:1261] T 3f67591d6a3744ab90a3e4d2375a59db P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:30.806589 20040 raft_consensus.cc:2932] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } }
I20240503 02:00:30.806869 19948 consensus_queue.cc:1035] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:30.807133 19948 consensus_queue.cc:1035] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:30.808547 17751 raft_consensus.cc:2932] T 3f67591d6a3744ab90a3e4d2375a59db P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } }
I20240503 02:00:30.808588 17619 raft_consensus.cc:2932] T 3f67591d6a3744ab90a3e4d2375a59db P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } }
I20240503 02:00:30.808867 16802 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 3f67591d6a3744ab90a3e4d2375a59db with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 02:00:30.811334 16818 catalog_manager.cc:5484] T 3f67591d6a3744ab90a3e4d2375a59db P 002cc3411acb4fdfa0b67c6f57f687a6 reported cstate change: config changed from index 4 to 5, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New cstate: current_term: 1 leader_uuid: "16d437fe629643948c95d628926d5ffa" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } }
I20240503 02:00:30.812005 17223 raft_consensus.cc:2932] T 3f67591d6a3744ab90a3e4d2375a59db P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } }
I20240503 02:00:30.815577 16958 raft_consensus.cc:2932] T 3f67591d6a3744ab90a3e4d2375a59db P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } }
W20240503 02:00:30.818779 17422 consensus_peers.cc:485] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3f67591d6a3744ab90a3e4d2375a59db. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:30.818987 17353 raft_consensus.cc:2932] T 3f67591d6a3744ab90a3e4d2375a59db P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } }
I20240503 02:00:30.821370 17091 raft_consensus.cc:2932] T 3f67591d6a3744ab90a3e4d2375a59db P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } }
I20240503 02:00:30.823033 17090 consensus_queue.cc:237] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:30.825773 16957 raft_consensus.cc:1261] T 78f81fe4c6234ff0b5fcabd974779a66 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:30.825920 17223 raft_consensus.cc:1261] T 78f81fe4c6234ff0b5fcabd974779a66 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:30.826352 20057 consensus_queue.cc:1035] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:30.826504 17750 raft_consensus.cc:1261] T 78f81fe4c6234ff0b5fcabd974779a66 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:30.826527 19676 consensus_queue.cc:1035] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:30.826578 17353 raft_consensus.cc:1261] T 78f81fe4c6234ff0b5fcabd974779a66 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:30.826972 20057 consensus_queue.cc:1035] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:30.827216 17619 raft_consensus.cc:1261] T 78f81fe4c6234ff0b5fcabd974779a66 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:30.827416 20055 consensus_queue.cc:1035] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:30.827459 17487 raft_consensus.cc:1261] T 78f81fe4c6234ff0b5fcabd974779a66 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:30.827742 20057 consensus_queue.cc:1035] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:30.828040 20057 consensus_queue.cc:1035] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:30.831895 20055 raft_consensus.cc:2932] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } }
I20240503 02:00:30.832629 16957 raft_consensus.cc:2932] T 78f81fe4c6234ff0b5fcabd974779a66 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } }
I20240503 02:00:30.833828 17619 raft_consensus.cc:2932] T 78f81fe4c6234ff0b5fcabd974779a66 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } }
I20240503 02:00:30.834234 17223 raft_consensus.cc:2932] T 78f81fe4c6234ff0b5fcabd974779a66 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } }
I20240503 02:00:30.836524 16817 catalog_manager.cc:5484] T 78f81fe4c6234ff0b5fcabd974779a66 P 6398c8993c3c45dcaab25b42142d5671 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "89dd09f8aaee4b249ae3ce924a422a67" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } }
I20240503 02:00:30.836630 17487 raft_consensus.cc:2932] T 78f81fe4c6234ff0b5fcabd974779a66 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } }
I20240503 02:00:30.841068 17750 raft_consensus.cc:2932] T 78f81fe4c6234ff0b5fcabd974779a66 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } }
I20240503 02:00:30.844766 17353 raft_consensus.cc:2932] T 78f81fe4c6234ff0b5fcabd974779a66 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } }
I20240503 02:00:30.845121 17091 consensus_queue.cc:237] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } }
I20240503 02:00:30.847978 16957 raft_consensus.cc:1261] T 78f81fe4c6234ff0b5fcabd974779a66 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:30.848428 19691 consensus_queue.cc:1035] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:30.848526 17223 raft_consensus.cc:1261] T 78f81fe4c6234ff0b5fcabd974779a66 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:30.849273 17750 raft_consensus.cc:1261] T 78f81fe4c6234ff0b5fcabd974779a66 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:30.849830 19896 consensus_queue.cc:1035] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 02:00:30.850329 17026 consensus_peers.cc:485] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 78f81fe4c6234ff0b5fcabd974779a66. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:30.850463 19691 consensus_queue.cc:1035] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:30.850678 17487 raft_consensus.cc:1261] T 78f81fe4c6234ff0b5fcabd974779a66 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:30.851094 17619 raft_consensus.cc:1261] T 78f81fe4c6234ff0b5fcabd974779a66 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:30.851406 19676 consensus_queue.cc:1035] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:30.851512 17353 raft_consensus.cc:1261] T 78f81fe4c6234ff0b5fcabd974779a66 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:30.851935 19896 consensus_queue.cc:1035] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:30.850325 17091 consensus_queue.cc:237] T e245c9cf642043cfbb3b87114beeb002 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:30.852079 19676 consensus_queue.cc:1035] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:30.854136 16957 raft_consensus.cc:1261] T e245c9cf642043cfbb3b87114beeb002 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:30.854451 17353 raft_consensus.cc:1261] T e245c9cf642043cfbb3b87114beeb002 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:30.854472 17222 raft_consensus.cc:1261] T e245c9cf642043cfbb3b87114beeb002 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:30.854537 19896 consensus_queue.cc:1035] T e245c9cf642043cfbb3b87114beeb002 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:30.854887 19896 consensus_queue.cc:1035] T e245c9cf642043cfbb3b87114beeb002 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:30.854936 17486 raft_consensus.cc:1261] T e245c9cf642043cfbb3b87114beeb002 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:30.855068 17751 raft_consensus.cc:1261] T e245c9cf642043cfbb3b87114beeb002 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:30.855079 20057 raft_consensus.cc:2932] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } }
I20240503 02:00:30.856415 19691 consensus_queue.cc:1035] T e245c9cf642043cfbb3b87114beeb002 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:30.856889 16805 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 78f81fe4c6234ff0b5fcabd974779a66 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 02:00:30.857118 19896 consensus_queue.cc:1035] T e245c9cf642043cfbb3b87114beeb002 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:30.857199 17619 raft_consensus.cc:1261] T e245c9cf642043cfbb3b87114beeb002 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:30.857445 19896 consensus_queue.cc:1035] T e245c9cf642043cfbb3b87114beeb002 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:30.857388 17620 raft_consensus.cc:2932] T 78f81fe4c6234ff0b5fcabd974779a66 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } }
I20240503 02:00:30.857961 19896 consensus_queue.cc:1035] T e245c9cf642043cfbb3b87114beeb002 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:30.861395 16818 catalog_manager.cc:5484] T 78f81fe4c6234ff0b5fcabd974779a66 P c843193b801b4f1a88f630ed3ed0f375 reported cstate change: config changed from index 4 to 5, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New cstate: current_term: 1 leader_uuid: "89dd09f8aaee4b249ae3ce924a422a67" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } }
I20240503 02:00:30.861387 17353 raft_consensus.cc:2932] T 78f81fe4c6234ff0b5fcabd974779a66 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } }
I20240503 02:00:30.861970 17487 raft_consensus.cc:2932] T 78f81fe4c6234ff0b5fcabd974779a66 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } }
I20240503 02:00:30.862677 20059 raft_consensus.cc:2932] T e245c9cf642043cfbb3b87114beeb002 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } }
I20240503 02:00:30.867292 16958 raft_consensus.cc:2932] T e245c9cf642043cfbb3b87114beeb002 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } }
I20240503 02:00:30.867307 17751 raft_consensus.cc:2932] T e245c9cf642043cfbb3b87114beeb002 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } }
I20240503 02:00:30.868775 16957 raft_consensus.cc:2932] T 78f81fe4c6234ff0b5fcabd974779a66 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } }
I20240503 02:00:30.871683 16818 catalog_manager.cc:5484] T e245c9cf642043cfbb3b87114beeb002 P 002cc3411acb4fdfa0b67c6f57f687a6 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "89dd09f8aaee4b249ae3ce924a422a67" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } }
I20240503 02:00:30.873028 17223 raft_consensus.cc:2932] T e245c9cf642043cfbb3b87114beeb002 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } }
I20240503 02:00:30.873136 17222 raft_consensus.cc:2932] T 78f81fe4c6234ff0b5fcabd974779a66 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } }
I20240503 02:00:30.875005 17353 raft_consensus.cc:2932] T e245c9cf642043cfbb3b87114beeb002 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } }
I20240503 02:00:30.880537 17486 raft_consensus.cc:2932] T e245c9cf642043cfbb3b87114beeb002 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } }
I20240503 02:00:30.880517 17620 raft_consensus.cc:2932] T e245c9cf642043cfbb3b87114beeb002 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } }
I20240503 02:00:30.880877 17091 consensus_queue.cc:237] T e245c9cf642043cfbb3b87114beeb002 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } }
I20240503 02:00:30.882567 17751 raft_consensus.cc:2932] T 78f81fe4c6234ff0b5fcabd974779a66 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } }
I20240503 02:00:30.883821 17222 raft_consensus.cc:1261] T e245c9cf642043cfbb3b87114beeb002 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:30.884213 17750 raft_consensus.cc:1261] T e245c9cf642043cfbb3b87114beeb002 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240503 02:00:30.884407 17026 consensus_peers.cc:485] T e245c9cf642043cfbb3b87114beeb002 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e245c9cf642043cfbb3b87114beeb002. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:30.885226 17620 raft_consensus.cc:1261] T e245c9cf642043cfbb3b87114beeb002 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:30.885484 16957 raft_consensus.cc:1261] T e245c9cf642043cfbb3b87114beeb002 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:30.886034 17486 raft_consensus.cc:1261] T e245c9cf642043cfbb3b87114beeb002 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:30.886204 19896 consensus_queue.cc:1035] T e245c9cf642043cfbb3b87114beeb002 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:30.886651 17353 raft_consensus.cc:1261] T e245c9cf642043cfbb3b87114beeb002 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:30.889168 19896 consensus_queue.cc:1035] T e245c9cf642043cfbb3b87114beeb002 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:30.889521 20055 consensus_queue.cc:1035] T e245c9cf642043cfbb3b87114beeb002 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:30.890830 19896 consensus_queue.cc:1035] T e245c9cf642043cfbb3b87114beeb002 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:30.891023 20056 consensus_queue.cc:1035] T e245c9cf642043cfbb3b87114beeb002 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:30.891150 19691 consensus_queue.cc:1035] T e245c9cf642043cfbb3b87114beeb002 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:30.893033 20056 raft_consensus.cc:2932] T e245c9cf642043cfbb3b87114beeb002 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } }
I20240503 02:00:30.893693 17353 raft_consensus.cc:2932] T e245c9cf642043cfbb3b87114beeb002 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } }
I20240503 02:00:30.893958 17486 raft_consensus.cc:2932] T e245c9cf642043cfbb3b87114beeb002 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } }
I20240503 02:00:30.895349 16958 raft_consensus.cc:2932] T e245c9cf642043cfbb3b87114beeb002 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } }
I20240503 02:00:30.895871 17620 raft_consensus.cc:2932] T e245c9cf642043cfbb3b87114beeb002 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } }
I20240503 02:00:30.896853 16812 catalog_manager.cc:5484] T e245c9cf642043cfbb3b87114beeb002 P d43ed6b8663947d99a72cd922cc323aa reported cstate change: config changed from index 4 to 5, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New cstate: current_term: 1 leader_uuid: "89dd09f8aaee4b249ae3ce924a422a67" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } }
I20240503 02:00:30.897336 16805 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet e245c9cf642043cfbb3b87114beeb002 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 02:00:30.897933 17222 raft_consensus.cc:2932] T e245c9cf642043cfbb3b87114beeb002 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } }
I20240503 02:00:30.898181 17750 raft_consensus.cc:2932] T e245c9cf642043cfbb3b87114beeb002 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } }
I20240503 02:00:31.154392 17222 consensus_queue.cc:237] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:31.156666 17353 raft_consensus.cc:1261] T 4782de9826d6460e9e993e01327034e4 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Refusing update from remote peer 5b8291b26a7f448b8636187d1caf86d0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:31.156684 17750 raft_consensus.cc:1261] T 4782de9826d6460e9e993e01327034e4 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer 5b8291b26a7f448b8636187d1caf86d0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:31.156780 17091 raft_consensus.cc:1261] T 4782de9826d6460e9e993e01327034e4 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer 5b8291b26a7f448b8636187d1caf86d0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:31.157119 20231 consensus_queue.cc:1035] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:31.157369 20233 consensus_queue.cc:1035] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:31.157616 16958 raft_consensus.cc:1261] T 4782de9826d6460e9e993e01327034e4 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 5b8291b26a7f448b8636187d1caf86d0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:31.157580 20232 consensus_queue.cc:1035] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:31.157828 17620 raft_consensus.cc:1261] T 4782de9826d6460e9e993e01327034e4 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer 5b8291b26a7f448b8636187d1caf86d0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:31.158354 20233 consensus_queue.cc:1035] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:31.158643 17486 raft_consensus.cc:1261] T 4782de9826d6460e9e993e01327034e4 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Refusing update from remote peer 5b8291b26a7f448b8636187d1caf86d0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:31.159190 20231 consensus_queue.cc:1035] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:31.160064 20232 consensus_queue.cc:1035] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:31.161397 20232 raft_consensus.cc:2932] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } }
I20240503 02:00:31.161787 17091 raft_consensus.cc:2932] T 4782de9826d6460e9e993e01327034e4 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } }
I20240503 02:00:31.163332 17486 raft_consensus.cc:2932] T 4782de9826d6460e9e993e01327034e4 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } }
I20240503 02:00:31.163336 16958 raft_consensus.cc:2932] T 4782de9826d6460e9e993e01327034e4 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } }
I20240503 02:00:31.165024 16818 catalog_manager.cc:5484] T 4782de9826d6460e9e993e01327034e4 P 89dd09f8aaee4b249ae3ce924a422a67 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "5b8291b26a7f448b8636187d1caf86d0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } }
I20240503 02:00:31.165884 17750 raft_consensus.cc:2932] T 4782de9826d6460e9e993e01327034e4 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } }
I20240503 02:00:31.167804 17353 raft_consensus.cc:2932] T 4782de9826d6460e9e993e01327034e4 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } }
I20240503 02:00:31.169581 17620 raft_consensus.cc:2932] T 4782de9826d6460e9e993e01327034e4 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } }
I20240503 02:00:31.170846 17222 consensus_queue.cc:237] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } }
I20240503 02:00:31.173288 17091 raft_consensus.cc:1261] T 4782de9826d6460e9e993e01327034e4 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer 5b8291b26a7f448b8636187d1caf86d0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:31.173494 17486 raft_consensus.cc:1261] T 4782de9826d6460e9e993e01327034e4 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Refusing update from remote peer 5b8291b26a7f448b8636187d1caf86d0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:31.174245 17751 raft_consensus.cc:1261] T 4782de9826d6460e9e993e01327034e4 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer 5b8291b26a7f448b8636187d1caf86d0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:31.174226 18997 consensus_queue.cc:1035] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:31.174654 18997 consensus_queue.cc:1035] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:31.176508 16958 raft_consensus.cc:1261] T 4782de9826d6460e9e993e01327034e4 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 5b8291b26a7f448b8636187d1caf86d0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:31.177027 20246 consensus_queue.cc:1035] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:31.177175 17353 raft_consensus.cc:1261] T 4782de9826d6460e9e993e01327034e4 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Refusing update from remote peer 5b8291b26a7f448b8636187d1caf86d0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:31.177572 20246 consensus_queue.cc:1035] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:31.177796 20192 consensus_queue.cc:1035] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:31.179026 17620 raft_consensus.cc:1261] T 4782de9826d6460e9e993e01327034e4 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer 5b8291b26a7f448b8636187d1caf86d0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:31.179502 18997 consensus_queue.cc:1035] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:31.180473 18997 raft_consensus.cc:2932] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:31.181207 17353 raft_consensus.cc:2932] T 4782de9826d6460e9e993e01327034e4 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:31.182371 16803 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 4782de9826d6460e9e993e01327034e4 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 02:00:31.182657 17091 raft_consensus.cc:2932] T 4782de9826d6460e9e993e01327034e4 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:31.183097 17486 raft_consensus.cc:2932] T 4782de9826d6460e9e993e01327034e4 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:31.186280 16811 catalog_manager.cc:5484] T 4782de9826d6460e9e993e01327034e4 P d43ed6b8663947d99a72cd922cc323aa reported cstate change: config changed from index 4 to 5, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "5b8291b26a7f448b8636187d1caf86d0" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:31.187049 17750 raft_consensus.cc:2932] T 4782de9826d6460e9e993e01327034e4 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
W20240503 02:00:31.188802 17158 consensus_peers.cc:485] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4782de9826d6460e9e993e01327034e4. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:31.191463 16958 raft_consensus.cc:2932] T 4782de9826d6460e9e993e01327034e4 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:31.194226 17620 raft_consensus.cc:2932] T 4782de9826d6460e9e993e01327034e4 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:31.201579 17090 consensus_queue.cc:237] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:31.203717 17353 raft_consensus.cc:1261] T 30abfada8ab44449b9a4237c57093b73 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:31.203776 17222 raft_consensus.cc:1261] T 30abfada8ab44449b9a4237c57093b73 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:31.203967 17750 raft_consensus.cc:1261] T 30abfada8ab44449b9a4237c57093b73 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:31.204195 17620 raft_consensus.cc:1261] T 30abfada8ab44449b9a4237c57093b73 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:31.204247 19676 consensus_queue.cc:1035] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:31.204555 19691 consensus_queue.cc:1035] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:31.204666 16958 raft_consensus.cc:1261] T 30abfada8ab44449b9a4237c57093b73 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:31.204749 20058 consensus_queue.cc:1035] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:31.205102 20058 consensus_queue.cc:1035] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:31.206583 19676 consensus_queue.cc:1035] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:31.207803 17486 raft_consensus.cc:1261] T 30abfada8ab44449b9a4237c57093b73 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:31.207787 19896 raft_consensus.cc:2932] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } }
I20240503 02:00:31.208318 19676 consensus_queue.cc:1035] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:31.208284 17353 raft_consensus.cc:2932] T 30abfada8ab44449b9a4237c57093b73 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } }
I20240503 02:00:31.210793 17619 raft_consensus.cc:2932] T 30abfada8ab44449b9a4237c57093b73 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } }
I20240503 02:00:31.210808 17486 raft_consensus.cc:2932] T 30abfada8ab44449b9a4237c57093b73 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } }
I20240503 02:00:31.212707 16818 catalog_manager.cc:5484] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "89dd09f8aaee4b249ae3ce924a422a67" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:31.216331 16957 raft_consensus.cc:2932] T 30abfada8ab44449b9a4237c57093b73 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } }
I20240503 02:00:31.216552 17222 raft_consensus.cc:2932] T 30abfada8ab44449b9a4237c57093b73 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } }
I20240503 02:00:31.217175 17750 raft_consensus.cc:2932] T 30abfada8ab44449b9a4237c57093b73 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } }
I20240503 02:00:31.221881 17090 consensus_queue.cc:237] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } }
I20240503 02:00:31.224323 17619 raft_consensus.cc:1261] T 30abfada8ab44449b9a4237c57093b73 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:31.224320 17486 raft_consensus.cc:1261] T 30abfada8ab44449b9a4237c57093b73 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:31.224522 17750 raft_consensus.cc:1261] T 30abfada8ab44449b9a4237c57093b73 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:31.224960 19896 consensus_queue.cc:1035] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:31.225075 20058 consensus_queue.cc:1035] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:31.225283 19896 consensus_queue.cc:1035] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 02:00:31.225616 17026 consensus_peers.cc:485] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 30abfada8ab44449b9a4237c57093b73. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:31.225834 17353 raft_consensus.cc:1261] T 30abfada8ab44449b9a4237c57093b73 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:31.226011 17222 raft_consensus.cc:1261] T 30abfada8ab44449b9a4237c57093b73 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:31.226330 19896 consensus_queue.cc:1035] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:31.226487 16957 raft_consensus.cc:1261] T 30abfada8ab44449b9a4237c57093b73 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:31.226585 19896 consensus_queue.cc:1035] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:31.226889 20057 consensus_queue.cc:1035] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:31.228696 20056 raft_consensus.cc:2932] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:31.229082 17750 raft_consensus.cc:2932] T 30abfada8ab44449b9a4237c57093b73 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:31.230417 16957 raft_consensus.cc:2932] T 30abfada8ab44449b9a4237c57093b73 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:31.231204 17619 raft_consensus.cc:2932] T 30abfada8ab44449b9a4237c57093b73 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:31.231604 16805 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 30abfada8ab44449b9a4237c57093b73 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 02:00:31.231464 17486 raft_consensus.cc:2932] T 30abfada8ab44449b9a4237c57093b73 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:31.232343 16813 catalog_manager.cc:5484] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 reported cstate change: config changed from index 4 to 5, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "89dd09f8aaee4b249ae3ce924a422a67" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:31.234588 17222 raft_consensus.cc:2932] T 30abfada8ab44449b9a4237c57093b73 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:31.235688 17353 raft_consensus.cc:2932] T 30abfada8ab44449b9a4237c57093b73 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:31.238476 17486 consensus_queue.cc:237] T 42a900125241432a8bb31dad0b940968 P 16d437fe629643948c95d628926d5ffa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:31.241412 16957 raft_consensus.cc:1261] T 42a900125241432a8bb31dad0b940968 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:31.241875 17619 raft_consensus.cc:1261] T 42a900125241432a8bb31dad0b940968 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:31.241896 19947 consensus_queue.cc:1035] T 42a900125241432a8bb31dad0b940968 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:31.242412 17222 raft_consensus.cc:1261] T 42a900125241432a8bb31dad0b940968 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:31.242467 19949 consensus_queue.cc:1035] T 42a900125241432a8bb31dad0b940968 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:31.242854 17750 raft_consensus.cc:1261] T 42a900125241432a8bb31dad0b940968 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:31.243036 19949 consensus_queue.cc:1035] T 42a900125241432a8bb31dad0b940968 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:31.243355 17090 raft_consensus.cc:1261] T 42a900125241432a8bb31dad0b940968 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:31.243420 19947 consensus_queue.cc:1035] T 42a900125241432a8bb31dad0b940968 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:31.243822 19875 consensus_queue.cc:1035] T 42a900125241432a8bb31dad0b940968 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:31.244885 17353 raft_consensus.cc:1261] T 42a900125241432a8bb31dad0b940968 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:31.245323 19875 consensus_queue.cc:1035] T 42a900125241432a8bb31dad0b940968 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:31.245927 19953 raft_consensus.cc:2932] T 42a900125241432a8bb31dad0b940968 P 16d437fe629643948c95d628926d5ffa [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } }
I20240503 02:00:31.246263 17223 raft_consensus.cc:2932] T 42a900125241432a8bb31dad0b940968 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } }
I20240503 02:00:31.246385 17090 raft_consensus.cc:2932] T 42a900125241432a8bb31dad0b940968 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } }
I20240503 02:00:31.248189 16957 raft_consensus.cc:2932] T 42a900125241432a8bb31dad0b940968 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } }
I20240503 02:00:31.248296 17750 raft_consensus.cc:2932] T 42a900125241432a8bb31dad0b940968 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } }
I20240503 02:00:31.250048 16817 catalog_manager.cc:5484] T 42a900125241432a8bb31dad0b940968 P 5b8291b26a7f448b8636187d1caf86d0 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "16d437fe629643948c95d628926d5ffa" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } }
I20240503 02:00:31.251787 17619 raft_consensus.cc:2932] T 42a900125241432a8bb31dad0b940968 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } }
I20240503 02:00:31.256086 17353 raft_consensus.cc:2932] T 42a900125241432a8bb31dad0b940968 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } }
I20240503 02:00:31.256845 17486 consensus_queue.cc:237] T 42a900125241432a8bb31dad0b940968 P 16d437fe629643948c95d628926d5ffa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } }
I20240503 02:00:31.258891 17750 raft_consensus.cc:1261] T 42a900125241432a8bb31dad0b940968 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:31.259670 17090 raft_consensus.cc:1261] T 42a900125241432a8bb31dad0b940968 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:31.259692 17619 raft_consensus.cc:1261] T 42a900125241432a8bb31dad0b940968 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:31.259884 19875 consensus_queue.cc:1035] T 42a900125241432a8bb31dad0b940968 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:31.260156 19875 consensus_queue.cc:1035] T 42a900125241432a8bb31dad0b940968 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:31.261673 16957 raft_consensus.cc:1261] T 42a900125241432a8bb31dad0b940968 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:31.262032 20040 consensus_queue.cc:1035] T 42a900125241432a8bb31dad0b940968 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:31.262650 17223 raft_consensus.cc:1261] T 42a900125241432a8bb31dad0b940968 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:31.263526 19620 consensus_queue.cc:1035] T 42a900125241432a8bb31dad0b940968 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:31.265820 19875 raft_consensus.cc:2932] T 42a900125241432a8bb31dad0b940968 P 16d437fe629643948c95d628926d5ffa [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
W20240503 02:00:31.266198 17422 consensus_peers.cc:485] T 42a900125241432a8bb31dad0b940968 P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 42a900125241432a8bb31dad0b940968. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:31.266337 20040 consensus_queue.cc:1035] T 42a900125241432a8bb31dad0b940968 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:31.266261 17090 raft_consensus.cc:2932] T 42a900125241432a8bb31dad0b940968 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:31.267982 16802 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 42a900125241432a8bb31dad0b940968 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 02:00:31.267838 17750 raft_consensus.cc:2932] T 42a900125241432a8bb31dad0b940968 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:31.268394 16958 raft_consensus.cc:2932] T 42a900125241432a8bb31dad0b940968 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:31.269289 17619 raft_consensus.cc:2932] T 42a900125241432a8bb31dad0b940968 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:31.270094 16811 catalog_manager.cc:5484] T 42a900125241432a8bb31dad0b940968 P 89dd09f8aaee4b249ae3ce924a422a67 reported cstate change: config changed from index 4 to 5, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "16d437fe629643948c95d628926d5ffa" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:31.271675 17222 raft_consensus.cc:2932] T 42a900125241432a8bb31dad0b940968 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:31.274252 17353 raft_consensus.cc:1261] T 42a900125241432a8bb31dad0b940968 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:31.274819 19620 consensus_queue.cc:1035] T 42a900125241432a8bb31dad0b940968 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:31.276252 17353 raft_consensus.cc:2932] T 42a900125241432a8bb31dad0b940968 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:31.281397 20278 ts_tablet_manager.cc:902] T a39f284167e942188c181147e8a9e74e P 838ef968bb45484bb27f5af6f3989bae: Initiating tablet copy from peer d43ed6b8663947d99a72cd922cc323aa (127.0.99.132:40023)
I20240503 02:00:31.282193 20278 tablet_copy_client.cc:308] T a39f284167e942188c181147e8a9e74e P 838ef968bb45484bb27f5af6f3989bae: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:40023
I20240503 02:00:31.287169 17376 tablet_copy_service.cc:140] P d43ed6b8663947d99a72cd922cc323aa: Received BeginTabletCopySession request for tablet a39f284167e942188c181147e8a9e74e from peer 838ef968bb45484bb27f5af6f3989bae ({username='slave'} at 127.0.99.137:39379)
I20240503 02:00:31.287266 17376 tablet_copy_service.cc:161] P d43ed6b8663947d99a72cd922cc323aa: Beginning new tablet copy session on tablet a39f284167e942188c181147e8a9e74e from peer 838ef968bb45484bb27f5af6f3989bae at {username='slave'} at 127.0.99.137:39379: session id = 838ef968bb45484bb27f5af6f3989bae-a39f284167e942188c181147e8a9e74e
I20240503 02:00:31.287992 17376 tablet_copy_source_session.cc:205] T a39f284167e942188c181147e8a9e74e P d43ed6b8663947d99a72cd922cc323aa: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:31.289019 20278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a39f284167e942188c181147e8a9e74e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:31.292418 20278 tablet_copy_client.cc:783] T a39f284167e942188c181147e8a9e74e P 838ef968bb45484bb27f5af6f3989bae: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:31.292740 20278 tablet_copy_client.cc:647] T a39f284167e942188c181147e8a9e74e P 838ef968bb45484bb27f5af6f3989bae: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:31.296289 20278 tablet_copy_client.cc:523] T a39f284167e942188c181147e8a9e74e P 838ef968bb45484bb27f5af6f3989bae: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:31.297979 20278 tablet_bootstrap.cc:492] T a39f284167e942188c181147e8a9e74e P 838ef968bb45484bb27f5af6f3989bae: Bootstrap starting.
I20240503 02:00:31.319311 20292 ts_tablet_manager.cc:902] T e245c9cf642043cfbb3b87114beeb002 P 838ef968bb45484bb27f5af6f3989bae: Initiating tablet copy from peer 89dd09f8aaee4b249ae3ce924a422a67 (127.0.99.130:38889)
I20240503 02:00:31.319957 20292 tablet_copy_client.cc:308] T e245c9cf642043cfbb3b87114beeb002 P 838ef968bb45484bb27f5af6f3989bae: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:38889
I20240503 02:00:31.320497 17112 tablet_copy_service.cc:140] P 89dd09f8aaee4b249ae3ce924a422a67: Received BeginTabletCopySession request for tablet e245c9cf642043cfbb3b87114beeb002 from peer 838ef968bb45484bb27f5af6f3989bae ({username='slave'} at 127.0.99.137:47267)
I20240503 02:00:31.320606 17112 tablet_copy_service.cc:161] P 89dd09f8aaee4b249ae3ce924a422a67: Beginning new tablet copy session on tablet e245c9cf642043cfbb3b87114beeb002 from peer 838ef968bb45484bb27f5af6f3989bae at {username='slave'} at 127.0.99.137:47267: session id = 838ef968bb45484bb27f5af6f3989bae-e245c9cf642043cfbb3b87114beeb002
I20240503 02:00:31.320676 20291 ts_tablet_manager.cc:902] T 78f81fe4c6234ff0b5fcabd974779a66 P 838ef968bb45484bb27f5af6f3989bae: Initiating tablet copy from peer 89dd09f8aaee4b249ae3ce924a422a67 (127.0.99.130:38889)
I20240503 02:00:31.321831 17112 tablet_copy_source_session.cc:205] T e245c9cf642043cfbb3b87114beeb002 P 89dd09f8aaee4b249ae3ce924a422a67: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:31.322772 20291 tablet_copy_client.cc:308] T 78f81fe4c6234ff0b5fcabd974779a66 P 838ef968bb45484bb27f5af6f3989bae: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:38889
I20240503 02:00:31.322868 20292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e245c9cf642043cfbb3b87114beeb002. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:31.323472 17112 tablet_copy_service.cc:140] P 89dd09f8aaee4b249ae3ce924a422a67: Received BeginTabletCopySession request for tablet 78f81fe4c6234ff0b5fcabd974779a66 from peer 838ef968bb45484bb27f5af6f3989bae ({username='slave'} at 127.0.99.137:47267)
I20240503 02:00:31.323571 17112 tablet_copy_service.cc:161] P 89dd09f8aaee4b249ae3ce924a422a67: Beginning new tablet copy session on tablet 78f81fe4c6234ff0b5fcabd974779a66 from peer 838ef968bb45484bb27f5af6f3989bae at {username='slave'} at 127.0.99.137:47267: session id = 838ef968bb45484bb27f5af6f3989bae-78f81fe4c6234ff0b5fcabd974779a66
I20240503 02:00:31.324391 17112 tablet_copy_source_session.cc:205] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:31.326726 20292 tablet_copy_client.cc:783] T e245c9cf642043cfbb3b87114beeb002 P 838ef968bb45484bb27f5af6f3989bae: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:31.326933 20292 tablet_copy_client.cc:647] T e245c9cf642043cfbb3b87114beeb002 P 838ef968bb45484bb27f5af6f3989bae: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:31.327574 20291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78f81fe4c6234ff0b5fcabd974779a66. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:31.329023 20292 tablet_copy_client.cc:523] T e245c9cf642043cfbb3b87114beeb002 P 838ef968bb45484bb27f5af6f3989bae: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:31.329807 20278 tablet_bootstrap.cc:492] T a39f284167e942188c181147e8a9e74e P 838ef968bb45484bb27f5af6f3989bae: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:31.330148 20278 tablet_bootstrap.cc:492] T a39f284167e942188c181147e8a9e74e P 838ef968bb45484bb27f5af6f3989bae: Bootstrap complete.
I20240503 02:00:31.330359 20278 ts_tablet_manager.cc:1367] T a39f284167e942188c181147e8a9e74e P 838ef968bb45484bb27f5af6f3989bae: Time spent bootstrapping tablet: real 0.032s user 0.024s sys 0.001s
I20240503 02:00:31.330801 20292 tablet_bootstrap.cc:492] T e245c9cf642043cfbb3b87114beeb002 P 838ef968bb45484bb27f5af6f3989bae: Bootstrap starting.
I20240503 02:00:31.330817 20291 tablet_copy_client.cc:783] T 78f81fe4c6234ff0b5fcabd974779a66 P 838ef968bb45484bb27f5af6f3989bae: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:31.331194 20291 tablet_copy_client.cc:647] T 78f81fe4c6234ff0b5fcabd974779a66 P 838ef968bb45484bb27f5af6f3989bae: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:31.331660 20278 raft_consensus.cc:348] T a39f284167e942188c181147e8a9e74e P 838ef968bb45484bb27f5af6f3989bae [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } }
I20240503 02:00:31.331809 20278 raft_consensus.cc:729] T a39f284167e942188c181147e8a9e74e P 838ef968bb45484bb27f5af6f3989bae [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 838ef968bb45484bb27f5af6f3989bae, State: Initialized, Role: LEARNER
I20240503 02:00:31.331897 20278 consensus_queue.cc:260] T a39f284167e942188c181147e8a9e74e P 838ef968bb45484bb27f5af6f3989bae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } }
I20240503 02:00:31.332300 20278 ts_tablet_manager.cc:1398] T a39f284167e942188c181147e8a9e74e P 838ef968bb45484bb27f5af6f3989bae: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 02:00:31.333124 17376 tablet_copy_service.cc:342] P d43ed6b8663947d99a72cd922cc323aa: Request end of tablet copy session 838ef968bb45484bb27f5af6f3989bae-a39f284167e942188c181147e8a9e74e received from {username='slave'} at 127.0.99.137:39379
I20240503 02:00:31.333226 17376 tablet_copy_service.cc:433] P d43ed6b8663947d99a72cd922cc323aa: ending tablet copy session 838ef968bb45484bb27f5af6f3989bae-a39f284167e942188c181147e8a9e74e on tablet a39f284167e942188c181147e8a9e74e with peer 838ef968bb45484bb27f5af6f3989bae
I20240503 02:00:31.333406 20291 tablet_copy_client.cc:523] T 78f81fe4c6234ff0b5fcabd974779a66 P 838ef968bb45484bb27f5af6f3989bae: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:31.334893 20291 tablet_bootstrap.cc:492] T 78f81fe4c6234ff0b5fcabd974779a66 P 838ef968bb45484bb27f5af6f3989bae: Bootstrap starting.
I20240503 02:00:31.363366 20291 tablet_bootstrap.cc:492] T 78f81fe4c6234ff0b5fcabd974779a66 P 838ef968bb45484bb27f5af6f3989bae: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:31.363845 20291 tablet_bootstrap.cc:492] T 78f81fe4c6234ff0b5fcabd974779a66 P 838ef968bb45484bb27f5af6f3989bae: Bootstrap complete.
I20240503 02:00:31.364105 20291 ts_tablet_manager.cc:1367] T 78f81fe4c6234ff0b5fcabd974779a66 P 838ef968bb45484bb27f5af6f3989bae: Time spent bootstrapping tablet: real 0.029s user 0.029s sys 0.000s
I20240503 02:00:31.364127 20292 tablet_bootstrap.cc:492] T e245c9cf642043cfbb3b87114beeb002 P 838ef968bb45484bb27f5af6f3989bae: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:31.364511 20292 tablet_bootstrap.cc:492] T e245c9cf642043cfbb3b87114beeb002 P 838ef968bb45484bb27f5af6f3989bae: Bootstrap complete.
I20240503 02:00:31.364722 20292 ts_tablet_manager.cc:1367] T e245c9cf642043cfbb3b87114beeb002 P 838ef968bb45484bb27f5af6f3989bae: Time spent bootstrapping tablet: real 0.034s user 0.026s sys 0.001s
I20240503 02:00:31.364766 20291 raft_consensus.cc:348] T 78f81fe4c6234ff0b5fcabd974779a66 P 838ef968bb45484bb27f5af6f3989bae [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } }
I20240503 02:00:31.364921 20291 raft_consensus.cc:729] T 78f81fe4c6234ff0b5fcabd974779a66 P 838ef968bb45484bb27f5af6f3989bae [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 838ef968bb45484bb27f5af6f3989bae, State: Initialized, Role: LEARNER
I20240503 02:00:31.365087 20291 consensus_queue.cc:260] T 78f81fe4c6234ff0b5fcabd974779a66 P 838ef968bb45484bb27f5af6f3989bae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } }
I20240503 02:00:31.365199 20292 raft_consensus.cc:348] T e245c9cf642043cfbb3b87114beeb002 P 838ef968bb45484bb27f5af6f3989bae [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } }
I20240503 02:00:31.365288 20292 raft_consensus.cc:729] T e245c9cf642043cfbb3b87114beeb002 P 838ef968bb45484bb27f5af6f3989bae [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 838ef968bb45484bb27f5af6f3989bae, State: Initialized, Role: LEARNER
I20240503 02:00:31.365450 20291 ts_tablet_manager.cc:1398] T 78f81fe4c6234ff0b5fcabd974779a66 P 838ef968bb45484bb27f5af6f3989bae: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:31.365490 20292 consensus_queue.cc:260] T e245c9cf642043cfbb3b87114beeb002 P 838ef968bb45484bb27f5af6f3989bae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } }
I20240503 02:00:31.365839 20292 ts_tablet_manager.cc:1398] T e245c9cf642043cfbb3b87114beeb002 P 838ef968bb45484bb27f5af6f3989bae: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:31.366008 17112 tablet_copy_service.cc:342] P 89dd09f8aaee4b249ae3ce924a422a67: Request end of tablet copy session 838ef968bb45484bb27f5af6f3989bae-78f81fe4c6234ff0b5fcabd974779a66 received from {username='slave'} at 127.0.99.137:47267
I20240503 02:00:31.366092 17112 tablet_copy_service.cc:433] P 89dd09f8aaee4b249ae3ce924a422a67: ending tablet copy session 838ef968bb45484bb27f5af6f3989bae-78f81fe4c6234ff0b5fcabd974779a66 on tablet 78f81fe4c6234ff0b5fcabd974779a66 with peer 838ef968bb45484bb27f5af6f3989bae
I20240503 02:00:31.366408 17111 tablet_copy_service.cc:342] P 89dd09f8aaee4b249ae3ce924a422a67: Request end of tablet copy session 838ef968bb45484bb27f5af6f3989bae-e245c9cf642043cfbb3b87114beeb002 received from {username='slave'} at 127.0.99.137:47267
I20240503 02:00:31.366509 17111 tablet_copy_service.cc:433] P 89dd09f8aaee4b249ae3ce924a422a67: ending tablet copy session 838ef968bb45484bb27f5af6f3989bae-e245c9cf642043cfbb3b87114beeb002 on tablet e245c9cf642043cfbb3b87114beeb002 with peer 838ef968bb45484bb27f5af6f3989bae
I20240503 02:00:31.366966 20278 ts_tablet_manager.cc:902] T 3f67591d6a3744ab90a3e4d2375a59db P 838ef968bb45484bb27f5af6f3989bae: Initiating tablet copy from peer 16d437fe629643948c95d628926d5ffa (127.0.99.133:46581)
I20240503 02:00:31.367547 20278 tablet_copy_client.cc:308] T 3f67591d6a3744ab90a3e4d2375a59db P 838ef968bb45484bb27f5af6f3989bae: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.133:46581
I20240503 02:00:31.372449 17508 tablet_copy_service.cc:140] P 16d437fe629643948c95d628926d5ffa: Received BeginTabletCopySession request for tablet 3f67591d6a3744ab90a3e4d2375a59db from peer 838ef968bb45484bb27f5af6f3989bae ({username='slave'} at 127.0.99.137:37139)
I20240503 02:00:31.372556 17508 tablet_copy_service.cc:161] P 16d437fe629643948c95d628926d5ffa: Beginning new tablet copy session on tablet 3f67591d6a3744ab90a3e4d2375a59db from peer 838ef968bb45484bb27f5af6f3989bae at {username='slave'} at 127.0.99.137:37139: session id = 838ef968bb45484bb27f5af6f3989bae-3f67591d6a3744ab90a3e4d2375a59db
I20240503 02:00:31.373293 17508 tablet_copy_source_session.cc:205] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:31.374260 20278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f67591d6a3744ab90a3e4d2375a59db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:31.376720 20278 tablet_copy_client.cc:783] T 3f67591d6a3744ab90a3e4d2375a59db P 838ef968bb45484bb27f5af6f3989bae: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:31.376906 20278 tablet_copy_client.cc:647] T 3f67591d6a3744ab90a3e4d2375a59db P 838ef968bb45484bb27f5af6f3989bae: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:31.378253 20278 tablet_copy_client.cc:523] T 3f67591d6a3744ab90a3e4d2375a59db P 838ef968bb45484bb27f5af6f3989bae: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:31.379542 20278 tablet_bootstrap.cc:492] T 3f67591d6a3744ab90a3e4d2375a59db P 838ef968bb45484bb27f5af6f3989bae: Bootstrap starting.
I20240503 02:00:31.400871 20278 tablet_bootstrap.cc:492] T 3f67591d6a3744ab90a3e4d2375a59db P 838ef968bb45484bb27f5af6f3989bae: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:31.401149 20278 tablet_bootstrap.cc:492] T 3f67591d6a3744ab90a3e4d2375a59db P 838ef968bb45484bb27f5af6f3989bae: Bootstrap complete.
I20240503 02:00:31.401345 20278 ts_tablet_manager.cc:1367] T 3f67591d6a3744ab90a3e4d2375a59db P 838ef968bb45484bb27f5af6f3989bae: Time spent bootstrapping tablet: real 0.022s user 0.021s sys 0.000s
I20240503 02:00:31.401768 20278 raft_consensus.cc:348] T 3f67591d6a3744ab90a3e4d2375a59db P 838ef968bb45484bb27f5af6f3989bae [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } }
I20240503 02:00:31.401887 20278 raft_consensus.cc:729] T 3f67591d6a3744ab90a3e4d2375a59db P 838ef968bb45484bb27f5af6f3989bae [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 838ef968bb45484bb27f5af6f3989bae, State: Initialized, Role: LEARNER
I20240503 02:00:31.401988 20278 consensus_queue.cc:260] T 3f67591d6a3744ab90a3e4d2375a59db P 838ef968bb45484bb27f5af6f3989bae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } }
I20240503 02:00:31.402727 20278 ts_tablet_manager.cc:1398] T 3f67591d6a3744ab90a3e4d2375a59db P 838ef968bb45484bb27f5af6f3989bae: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 02:00:31.403249 17508 tablet_copy_service.cc:342] P 16d437fe629643948c95d628926d5ffa: Request end of tablet copy session 838ef968bb45484bb27f5af6f3989bae-3f67591d6a3744ab90a3e4d2375a59db received from {username='slave'} at 127.0.99.137:37139
I20240503 02:00:31.403344 17508 tablet_copy_service.cc:433] P 16d437fe629643948c95d628926d5ffa: ending tablet copy session 838ef968bb45484bb27f5af6f3989bae-3f67591d6a3744ab90a3e4d2375a59db on tablet 3f67591d6a3744ab90a3e4d2375a59db with peer 838ef968bb45484bb27f5af6f3989bae
I20240503 02:00:31.611433 20359 ts_tablet_manager.cc:902] T 4782de9826d6460e9e993e01327034e4 P 375d0f2ec11d4e25bd9d093bb6a5572d: Initiating tablet copy from peer 5b8291b26a7f448b8636187d1caf86d0 (127.0.99.131:44479)
I20240503 02:00:31.612103 20359 tablet_copy_client.cc:308] T 4782de9826d6460e9e993e01327034e4 P 375d0f2ec11d4e25bd9d093bb6a5572d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:44479
I20240503 02:00:31.616083 17244 tablet_copy_service.cc:140] P 5b8291b26a7f448b8636187d1caf86d0: Received BeginTabletCopySession request for tablet 4782de9826d6460e9e993e01327034e4 from peer 375d0f2ec11d4e25bd9d093bb6a5572d ({username='slave'} at 127.0.99.136:39149)
I20240503 02:00:31.616298 17244 tablet_copy_service.cc:161] P 5b8291b26a7f448b8636187d1caf86d0: Beginning new tablet copy session on tablet 4782de9826d6460e9e993e01327034e4 from peer 375d0f2ec11d4e25bd9d093bb6a5572d at {username='slave'} at 127.0.99.136:39149: session id = 375d0f2ec11d4e25bd9d093bb6a5572d-4782de9826d6460e9e993e01327034e4
I20240503 02:00:31.617836 17244 tablet_copy_source_session.cc:205] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:31.619695 20359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4782de9826d6460e9e993e01327034e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:31.622484 20359 tablet_copy_client.cc:783] T 4782de9826d6460e9e993e01327034e4 P 375d0f2ec11d4e25bd9d093bb6a5572d: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:31.622610 20359 tablet_copy_client.cc:647] T 4782de9826d6460e9e993e01327034e4 P 375d0f2ec11d4e25bd9d093bb6a5572d: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:31.624255 20359 tablet_copy_client.cc:523] T 4782de9826d6460e9e993e01327034e4 P 375d0f2ec11d4e25bd9d093bb6a5572d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:31.625646 20359 tablet_bootstrap.cc:492] T 4782de9826d6460e9e993e01327034e4 P 375d0f2ec11d4e25bd9d093bb6a5572d: Bootstrap starting.
I20240503 02:00:31.644575 20359 tablet_bootstrap.cc:492] T 4782de9826d6460e9e993e01327034e4 P 375d0f2ec11d4e25bd9d093bb6a5572d: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:31.644886 20359 tablet_bootstrap.cc:492] T 4782de9826d6460e9e993e01327034e4 P 375d0f2ec11d4e25bd9d093bb6a5572d: Bootstrap complete.
I20240503 02:00:31.645063 20359 ts_tablet_manager.cc:1367] T 4782de9826d6460e9e993e01327034e4 P 375d0f2ec11d4e25bd9d093bb6a5572d: Time spent bootstrapping tablet: real 0.019s user 0.013s sys 0.007s
I20240503 02:00:31.645488 20359 raft_consensus.cc:348] T 4782de9826d6460e9e993e01327034e4 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } }
I20240503 02:00:31.645615 20359 raft_consensus.cc:729] T 4782de9826d6460e9e993e01327034e4 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 375d0f2ec11d4e25bd9d093bb6a5572d, State: Initialized, Role: LEARNER
I20240503 02:00:31.645730 20359 consensus_queue.cc:260] T 4782de9826d6460e9e993e01327034e4 P 375d0f2ec11d4e25bd9d093bb6a5572d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } }
I20240503 02:00:31.646090 20359 ts_tablet_manager.cc:1398] T 4782de9826d6460e9e993e01327034e4 P 375d0f2ec11d4e25bd9d093bb6a5572d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:31.646889 17244 tablet_copy_service.cc:342] P 5b8291b26a7f448b8636187d1caf86d0: Request end of tablet copy session 375d0f2ec11d4e25bd9d093bb6a5572d-4782de9826d6460e9e993e01327034e4 received from {username='slave'} at 127.0.99.136:39149
I20240503 02:00:31.647059 17244 tablet_copy_service.cc:433] P 5b8291b26a7f448b8636187d1caf86d0: ending tablet copy session 375d0f2ec11d4e25bd9d093bb6a5572d-4782de9826d6460e9e993e01327034e4 on tablet 4782de9826d6460e9e993e01327034e4 with peer 375d0f2ec11d4e25bd9d093bb6a5572d
I20240503 02:00:31.719650 19108 raft_consensus.cc:1203] T a39f284167e942188c181147e8a9e74e P 838ef968bb45484bb27f5af6f3989bae [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 02:00:31.730494 19108 raft_consensus.cc:1203] T e245c9cf642043cfbb3b87114beeb002 P 838ef968bb45484bb27f5af6f3989bae [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 02:00:31.773206 20359 ts_tablet_manager.cc:902] T 30abfada8ab44449b9a4237c57093b73 P 375d0f2ec11d4e25bd9d093bb6a5572d: Initiating tablet copy from peer 89dd09f8aaee4b249ae3ce924a422a67 (127.0.99.130:38889)
I20240503 02:00:31.773900 20359 tablet_copy_client.cc:308] T 30abfada8ab44449b9a4237c57093b73 P 375d0f2ec11d4e25bd9d093bb6a5572d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:38889
I20240503 02:00:31.774506 17111 tablet_copy_service.cc:140] P 89dd09f8aaee4b249ae3ce924a422a67: Received BeginTabletCopySession request for tablet 30abfada8ab44449b9a4237c57093b73 from peer 375d0f2ec11d4e25bd9d093bb6a5572d ({username='slave'} at 127.0.99.136:56011)
I20240503 02:00:31.774606 17111 tablet_copy_service.cc:161] P 89dd09f8aaee4b249ae3ce924a422a67: Beginning new tablet copy session on tablet 30abfada8ab44449b9a4237c57093b73 from peer 375d0f2ec11d4e25bd9d093bb6a5572d at {username='slave'} at 127.0.99.136:56011: session id = 375d0f2ec11d4e25bd9d093bb6a5572d-30abfada8ab44449b9a4237c57093b73
I20240503 02:00:31.775427 17111 tablet_copy_source_session.cc:205] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:31.776377 20359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30abfada8ab44449b9a4237c57093b73. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:31.778306 20359 tablet_copy_client.cc:783] T 30abfada8ab44449b9a4237c57093b73 P 375d0f2ec11d4e25bd9d093bb6a5572d: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:31.778481 20359 tablet_copy_client.cc:647] T 30abfada8ab44449b9a4237c57093b73 P 375d0f2ec11d4e25bd9d093bb6a5572d: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:31.779027 20379 ts_tablet_manager.cc:902] T 42a900125241432a8bb31dad0b940968 P 375d0f2ec11d4e25bd9d093bb6a5572d: Initiating tablet copy from peer 16d437fe629643948c95d628926d5ffa (127.0.99.133:46581)
I20240503 02:00:31.779982 20359 tablet_copy_client.cc:523] T 30abfada8ab44449b9a4237c57093b73 P 375d0f2ec11d4e25bd9d093bb6a5572d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:31.780359 20379 tablet_copy_client.cc:308] T 42a900125241432a8bb31dad0b940968 P 375d0f2ec11d4e25bd9d093bb6a5572d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.133:46581
I20240503 02:00:31.781332 20359 tablet_bootstrap.cc:492] T 30abfada8ab44449b9a4237c57093b73 P 375d0f2ec11d4e25bd9d093bb6a5572d: Bootstrap starting.
I20240503 02:00:31.782148 17508 tablet_copy_service.cc:140] P 16d437fe629643948c95d628926d5ffa: Received BeginTabletCopySession request for tablet 42a900125241432a8bb31dad0b940968 from peer 375d0f2ec11d4e25bd9d093bb6a5572d ({username='slave'} at 127.0.99.136:36855)
I20240503 02:00:31.782266 17508 tablet_copy_service.cc:161] P 16d437fe629643948c95d628926d5ffa: Beginning new tablet copy session on tablet 42a900125241432a8bb31dad0b940968 from peer 375d0f2ec11d4e25bd9d093bb6a5572d at {username='slave'} at 127.0.99.136:36855: session id = 375d0f2ec11d4e25bd9d093bb6a5572d-42a900125241432a8bb31dad0b940968
I20240503 02:00:31.783355 17508 tablet_copy_source_session.cc:205] T 42a900125241432a8bb31dad0b940968 P 16d437fe629643948c95d628926d5ffa: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:31.784535 20379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42a900125241432a8bb31dad0b940968. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:31.787506 20379 tablet_copy_client.cc:783] T 42a900125241432a8bb31dad0b940968 P 375d0f2ec11d4e25bd9d093bb6a5572d: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:31.787653 20379 tablet_copy_client.cc:647] T 42a900125241432a8bb31dad0b940968 P 375d0f2ec11d4e25bd9d093bb6a5572d: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:31.788893 20379 tablet_copy_client.cc:523] T 42a900125241432a8bb31dad0b940968 P 375d0f2ec11d4e25bd9d093bb6a5572d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:31.789907 20379 tablet_bootstrap.cc:492] T 42a900125241432a8bb31dad0b940968 P 375d0f2ec11d4e25bd9d093bb6a5572d: Bootstrap starting.
I20240503 02:00:31.804898 20359 tablet_bootstrap.cc:492] T 30abfada8ab44449b9a4237c57093b73 P 375d0f2ec11d4e25bd9d093bb6a5572d: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:31.805212 20359 tablet_bootstrap.cc:492] T 30abfada8ab44449b9a4237c57093b73 P 375d0f2ec11d4e25bd9d093bb6a5572d: Bootstrap complete.
I20240503 02:00:31.805387 20359 ts_tablet_manager.cc:1367] T 30abfada8ab44449b9a4237c57093b73 P 375d0f2ec11d4e25bd9d093bb6a5572d: Time spent bootstrapping tablet: real 0.024s user 0.019s sys 0.003s
I20240503 02:00:31.806011 20359 raft_consensus.cc:348] T 30abfada8ab44449b9a4237c57093b73 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } }
I20240503 02:00:31.806133 20359 raft_consensus.cc:729] T 30abfada8ab44449b9a4237c57093b73 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 375d0f2ec11d4e25bd9d093bb6a5572d, State: Initialized, Role: LEARNER
I20240503 02:00:31.806236 20359 consensus_queue.cc:260] T 30abfada8ab44449b9a4237c57093b73 P 375d0f2ec11d4e25bd9d093bb6a5572d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } }
I20240503 02:00:31.807034 20359 ts_tablet_manager.cc:1398] T 30abfada8ab44449b9a4237c57093b73 P 375d0f2ec11d4e25bd9d093bb6a5572d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:31.807480 17111 tablet_copy_service.cc:342] P 89dd09f8aaee4b249ae3ce924a422a67: Request end of tablet copy session 375d0f2ec11d4e25bd9d093bb6a5572d-30abfada8ab44449b9a4237c57093b73 received from {username='slave'} at 127.0.99.136:56011
I20240503 02:00:31.807559 17111 tablet_copy_service.cc:433] P 89dd09f8aaee4b249ae3ce924a422a67: ending tablet copy session 375d0f2ec11d4e25bd9d093bb6a5572d-30abfada8ab44449b9a4237c57093b73 on tablet 30abfada8ab44449b9a4237c57093b73 with peer 375d0f2ec11d4e25bd9d093bb6a5572d
I20240503 02:00:31.812484 20379 tablet_bootstrap.cc:492] T 42a900125241432a8bb31dad0b940968 P 375d0f2ec11d4e25bd9d093bb6a5572d: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:31.812812 20379 tablet_bootstrap.cc:492] T 42a900125241432a8bb31dad0b940968 P 375d0f2ec11d4e25bd9d093bb6a5572d: Bootstrap complete.
I20240503 02:00:31.812994 20379 ts_tablet_manager.cc:1367] T 42a900125241432a8bb31dad0b940968 P 375d0f2ec11d4e25bd9d093bb6a5572d: Time spent bootstrapping tablet: real 0.023s user 0.016s sys 0.007s
I20240503 02:00:31.813442 20379 raft_consensus.cc:348] T 42a900125241432a8bb31dad0b940968 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } }
I20240503 02:00:31.813591 20379 raft_consensus.cc:729] T 42a900125241432a8bb31dad0b940968 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 375d0f2ec11d4e25bd9d093bb6a5572d, State: Initialized, Role: LEARNER
I20240503 02:00:31.813689 20379 consensus_queue.cc:260] T 42a900125241432a8bb31dad0b940968 P 375d0f2ec11d4e25bd9d093bb6a5572d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } }
I20240503 02:00:31.813994 20379 ts_tablet_manager.cc:1398] T 42a900125241432a8bb31dad0b940968 P 375d0f2ec11d4e25bd9d093bb6a5572d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:31.814496 17508 tablet_copy_service.cc:342] P 16d437fe629643948c95d628926d5ffa: Request end of tablet copy session 375d0f2ec11d4e25bd9d093bb6a5572d-42a900125241432a8bb31dad0b940968 received from {username='slave'} at 127.0.99.136:36855
I20240503 02:00:31.814628 17508 tablet_copy_service.cc:433] P 16d437fe629643948c95d628926d5ffa: ending tablet copy session 375d0f2ec11d4e25bd9d093bb6a5572d-42a900125241432a8bb31dad0b940968 on tablet 42a900125241432a8bb31dad0b940968 with peer 375d0f2ec11d4e25bd9d093bb6a5572d
I20240503 02:00:31.917519 19108 raft_consensus.cc:1203] T 78f81fe4c6234ff0b5fcabd974779a66 P 838ef968bb45484bb27f5af6f3989bae [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 02:00:31.964706 19108 raft_consensus.cc:1203] T 3f67591d6a3744ab90a3e4d2375a59db P 838ef968bb45484bb27f5af6f3989bae [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 02:00:31.993963 19370 raft_consensus.cc:1203] T 4782de9826d6460e9e993e01327034e4 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 02:00:32.157387 20372 raft_consensus.cc:1050] T a39f284167e942188c181147e8a9e74e P d43ed6b8663947d99a72cd922cc323aa: attempting to promote NON_VOTER 838ef968bb45484bb27f5af6f3989bae to VOTER
I20240503 02:00:32.158349 20372 consensus_queue.cc:237] T a39f284167e942188c181147e8a9e74e P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } }
I20240503 02:00:32.161104 17091 raft_consensus.cc:1261] T a39f284167e942188c181147e8a9e74e P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:32.161254 17223 raft_consensus.cc:1261] T a39f284167e942188c181147e8a9e74e P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:32.161286 17620 raft_consensus.cc:1261] T a39f284167e942188c181147e8a9e74e P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:32.161871 20458 consensus_queue.cc:1035] T a39f284167e942188c181147e8a9e74e P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:32.161913 17486 raft_consensus.cc:1261] T a39f284167e942188c181147e8a9e74e P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:32.162175 20373 consensus_queue.cc:1035] T a39f284167e942188c181147e8a9e74e P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:32.162339 19894 consensus_queue.cc:1035] T a39f284167e942188c181147e8a9e74e P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:32.162469 20372 consensus_queue.cc:1035] T a39f284167e942188c181147e8a9e74e P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:32.162671 19108 raft_consensus.cc:1261] T a39f284167e942188c181147e8a9e74e P 838ef968bb45484bb27f5af6f3989bae [term 1 LEARNER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:32.163332 19894 consensus_queue.cc:1035] T a39f284167e942188c181147e8a9e74e P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:32.163421 17750 raft_consensus.cc:1261] T a39f284167e942188c181147e8a9e74e P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:32.164695 20374 consensus_queue.cc:1035] T a39f284167e942188c181147e8a9e74e P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:32.166786 16957 raft_consensus.cc:1261] T a39f284167e942188c181147e8a9e74e P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:32.167529 20459 consensus_queue.cc:1035] T a39f284167e942188c181147e8a9e74e P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:32.168838 17750 raft_consensus.cc:2932] T a39f284167e942188c181147e8a9e74e P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:32.168793 20372 raft_consensus.cc:2932] T a39f284167e942188c181147e8a9e74e P d43ed6b8663947d99a72cd922cc323aa [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:32.171293 19108 raft_consensus.cc:2932] T a39f284167e942188c181147e8a9e74e P 838ef968bb45484bb27f5af6f3989bae [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:32.172027 17486 raft_consensus.cc:2932] T a39f284167e942188c181147e8a9e74e P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:32.172267 17091 raft_consensus.cc:2932] T a39f284167e942188c181147e8a9e74e P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:32.173553 17620 raft_consensus.cc:2932] T a39f284167e942188c181147e8a9e74e P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:32.175145 16813 catalog_manager.cc:5484] T a39f284167e942188c181147e8a9e74e P d43ed6b8663947d99a72cd922cc323aa reported cstate change: config changed from index 5 to 6, 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d43ed6b8663947d99a72cd922cc323aa" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:00:32.180388 17223 raft_consensus.cc:2932] T a39f284167e942188c181147e8a9e74e P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:32.181942 16957 raft_consensus.cc:2932] T a39f284167e942188c181147e8a9e74e P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:32.188923 17353 consensus_queue.cc:237] T a39f284167e942188c181147e8a9e74e P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } }
I20240503 02:00:32.191226 17750 raft_consensus.cc:1261] T a39f284167e942188c181147e8a9e74e P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:32.191274 17620 raft_consensus.cc:1261] T a39f284167e942188c181147e8a9e74e P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:32.191274 16957 raft_consensus.cc:1261] T a39f284167e942188c181147e8a9e74e P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:32.191622 19108 raft_consensus.cc:1261] T a39f284167e942188c181147e8a9e74e P 838ef968bb45484bb27f5af6f3989bae [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:32.192137 19894 consensus_queue.cc:1035] T a39f284167e942188c181147e8a9e74e P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:32.192510 19894 consensus_queue.cc:1035] T a39f284167e942188c181147e8a9e74e P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:32.192715 17223 raft_consensus.cc:1261] T a39f284167e942188c181147e8a9e74e P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:32.192873 19894 consensus_queue.cc:1035] T a39f284167e942188c181147e8a9e74e P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:32.193066 20458 consensus_queue.cc:1035] T a39f284167e942188c181147e8a9e74e P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:32.193338 20458 consensus_queue.cc:1035] T a39f284167e942188c181147e8a9e74e P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:32.193578 17091 raft_consensus.cc:1261] T a39f284167e942188c181147e8a9e74e P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:32.194003 20372 consensus_queue.cc:1035] T a39f284167e942188c181147e8a9e74e P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:32.196281 19894 raft_consensus.cc:2932] T a39f284167e942188c181147e8a9e74e P d43ed6b8663947d99a72cd922cc323aa [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 16d437fe629643948c95d628926d5ffa (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:32.198559 16803 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a39f284167e942188c181147e8a9e74e with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:00:32.199087 17091 raft_consensus.cc:2932] T a39f284167e942188c181147e8a9e74e P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 16d437fe629643948c95d628926d5ffa (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:32.199676 19108 raft_consensus.cc:2932] T a39f284167e942188c181147e8a9e74e P 838ef968bb45484bb27f5af6f3989bae [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 16d437fe629643948c95d628926d5ffa (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:32.201071 16817 catalog_manager.cc:5484] T a39f284167e942188c181147e8a9e74e P d43ed6b8663947d99a72cd922cc323aa reported cstate change: config changed from index 6 to 7, VOTER 16d437fe629643948c95d628926d5ffa (127.0.99.133) evicted. New cstate: current_term: 1 leader_uuid: "d43ed6b8663947d99a72cd922cc323aa" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:00:32.202809 17223 raft_consensus.cc:2932] T a39f284167e942188c181147e8a9e74e P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 16d437fe629643948c95d628926d5ffa (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:32.202713 17750 raft_consensus.cc:2932] T a39f284167e942188c181147e8a9e74e P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 16d437fe629643948c95d628926d5ffa (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:32.203696 16957 raft_consensus.cc:2932] T a39f284167e942188c181147e8a9e74e P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 16d437fe629643948c95d628926d5ffa (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:32.205476 17620 raft_consensus.cc:2932] T a39f284167e942188c181147e8a9e74e P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 16d437fe629643948c95d628926d5ffa (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:32.208325 17467 tablet_service.cc:1514] Processing DeleteTablet for tablet a39f284167e942188c181147e8a9e74e with delete_type TABLET_DATA_TOMBSTONED (TS 16d437fe629643948c95d628926d5ffa not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:33592
I20240503 02:00:32.209905 20471 tablet_replica.cc:330] T a39f284167e942188c181147e8a9e74e P 16d437fe629643948c95d628926d5ffa: stopping tablet replica
I20240503 02:00:32.210323 20471 raft_consensus.cc:2229] T a39f284167e942188c181147e8a9e74e P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:32.210641 20471 raft_consensus.cc:2258] T a39f284167e942188c181147e8a9e74e P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:32.211485 20471 ts_tablet_manager.cc:1875] T a39f284167e942188c181147e8a9e74e P 16d437fe629643948c95d628926d5ffa: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:32.213402 20471 ts_tablet_manager.cc:1888] T a39f284167e942188c181147e8a9e74e P 16d437fe629643948c95d628926d5ffa: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 02:00:32.213471 20471 log.cc:1196] T a39f284167e942188c181147e8a9e74e P 16d437fe629643948c95d628926d5ffa: Deleting WAL directory at /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/a39f284167e942188c181147e8a9e74e
I20240503 02:00:32.213963 16802 catalog_manager.cc:4830] TS 16d437fe629643948c95d628926d5ffa (127.0.99.133:46581): tablet a39f284167e942188c181147e8a9e74e (table rebalance_test_table_4 [id=421dd60555d242a78638e28180581a9c]) successfully deleted
I20240503 02:00:32.297457 19370 raft_consensus.cc:1203] T 30abfada8ab44449b9a4237c57093b73 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 02:00:32.300256 19896 raft_consensus.cc:1050] T e245c9cf642043cfbb3b87114beeb002 P 89dd09f8aaee4b249ae3ce924a422a67: attempting to promote NON_VOTER 838ef968bb45484bb27f5af6f3989bae to VOTER
I20240503 02:00:32.300840 19896 consensus_queue.cc:237] T e245c9cf642043cfbb3b87114beeb002 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } }
I20240503 02:00:32.302865 16957 raft_consensus.cc:1261] T e245c9cf642043cfbb3b87114beeb002 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:32.303035 17223 raft_consensus.cc:1261] T e245c9cf642043cfbb3b87114beeb002 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:32.303037 17353 raft_consensus.cc:1261] T e245c9cf642043cfbb3b87114beeb002 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:32.303406 20474 consensus_queue.cc:1035] T e245c9cf642043cfbb3b87114beeb002 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:32.303764 19108 raft_consensus.cc:1261] T e245c9cf642043cfbb3b87114beeb002 P 838ef968bb45484bb27f5af6f3989bae [term 1 LEARNER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:32.304164 20474 consensus_queue.cc:1035] T e245c9cf642043cfbb3b87114beeb002 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:32.304323 19896 consensus_queue.cc:1035] T e245c9cf642043cfbb3b87114beeb002 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:32.304497 19676 consensus_queue.cc:1035] T e245c9cf642043cfbb3b87114beeb002 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:32.308053 19896 raft_consensus.cc:2932] T e245c9cf642043cfbb3b87114beeb002 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:32.308494 17750 raft_consensus.cc:1261] T e245c9cf642043cfbb3b87114beeb002 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:32.309062 19108 raft_consensus.cc:2932] T e245c9cf642043cfbb3b87114beeb002 P 838ef968bb45484bb27f5af6f3989bae [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:32.311793 17486 raft_consensus.cc:1261] T e245c9cf642043cfbb3b87114beeb002 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:32.311854 17620 raft_consensus.cc:1261] T e245c9cf642043cfbb3b87114beeb002 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:32.312204 17353 raft_consensus.cc:2932] T e245c9cf642043cfbb3b87114beeb002 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:32.312302 16957 raft_consensus.cc:2932] T e245c9cf642043cfbb3b87114beeb002 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:32.312304 16818 catalog_manager.cc:5484] T e245c9cf642043cfbb3b87114beeb002 P 89dd09f8aaee4b249ae3ce924a422a67 reported cstate change: config changed from index 5 to 6, 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "89dd09f8aaee4b249ae3ce924a422a67" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:00:32.313377 19676 consensus_queue.cc:1035] T e245c9cf642043cfbb3b87114beeb002 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:32.314783 17750 raft_consensus.cc:2932] T e245c9cf642043cfbb3b87114beeb002 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:32.315398 17223 raft_consensus.cc:2932] T e245c9cf642043cfbb3b87114beeb002 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:32.318894 20475 consensus_queue.cc:1035] T e245c9cf642043cfbb3b87114beeb002 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:32.319144 19676 consensus_queue.cc:1035] T e245c9cf642043cfbb3b87114beeb002 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:32.320686 17620 raft_consensus.cc:2932] T e245c9cf642043cfbb3b87114beeb002 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:32.323091 17486 raft_consensus.cc:2932] T e245c9cf642043cfbb3b87114beeb002 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:32.331897 19370 raft_consensus.cc:1203] T 42a900125241432a8bb31dad0b940968 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 02:00:32.366782 19620 raft_consensus.cc:1050] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa: attempting to promote NON_VOTER 838ef968bb45484bb27f5af6f3989bae to VOTER
I20240503 02:00:32.367326 19620 consensus_queue.cc:237] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } }
I20240503 02:00:32.369552 19108 raft_consensus.cc:1261] T 3f67591d6a3744ab90a3e4d2375a59db P 838ef968bb45484bb27f5af6f3989bae [term 1 LEARNER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:32.369655 17353 raft_consensus.cc:1261] T 3f67591d6a3744ab90a3e4d2375a59db P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:32.370007 20516 consensus_queue.cc:1035] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:32.370107 17223 raft_consensus.cc:1261] T 3f67591d6a3744ab90a3e4d2375a59db P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:32.370208 20040 consensus_queue.cc:1035] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:32.370800 17091 raft_consensus.cc:1261] T 3f67591d6a3744ab90a3e4d2375a59db P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:32.371230 19947 consensus_queue.cc:1035] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:32.371398 17620 raft_consensus.cc:1261] T 3f67591d6a3744ab90a3e4d2375a59db P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:32.371469 17750 raft_consensus.cc:1261] T 3f67591d6a3744ab90a3e4d2375a59db P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:32.371575 20040 consensus_queue.cc:1035] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:32.371769 19947 consensus_queue.cc:1035] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:32.372005 16957 raft_consensus.cc:1261] T 3f67591d6a3744ab90a3e4d2375a59db P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:32.372550 19947 consensus_queue.cc:1035] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:32.372912 19947 consensus_queue.cc:1035] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:32.375756 20421 raft_consensus.cc:1050] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0: attempting to promote NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d to VOTER
I20240503 02:00:32.376351 20421 consensus_queue.cc:237] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } }
I20240503 02:00:32.377377 20515 raft_consensus.cc:2932] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:32.377777 17353 raft_consensus.cc:2932] T 3f67591d6a3744ab90a3e4d2375a59db P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:32.379719 17751 raft_consensus.cc:1261] T 4782de9826d6460e9e993e01327034e4 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer 5b8291b26a7f448b8636187d1caf86d0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:32.380092 17620 raft_consensus.cc:1261] T 4782de9826d6460e9e993e01327034e4 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer 5b8291b26a7f448b8636187d1caf86d0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:32.380465 16957 raft_consensus.cc:1261] T 4782de9826d6460e9e993e01327034e4 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 5b8291b26a7f448b8636187d1caf86d0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:32.380264 16958 raft_consensus.cc:2932] T 3f67591d6a3744ab90a3e4d2375a59db P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:32.381707 17223 raft_consensus.cc:2932] T 3f67591d6a3744ab90a3e4d2375a59db P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:32.382117 16811 catalog_manager.cc:5484] T 3f67591d6a3744ab90a3e4d2375a59db P d43ed6b8663947d99a72cd922cc323aa reported cstate change: config changed from index 5 to 6, 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "16d437fe629643948c95d628926d5ffa" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:32.383137 17090 raft_consensus.cc:1261] T 4782de9826d6460e9e993e01327034e4 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer 5b8291b26a7f448b8636187d1caf86d0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:32.383972 19108 raft_consensus.cc:2932] T 3f67591d6a3744ab90a3e4d2375a59db P 838ef968bb45484bb27f5af6f3989bae [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:32.384508 20231 consensus_queue.cc:1035] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:32.388875 19370 raft_consensus.cc:1261] T 4782de9826d6460e9e993e01327034e4 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 LEARNER]: Refusing update from remote peer 5b8291b26a7f448b8636187d1caf86d0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:32.388943 17620 raft_consensus.cc:2932] T 3f67591d6a3744ab90a3e4d2375a59db P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:32.390945 20231 consensus_queue.cc:1035] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:32.391209 20231 consensus_queue.cc:1035] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:32.391459 20231 consensus_queue.cc:1035] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:32.391782 20231 consensus_queue.cc:1035] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:32.392292 17486 raft_consensus.cc:1261] T 4782de9826d6460e9e993e01327034e4 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Refusing update from remote peer 5b8291b26a7f448b8636187d1caf86d0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:32.392467 17353 raft_consensus.cc:1261] T 4782de9826d6460e9e993e01327034e4 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Refusing update from remote peer 5b8291b26a7f448b8636187d1caf86d0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:32.394474 20231 consensus_queue.cc:1035] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:32.394826 20231 consensus_queue.cc:1035] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:32.395956 17091 raft_consensus.cc:2932] T 3f67591d6a3744ab90a3e4d2375a59db P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:32.398308 17750 raft_consensus.cc:2932] T 3f67591d6a3744ab90a3e4d2375a59db P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:32.403057 17486 consensus_queue.cc:237] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } }
I20240503 02:00:32.404956 17223 raft_consensus.cc:1261] T 3f67591d6a3744ab90a3e4d2375a59db P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:32.405342 17750 raft_consensus.cc:1261] T 3f67591d6a3744ab90a3e4d2375a59db P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:32.405568 19108 raft_consensus.cc:1261] T 3f67591d6a3744ab90a3e4d2375a59db P 838ef968bb45484bb27f5af6f3989bae [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:32.405711 16957 raft_consensus.cc:1261] T 3f67591d6a3744ab90a3e4d2375a59db P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:32.405790 17353 raft_consensus.cc:1261] T 3f67591d6a3744ab90a3e4d2375a59db P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:32.406296 20517 consensus_queue.cc:1035] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:32.406414 17091 raft_consensus.cc:1261] T 3f67591d6a3744ab90a3e4d2375a59db P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:32.406589 20516 consensus_queue.cc:1035] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:32.406760 20517 consensus_queue.cc:1035] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:32.407141 20517 consensus_queue.cc:1035] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:32.407665 20231 raft_consensus.cc:2932] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:32.407956 19620 consensus_queue.cc:1035] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:32.408221 16958 raft_consensus.cc:2932] T 4782de9826d6460e9e993e01327034e4 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:32.409950 19370 raft_consensus.cc:2932] T 4782de9826d6460e9e993e01327034e4 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:32.410569 17356 raft_consensus.cc:2932] T 4782de9826d6460e9e993e01327034e4 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:32.411803 16811 catalog_manager.cc:5484] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 reported cstate change: config changed from index 5 to 6, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5b8291b26a7f448b8636187d1caf86d0" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:00:32.412434 17750 raft_consensus.cc:2932] T 4782de9826d6460e9e993e01327034e4 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:32.415292 19620 consensus_queue.cc:1035] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:32.416289 17486 raft_consensus.cc:2932] T 4782de9826d6460e9e993e01327034e4 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:32.417888 20519 raft_consensus.cc:2932] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:32.419394 17223 raft_consensus.cc:2932] T 3f67591d6a3744ab90a3e4d2375a59db P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:32.419404 19108 raft_consensus.cc:2932] T 3f67591d6a3744ab90a3e4d2375a59db P 838ef968bb45484bb27f5af6f3989bae [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:32.420383 17353 raft_consensus.cc:2932] T 3f67591d6a3744ab90a3e4d2375a59db P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:32.421998 17751 raft_consensus.cc:2932] T 3f67591d6a3744ab90a3e4d2375a59db P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:32.422214 16812 catalog_manager.cc:5484] T 3f67591d6a3744ab90a3e4d2375a59db P 838ef968bb45484bb27f5af6f3989bae reported cstate change: config changed from index 6 to 7, VOTER c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134) evicted. New cstate: current_term: 1 leader_uuid: "16d437fe629643948c95d628926d5ffa" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:32.422597 16802 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3f67591d6a3744ab90a3e4d2375a59db with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:00:32.423815 16957 raft_consensus.cc:2932] T 3f67591d6a3744ab90a3e4d2375a59db P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:32.425626 17620 raft_consensus.cc:2932] T 4782de9826d6460e9e993e01327034e4 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:32.426941 17598 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f67591d6a3744ab90a3e4d2375a59db with delete_type TABLET_DATA_TOMBSTONED (TS c843193b801b4f1a88f630ed3ed0f375 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:51030
I20240503 02:00:32.428352 17090 raft_consensus.cc:2932] T 3f67591d6a3744ab90a3e4d2375a59db P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:32.428684 20546 tablet_replica.cc:330] T 3f67591d6a3744ab90a3e4d2375a59db P c843193b801b4f1a88f630ed3ed0f375: stopping tablet replica
I20240503 02:00:32.428872 20546 raft_consensus.cc:2229] T 3f67591d6a3744ab90a3e4d2375a59db P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:32.429078 20546 raft_consensus.cc:2258] T 3f67591d6a3744ab90a3e4d2375a59db P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:32.430073 20546 ts_tablet_manager.cc:1875] T 3f67591d6a3744ab90a3e4d2375a59db P c843193b801b4f1a88f630ed3ed0f375: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:32.430334 17091 raft_consensus.cc:2932] T 4782de9826d6460e9e993e01327034e4 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:32.435380 20546 ts_tablet_manager.cc:1888] T 3f67591d6a3744ab90a3e4d2375a59db P c843193b801b4f1a88f630ed3ed0f375: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 02:00:32.435631 20546 log.cc:1196] T 3f67591d6a3744ab90a3e4d2375a59db P c843193b801b4f1a88f630ed3ed0f375: Deleting WAL directory at /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/3f67591d6a3744ab90a3e4d2375a59db
I20240503 02:00:32.436167 16802 catalog_manager.cc:4830] TS c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134:44343): tablet 3f67591d6a3744ab90a3e4d2375a59db (table rebalance_test_table_6 [id=fc19f05fb43b48068855575b19b79556]) successfully deleted
I20240503 02:00:32.436975 17223 consensus_queue.cc:237] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } }
I20240503 02:00:32.438558 17751 raft_consensus.cc:1261] T 4782de9826d6460e9e993e01327034e4 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer 5b8291b26a7f448b8636187d1caf86d0: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:32.438875 17486 raft_consensus.cc:1261] T 4782de9826d6460e9e993e01327034e4 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Refusing update from remote peer 5b8291b26a7f448b8636187d1caf86d0: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:32.438989 17620 raft_consensus.cc:1261] T 4782de9826d6460e9e993e01327034e4 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer 5b8291b26a7f448b8636187d1caf86d0: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:32.438985 20231 consensus_queue.cc:1035] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:32.439389 20532 consensus_queue.cc:1035] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:32.439460 19370 raft_consensus.cc:1261] T 4782de9826d6460e9e993e01327034e4 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 FOLLOWER]: Refusing update from remote peer 5b8291b26a7f448b8636187d1caf86d0: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:32.439574 16957 raft_consensus.cc:1261] T 4782de9826d6460e9e993e01327034e4 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 5b8291b26a7f448b8636187d1caf86d0: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:32.439898 20421 consensus_queue.cc:1035] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:32.440064 17091 raft_consensus.cc:1261] T 4782de9826d6460e9e993e01327034e4 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer 5b8291b26a7f448b8636187d1caf86d0: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:32.440166 20231 consensus_queue.cc:1035] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:32.440305 20421 consensus_queue.cc:1035] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:32.440503 20532 consensus_queue.cc:1035] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:32.442674 20421 raft_consensus.cc:2932] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:32.443504 19370 raft_consensus.cc:2932] T 4782de9826d6460e9e993e01327034e4 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:32.443954 17620 raft_consensus.cc:2932] T 4782de9826d6460e9e993e01327034e4 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:32.444955 16803 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4782de9826d6460e9e993e01327034e4 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:00:32.445573 17091 raft_consensus.cc:2932] T 4782de9826d6460e9e993e01327034e4 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:32.446765 16957 raft_consensus.cc:2932] T 4782de9826d6460e9e993e01327034e4 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:32.447907 16815 catalog_manager.cc:5484] T 4782de9826d6460e9e993e01327034e4 P c843193b801b4f1a88f630ed3ed0f375 reported cstate change: config changed from index 6 to 7, VOTER d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "5b8291b26a7f448b8636187d1caf86d0" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:32.451468 17335 tablet_service.cc:1514] Processing DeleteTablet for tablet 4782de9826d6460e9e993e01327034e4 with delete_type TABLET_DATA_TOMBSTONED (TS d43ed6b8663947d99a72cd922cc323aa not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:44362
I20240503 02:00:32.452498 20547 tablet_replica.cc:330] T 4782de9826d6460e9e993e01327034e4 P d43ed6b8663947d99a72cd922cc323aa: stopping tablet replica
I20240503 02:00:32.452720 20547 raft_consensus.cc:2229] T 4782de9826d6460e9e993e01327034e4 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:32.452948 20547 raft_consensus.cc:2258] T 4782de9826d6460e9e993e01327034e4 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:32.455034 17751 raft_consensus.cc:2932] T 4782de9826d6460e9e993e01327034e4 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:32.457098 17486 raft_consensus.cc:2932] T 4782de9826d6460e9e993e01327034e4 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:32.459408 20547 ts_tablet_manager.cc:1875] T 4782de9826d6460e9e993e01327034e4 P d43ed6b8663947d99a72cd922cc323aa: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:32.461894 20547 ts_tablet_manager.cc:1888] T 4782de9826d6460e9e993e01327034e4 P d43ed6b8663947d99a72cd922cc323aa: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 02:00:32.461967 20547 log.cc:1196] T 4782de9826d6460e9e993e01327034e4 P d43ed6b8663947d99a72cd922cc323aa: Deleting WAL directory at /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/4782de9826d6460e9e993e01327034e4
I20240503 02:00:32.462525 16803 catalog_manager.cc:4830] TS d43ed6b8663947d99a72cd922cc323aa (127.0.99.132:40023): tablet 4782de9826d6460e9e993e01327034e4 (table rebalance_test_table_6 [id=fc19f05fb43b48068855575b19b79556]) successfully deleted
I20240503 02:00:32.520021 19676 raft_consensus.cc:1050] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67: attempting to promote NON_VOTER 838ef968bb45484bb27f5af6f3989bae to VOTER
I20240503 02:00:32.520725 19676 consensus_queue.cc:237] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } }
I20240503 02:00:32.523033 17751 raft_consensus.cc:1261] T 78f81fe4c6234ff0b5fcabd974779a66 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:32.523120 17620 raft_consensus.cc:1261] T 78f81fe4c6234ff0b5fcabd974779a66 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:32.523165 16957 raft_consensus.cc:1261] T 78f81fe4c6234ff0b5fcabd974779a66 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:32.523451 19108 raft_consensus.cc:1261] T 78f81fe4c6234ff0b5fcabd974779a66 P 838ef968bb45484bb27f5af6f3989bae [term 1 LEARNER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:32.523851 17353 raft_consensus.cc:1261] T 78f81fe4c6234ff0b5fcabd974779a66 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:32.523978 20057 consensus_queue.cc:1035] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:32.524263 20474 consensus_queue.cc:1035] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:32.524433 20549 consensus_queue.cc:1035] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:32.524704 20550 consensus_queue.cc:1035] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:32.525115 19896 consensus_queue.cc:1035] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:32.526863 17486 raft_consensus.cc:1261] T 78f81fe4c6234ff0b5fcabd974779a66 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:32.527246 19676 consensus_queue.cc:1035] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:32.528263 17223 raft_consensus.cc:1261] T 78f81fe4c6234ff0b5fcabd974779a66 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:32.528712 20475 consensus_queue.cc:1035] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:32.529157 20057 raft_consensus.cc:2932] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:32.531008 17620 raft_consensus.cc:2932] T 78f81fe4c6234ff0b5fcabd974779a66 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:32.531108 17751 raft_consensus.cc:2932] T 78f81fe4c6234ff0b5fcabd974779a66 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:32.531298 17486 raft_consensus.cc:2932] T 78f81fe4c6234ff0b5fcabd974779a66 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:32.533107 17223 raft_consensus.cc:2932] T 78f81fe4c6234ff0b5fcabd974779a66 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:32.534801 16812 catalog_manager.cc:5484] T 78f81fe4c6234ff0b5fcabd974779a66 P 16d437fe629643948c95d628926d5ffa reported cstate change: config changed from index 5 to 6, 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "89dd09f8aaee4b249ae3ce924a422a67" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:32.536491 16958 raft_consensus.cc:2932] T 78f81fe4c6234ff0b5fcabd974779a66 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:32.538875 19108 raft_consensus.cc:2932] T 78f81fe4c6234ff0b5fcabd974779a66 P 838ef968bb45484bb27f5af6f3989bae [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:32.540606 17353 raft_consensus.cc:2932] T 78f81fe4c6234ff0b5fcabd974779a66 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:32.545670 17091 consensus_queue.cc:237] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } }
I20240503 02:00:32.547741 17620 raft_consensus.cc:1261] T 78f81fe4c6234ff0b5fcabd974779a66 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:32.547890 17750 raft_consensus.cc:1261] T 78f81fe4c6234ff0b5fcabd974779a66 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:32.548281 19108 raft_consensus.cc:1261] T 78f81fe4c6234ff0b5fcabd974779a66 P 838ef968bb45484bb27f5af6f3989bae [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:32.548233 20475 consensus_queue.cc:1035] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:32.548431 19896 consensus_queue.cc:1035] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:32.548661 20549 consensus_queue.cc:1035] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:32.548846 16958 raft_consensus.cc:1261] T 78f81fe4c6234ff0b5fcabd974779a66 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:32.549189 20475 consensus_queue.cc:1035] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:32.551108 17486 raft_consensus.cc:1261] T 78f81fe4c6234ff0b5fcabd974779a66 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:32.551499 20549 consensus_queue.cc:1035] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:32.552268 17223 raft_consensus.cc:1261] T 78f81fe4c6234ff0b5fcabd974779a66 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:32.552795 20057 consensus_queue.cc:1035] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:32.553117 20475 raft_consensus.cc:2932] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:32.554754 17486 raft_consensus.cc:2932] T 78f81fe4c6234ff0b5fcabd974779a66 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:32.554944 16805 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 78f81fe4c6234ff0b5fcabd974779a66 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:00:32.554898 17223 raft_consensus.cc:2932] T 78f81fe4c6234ff0b5fcabd974779a66 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:32.556134 17619 raft_consensus.cc:2932] T 78f81fe4c6234ff0b5fcabd974779a66 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:32.556214 17750 raft_consensus.cc:2932] T 78f81fe4c6234ff0b5fcabd974779a66 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:32.558262 19108 raft_consensus.cc:2932] T 78f81fe4c6234ff0b5fcabd974779a66 P 838ef968bb45484bb27f5af6f3989bae [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:32.559484 16811 catalog_manager.cc:5484] T 78f81fe4c6234ff0b5fcabd974779a66 P 5b8291b26a7f448b8636187d1caf86d0 reported cstate change: config changed from index 6 to 7, VOTER d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "89dd09f8aaee4b249ae3ce924a422a67" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:32.562203 16958 raft_consensus.cc:2932] T 78f81fe4c6234ff0b5fcabd974779a66 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:32.562844 17335 tablet_service.cc:1514] Processing DeleteTablet for tablet 78f81fe4c6234ff0b5fcabd974779a66 with delete_type TABLET_DATA_TOMBSTONED (TS d43ed6b8663947d99a72cd922cc323aa not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:44362
I20240503 02:00:32.563167 20547 tablet_replica.cc:330] T 78f81fe4c6234ff0b5fcabd974779a66 P d43ed6b8663947d99a72cd922cc323aa: stopping tablet replica
I20240503 02:00:32.563350 20547 raft_consensus.cc:2229] T 78f81fe4c6234ff0b5fcabd974779a66 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:32.563616 20547 raft_consensus.cc:2258] T 78f81fe4c6234ff0b5fcabd974779a66 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:32.564486 20547 ts_tablet_manager.cc:1875] T 78f81fe4c6234ff0b5fcabd974779a66 P d43ed6b8663947d99a72cd922cc323aa: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:32.567225 20547 ts_tablet_manager.cc:1888] T 78f81fe4c6234ff0b5fcabd974779a66 P d43ed6b8663947d99a72cd922cc323aa: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 02:00:32.567303 20547 log.cc:1196] T 78f81fe4c6234ff0b5fcabd974779a66 P d43ed6b8663947d99a72cd922cc323aa: Deleting WAL directory at /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/78f81fe4c6234ff0b5fcabd974779a66
I20240503 02:00:32.567847 16803 catalog_manager.cc:4830] TS d43ed6b8663947d99a72cd922cc323aa (127.0.99.132:40023): tablet 78f81fe4c6234ff0b5fcabd974779a66 (table rebalance_test_table_2 [id=9f4cafd5cdb041c1bbe1592f04370c0b]) successfully deleted
I20240503 02:00:32.685205 20475 raft_consensus.cc:1050] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67: attempting to promote NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d to VOTER
I20240503 02:00:32.685819 20475 consensus_queue.cc:237] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } }
I20240503 02:00:32.687842 19370 raft_consensus.cc:1261] T 30abfada8ab44449b9a4237c57093b73 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 LEARNER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:32.688017 17353 raft_consensus.cc:1261] T 30abfada8ab44449b9a4237c57093b73 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:32.688089 16958 raft_consensus.cc:1261] T 30abfada8ab44449b9a4237c57093b73 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:32.688524 20550 consensus_queue.cc:1035] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:32.688745 17223 raft_consensus.cc:1261] T 30abfada8ab44449b9a4237c57093b73 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:32.689301 20550 consensus_queue.cc:1035] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:32.689429 19676 consensus_queue.cc:1035] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:32.689529 20474 consensus_queue.cc:1035] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:32.690647 17486 raft_consensus.cc:1261] T 30abfada8ab44449b9a4237c57093b73 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:32.691215 17619 raft_consensus.cc:1261] T 30abfada8ab44449b9a4237c57093b73 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:32.692095 17750 raft_consensus.cc:1261] T 30abfada8ab44449b9a4237c57093b73 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:32.692917 19676 consensus_queue.cc:1035] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:32.693117 20550 consensus_queue.cc:1035] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:32.693256 19676 consensus_queue.cc:1035] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:32.695778 20550 raft_consensus.cc:2932] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:32.697571 17486 raft_consensus.cc:2932] T 30abfada8ab44449b9a4237c57093b73 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:32.698496 17619 raft_consensus.cc:2932] T 30abfada8ab44449b9a4237c57093b73 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:32.699788 16958 raft_consensus.cc:2932] T 30abfada8ab44449b9a4237c57093b73 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:32.700371 16813 catalog_manager.cc:5484] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 reported cstate change: config changed from index 5 to 6, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "89dd09f8aaee4b249ae3ce924a422a67" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:00:32.701630 19370 raft_consensus.cc:2932] T 30abfada8ab44449b9a4237c57093b73 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:32.701800 17353 raft_consensus.cc:2932] T 30abfada8ab44449b9a4237c57093b73 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:32.705821 17223 raft_consensus.cc:2932] T 30abfada8ab44449b9a4237c57093b73 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:32.709184 17750 raft_consensus.cc:2932] T 30abfada8ab44449b9a4237c57093b73 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:32.710948 17091 consensus_queue.cc:237] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } }
I20240503 02:00:32.713487 19370 raft_consensus.cc:1261] T 30abfada8ab44449b9a4237c57093b73 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:32.713593 16958 raft_consensus.cc:1261] T 30abfada8ab44449b9a4237c57093b73 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:32.713757 17353 raft_consensus.cc:1261] T 30abfada8ab44449b9a4237c57093b73 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:32.713959 17486 raft_consensus.cc:1261] T 30abfada8ab44449b9a4237c57093b73 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:32.714156 17223 raft_consensus.cc:1261] T 30abfada8ab44449b9a4237c57093b73 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:32.714218 20475 consensus_queue.cc:1035] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:32.714366 19896 consensus_queue.cc:1035] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:32.714489 20475 consensus_queue.cc:1035] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:32.714709 20474 consensus_queue.cc:1035] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:32.715206 20474 consensus_queue.cc:1035] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:32.715363 17750 raft_consensus.cc:1261] T 30abfada8ab44449b9a4237c57093b73 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:32.715729 20474 consensus_queue.cc:1035] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:32.717653 20474 raft_consensus.cc:2932] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:32.719043 17223 raft_consensus.cc:2932] T 30abfada8ab44449b9a4237c57093b73 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:32.718657 16958 raft_consensus.cc:2932] T 30abfada8ab44449b9a4237c57093b73 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:32.720049 17750 raft_consensus.cc:2932] T 30abfada8ab44449b9a4237c57093b73 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:32.721467 17486 raft_consensus.cc:2932] T 30abfada8ab44449b9a4237c57093b73 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:32.721691 16805 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 30abfada8ab44449b9a4237c57093b73 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:00:32.722437 19370 raft_consensus.cc:2932] T 30abfada8ab44449b9a4237c57093b73 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:32.722637 16818 catalog_manager.cc:5484] T 30abfada8ab44449b9a4237c57093b73 P 6398c8993c3c45dcaab25b42142d5671 reported cstate change: config changed from index 6 to 7, VOTER c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134) evicted. New cstate: current_term: 1 leader_uuid: "89dd09f8aaee4b249ae3ce924a422a67" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:32.722483 17353 raft_consensus.cc:2932] T 30abfada8ab44449b9a4237c57093b73 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:32.729408 17598 tablet_service.cc:1514] Processing DeleteTablet for tablet 30abfada8ab44449b9a4237c57093b73 with delete_type TABLET_DATA_TOMBSTONED (TS c843193b801b4f1a88f630ed3ed0f375 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:51030
I20240503 02:00:32.729723 20546 tablet_replica.cc:330] T 30abfada8ab44449b9a4237c57093b73 P c843193b801b4f1a88f630ed3ed0f375: stopping tablet replica
I20240503 02:00:32.729964 20546 raft_consensus.cc:2229] T 30abfada8ab44449b9a4237c57093b73 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:32.730232 20546 raft_consensus.cc:2258] T 30abfada8ab44449b9a4237c57093b73 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:32.731596 20546 ts_tablet_manager.cc:1875] T 30abfada8ab44449b9a4237c57093b73 P c843193b801b4f1a88f630ed3ed0f375: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:32.733565 20546 ts_tablet_manager.cc:1888] T 30abfada8ab44449b9a4237c57093b73 P c843193b801b4f1a88f630ed3ed0f375: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 02:00:32.733747 20546 log.cc:1196] T 30abfada8ab44449b9a4237c57093b73 P c843193b801b4f1a88f630ed3ed0f375: Deleting WAL directory at /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/30abfada8ab44449b9a4237c57093b73
I20240503 02:00:32.734875 16802 catalog_manager.cc:4830] TS c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134:44343): tablet 30abfada8ab44449b9a4237c57093b73 (table rebalance_test_table_1 [id=38eeb06923e14f2ead405ffb039fd231]) successfully deleted
I20240503 02:00:32.766049 19620 raft_consensus.cc:1050] T 42a900125241432a8bb31dad0b940968 P 16d437fe629643948c95d628926d5ffa: attempting to promote NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d to VOTER
I20240503 02:00:32.766583 19620 consensus_queue.cc:237] T 42a900125241432a8bb31dad0b940968 P 16d437fe629643948c95d628926d5ffa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } }
I20240503 02:00:32.768550 17091 raft_consensus.cc:1261] T 42a900125241432a8bb31dad0b940968 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:32.768587 17619 raft_consensus.cc:1261] T 42a900125241432a8bb31dad0b940968 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:32.768728 17750 raft_consensus.cc:1261] T 42a900125241432a8bb31dad0b940968 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:32.769316 17353 raft_consensus.cc:1261] T 42a900125241432a8bb31dad0b940968 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:32.769482 19370 raft_consensus.cc:1261] T 42a900125241432a8bb31dad0b940968 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 LEARNER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:32.769435 20515 consensus_queue.cc:1035] T 42a900125241432a8bb31dad0b940968 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:32.769672 16958 raft_consensus.cc:1261] T 42a900125241432a8bb31dad0b940968 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:32.769752 19620 consensus_queue.cc:1035] T 42a900125241432a8bb31dad0b940968 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:32.770545 20040 consensus_queue.cc:1035] T 42a900125241432a8bb31dad0b940968 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:32.770805 17223 raft_consensus.cc:1261] T 42a900125241432a8bb31dad0b940968 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:32.770926 20517 consensus_queue.cc:1035] T 42a900125241432a8bb31dad0b940968 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:32.771055 20515 consensus_queue.cc:1035] T 42a900125241432a8bb31dad0b940968 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:32.771210 20519 consensus_queue.cc:1035] T 42a900125241432a8bb31dad0b940968 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:32.772521 20515 consensus_queue.cc:1035] T 42a900125241432a8bb31dad0b940968 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:32.774904 20515 raft_consensus.cc:2932] T 42a900125241432a8bb31dad0b940968 P 16d437fe629643948c95d628926d5ffa [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:32.775537 17619 raft_consensus.cc:2932] T 42a900125241432a8bb31dad0b940968 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:32.776033 17356 raft_consensus.cc:2932] T 42a900125241432a8bb31dad0b940968 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:32.777392 16958 raft_consensus.cc:2932] T 42a900125241432a8bb31dad0b940968 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:32.777846 17750 raft_consensus.cc:2932] T 42a900125241432a8bb31dad0b940968 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:32.779227 19370 raft_consensus.cc:2932] T 42a900125241432a8bb31dad0b940968 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:32.781301 16809 catalog_manager.cc:5484] T 42a900125241432a8bb31dad0b940968 P 16d437fe629643948c95d628926d5ffa reported cstate change: config changed from index 5 to 6, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "16d437fe629643948c95d628926d5ffa" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:00:32.785832 17223 raft_consensus.cc:2932] T 42a900125241432a8bb31dad0b940968 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:32.787643 17090 raft_consensus.cc:2932] T 42a900125241432a8bb31dad0b940968 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:32.792361 17486 consensus_queue.cc:237] T 42a900125241432a8bb31dad0b940968 P 16d437fe629643948c95d628926d5ffa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } }
I20240503 02:00:32.794237 17750 raft_consensus.cc:1261] T 42a900125241432a8bb31dad0b940968 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:32.794322 17223 raft_consensus.cc:1261] T 42a900125241432a8bb31dad0b940968 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:32.794340 19370 raft_consensus.cc:1261] T 42a900125241432a8bb31dad0b940968 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:32.794380 17090 raft_consensus.cc:1261] T 42a900125241432a8bb31dad0b940968 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:32.794624 17619 raft_consensus.cc:1261] T 42a900125241432a8bb31dad0b940968 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:32.794734 19620 consensus_queue.cc:1035] T 42a900125241432a8bb31dad0b940968 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:32.794939 20519 consensus_queue.cc:1035] T 42a900125241432a8bb31dad0b940968 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:32.795193 20515 consensus_queue.cc:1035] T 42a900125241432a8bb31dad0b940968 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:32.795689 16958 raft_consensus.cc:1261] T 42a900125241432a8bb31dad0b940968 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:32.795688 20518 consensus_queue.cc:1035] T 42a900125241432a8bb31dad0b940968 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:32.795917 19620 consensus_queue.cc:1035] T 42a900125241432a8bb31dad0b940968 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:32.798293 20518 consensus_queue.cc:1035] T 42a900125241432a8bb31dad0b940968 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:32.799245 20040 raft_consensus.cc:2932] T 42a900125241432a8bb31dad0b940968 P 16d437fe629643948c95d628926d5ffa [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:32.799844 17223 raft_consensus.cc:2932] T 42a900125241432a8bb31dad0b940968 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:32.801446 17750 raft_consensus.cc:2932] T 42a900125241432a8bb31dad0b940968 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:32.802834 19370 raft_consensus.cc:2932] T 42a900125241432a8bb31dad0b940968 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:32.803035 17091 raft_consensus.cc:2932] T 42a900125241432a8bb31dad0b940968 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:32.804360 16813 catalog_manager.cc:5484] T 42a900125241432a8bb31dad0b940968 P 5b8291b26a7f448b8636187d1caf86d0 reported cstate change: config changed from index 6 to 7, VOTER d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "16d437fe629643948c95d628926d5ffa" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:32.805797 17619 raft_consensus.cc:2932] T 42a900125241432a8bb31dad0b940968 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:32.806154 16802 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 42a900125241432a8bb31dad0b940968 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:00:32.800668 16958 raft_consensus.cc:2932] T 42a900125241432a8bb31dad0b940968 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:32.809790 17335 tablet_service.cc:1514] Processing DeleteTablet for tablet 42a900125241432a8bb31dad0b940968 with delete_type TABLET_DATA_TOMBSTONED (TS d43ed6b8663947d99a72cd922cc323aa not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:44362
I20240503 02:00:32.810089 20547 tablet_replica.cc:330] T 42a900125241432a8bb31dad0b940968 P d43ed6b8663947d99a72cd922cc323aa: stopping tablet replica
I20240503 02:00:32.810283 20547 raft_consensus.cc:2229] T 42a900125241432a8bb31dad0b940968 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:32.810478 20547 raft_consensus.cc:2258] T 42a900125241432a8bb31dad0b940968 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:32.811305 20547 ts_tablet_manager.cc:1875] T 42a900125241432a8bb31dad0b940968 P d43ed6b8663947d99a72cd922cc323aa: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:32.813680 20547 ts_tablet_manager.cc:1888] T 42a900125241432a8bb31dad0b940968 P d43ed6b8663947d99a72cd922cc323aa: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 02:00:32.813759 20547 log.cc:1196] T 42a900125241432a8bb31dad0b940968 P d43ed6b8663947d99a72cd922cc323aa: Deleting WAL directory at /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/42a900125241432a8bb31dad0b940968
I20240503 02:00:32.814363 16803 catalog_manager.cc:4830] TS d43ed6b8663947d99a72cd922cc323aa (127.0.99.132:40023): tablet 42a900125241432a8bb31dad0b940968 (table rebalance_test_table_5 [id=90cb01160ab843bbae459ab57f0425ed]) successfully deleted
I20240503 02:00:32.838004 17091 consensus_queue.cc:237] T e245c9cf642043cfbb3b87114beeb002 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } }
I20240503 02:00:32.839754 19108 raft_consensus.cc:1261] T e245c9cf642043cfbb3b87114beeb002 P 838ef968bb45484bb27f5af6f3989bae [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:32.839829 17223 raft_consensus.cc:1261] T e245c9cf642043cfbb3b87114beeb002 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:32.840451 20474 consensus_queue.cc:1035] T e245c9cf642043cfbb3b87114beeb002 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:32.840708 17750 raft_consensus.cc:1261] T e245c9cf642043cfbb3b87114beeb002 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:32.840814 20474 consensus_queue.cc:1035] T e245c9cf642043cfbb3b87114beeb002 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:32.841246 17486 raft_consensus.cc:1261] T e245c9cf642043cfbb3b87114beeb002 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:32.841269 20474 consensus_queue.cc:1035] T e245c9cf642043cfbb3b87114beeb002 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:32.841826 17356 raft_consensus.cc:1261] T e245c9cf642043cfbb3b87114beeb002 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:32.842255 20474 consensus_queue.cc:1035] T e245c9cf642043cfbb3b87114beeb002 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:32.842453 16958 raft_consensus.cc:1261] T e245c9cf642043cfbb3b87114beeb002 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:32.843199 20475 consensus_queue.cc:1035] T e245c9cf642043cfbb3b87114beeb002 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:32.844341 19676 consensus_queue.cc:1035] T e245c9cf642043cfbb3b87114beeb002 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:32.847079 20475 raft_consensus.cc:2932] T e245c9cf642043cfbb3b87114beeb002 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:32.850281 17750 raft_consensus.cc:2932] T e245c9cf642043cfbb3b87114beeb002 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:32.850499 17486 raft_consensus.cc:2932] T e245c9cf642043cfbb3b87114beeb002 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:32.850996 16817 catalog_manager.cc:5484] T e245c9cf642043cfbb3b87114beeb002 P 89dd09f8aaee4b249ae3ce924a422a67 reported cstate change: config changed from index 6 to 7, VOTER c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134) evicted. New cstate: current_term: 1 leader_uuid: "89dd09f8aaee4b249ae3ce924a422a67" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:00:32.852694 17223 raft_consensus.cc:2932] T e245c9cf642043cfbb3b87114beeb002 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:32.853375 17356 raft_consensus.cc:2932] T e245c9cf642043cfbb3b87114beeb002 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:32.855283 17598 tablet_service.cc:1514] Processing DeleteTablet for tablet e245c9cf642043cfbb3b87114beeb002 with delete_type TABLET_DATA_TOMBSTONED (TS c843193b801b4f1a88f630ed3ed0f375 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:51030
I20240503 02:00:32.855626 19108 raft_consensus.cc:2932] T e245c9cf642043cfbb3b87114beeb002 P 838ef968bb45484bb27f5af6f3989bae [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:32.856127 20546 tablet_replica.cc:330] T e245c9cf642043cfbb3b87114beeb002 P c843193b801b4f1a88f630ed3ed0f375: stopping tablet replica
I20240503 02:00:32.856351 20546 raft_consensus.cc:2229] T e245c9cf642043cfbb3b87114beeb002 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:32.856525 20546 raft_consensus.cc:2258] T e245c9cf642043cfbb3b87114beeb002 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:32.858341 20546 ts_tablet_manager.cc:1875] T e245c9cf642043cfbb3b87114beeb002 P c843193b801b4f1a88f630ed3ed0f375: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:32.860888 16805 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e245c9cf642043cfbb3b87114beeb002 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:00:32.860621 20546 ts_tablet_manager.cc:1888] T e245c9cf642043cfbb3b87114beeb002 P c843193b801b4f1a88f630ed3ed0f375: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 02:00:32.862355 20546 log.cc:1196] T e245c9cf642043cfbb3b87114beeb002 P c843193b801b4f1a88f630ed3ed0f375: Deleting WAL directory at /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/e245c9cf642043cfbb3b87114beeb002
I20240503 02:00:32.862948 16802 catalog_manager.cc:4830] TS c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134:44343): tablet e245c9cf642043cfbb3b87114beeb002 (table rebalance_test_table_2 [id=9f4cafd5cdb041c1bbe1592f04370c0b]) successfully deleted
I20240503 02:00:32.863029 16958 raft_consensus.cc:2932] T e245c9cf642043cfbb3b87114beeb002 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:32.866014 17750 consensus_queue.cc:237] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:32.867939 17223 raft_consensus.cc:1261] T 78d106d705df4089a4dfae4d5dc533cc P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:32.868077 16958 raft_consensus.cc:1261] T 78d106d705df4089a4dfae4d5dc533cc P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:32.868085 17091 raft_consensus.cc:1261] T 78d106d705df4089a4dfae4d5dc533cc P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:32.868193 17356 raft_consensus.cc:1261] T 78d106d705df4089a4dfae4d5dc533cc P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:32.868369 20007 consensus_queue.cc:1035] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:32.868598 20612 consensus_queue.cc:1035] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:32.868999 20604 consensus_queue.cc:1035] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:32.869215 17619 raft_consensus.cc:1261] T 78d106d705df4089a4dfae4d5dc533cc P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:32.869307 17486 raft_consensus.cc:1261] T 78d106d705df4089a4dfae4d5dc533cc P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:32.869781 20614 consensus_queue.cc:1035] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:32.870747 20614 consensus_queue.cc:1035] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:32.870929 20613 consensus_queue.cc:1035] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:32.873921 20604 raft_consensus.cc:2932] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } }
I20240503 02:00:32.874574 17619 raft_consensus.cc:2932] T 78d106d705df4089a4dfae4d5dc533cc P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } }
I20240503 02:00:32.876425 17091 raft_consensus.cc:2932] T 78d106d705df4089a4dfae4d5dc533cc P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } }
I20240503 02:00:32.876829 17486 raft_consensus.cc:2932] T 78d106d705df4089a4dfae4d5dc533cc P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } }
I20240503 02:00:32.877425 16817 catalog_manager.cc:5484] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } health_report { overall_health: HEALTHY } } }
I20240503 02:00:32.879920 17223 raft_consensus.cc:2932] T 78d106d705df4089a4dfae4d5dc533cc P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } }
I20240503 02:00:32.879958 16958 raft_consensus.cc:2932] T 78d106d705df4089a4dfae4d5dc533cc P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } }
I20240503 02:00:32.880110 17356 raft_consensus.cc:2932] T 78d106d705df4089a4dfae4d5dc533cc P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } }
I20240503 02:00:32.886282 17750 consensus_queue.cc:237] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: true } }
I20240503 02:00:32.888159 17486 raft_consensus.cc:1261] T 78d106d705df4089a4dfae4d5dc533cc P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:32.888218 17619 raft_consensus.cc:1261] T 78d106d705df4089a4dfae4d5dc533cc P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:32.888520 17091 raft_consensus.cc:1261] T 78d106d705df4089a4dfae4d5dc533cc P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:32.889060 17356 raft_consensus.cc:1261] T 78d106d705df4089a4dfae4d5dc533cc P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:32.889086 20604 consensus_queue.cc:1035] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:32.889273 20614 consensus_queue.cc:1035] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:32.889415 17223 raft_consensus.cc:1261] T 78d106d705df4089a4dfae4d5dc533cc P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240503 02:00:32.889691 17688 consensus_peers.cc:485] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 9564311af29a4493831a8db658a085c4 (127.0.99.138:33181): Couldn't send request to peer 9564311af29a4493831a8db658a085c4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 78d106d705df4089a4dfae4d5dc533cc. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:32.890198 20612 consensus_queue.cc:1035] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:32.890563 20612 consensus_queue.cc:1035] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:32.890729 20614 consensus_queue.cc:1035] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:32.891443 16958 raft_consensus.cc:1261] T 78d106d705df4089a4dfae4d5dc533cc P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:32.891927 20007 consensus_queue.cc:1035] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:32.894095 20613 raft_consensus.cc:2932] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9564311af29a4493831a8db658a085c4 (127.0.99.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: true } } }
I20240503 02:00:32.895658 17222 raft_consensus.cc:2932] T 78d106d705df4089a4dfae4d5dc533cc P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9564311af29a4493831a8db658a085c4 (127.0.99.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: true } } }
I20240503 02:00:32.895902 16958 raft_consensus.cc:2932] T 78d106d705df4089a4dfae4d5dc533cc P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9564311af29a4493831a8db658a085c4 (127.0.99.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: true } } }
I20240503 02:00:32.897771 17619 raft_consensus.cc:2932] T 78d106d705df4089a4dfae4d5dc533cc P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9564311af29a4493831a8db658a085c4 (127.0.99.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: true } } }
I20240503 02:00:32.898020 16802 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 78d106d705df4089a4dfae4d5dc533cc with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 02:00:32.898959 16815 catalog_manager.cc:5484] T 78d106d705df4089a4dfae4d5dc533cc P 5b8291b26a7f448b8636187d1caf86d0 reported cstate change: config changed from index 4 to 5, NON_VOTER 9564311af29a4493831a8db658a085c4 (127.0.99.138) added. New cstate: current_term: 1 leader_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: true } } }
I20240503 02:00:32.899864 17091 raft_consensus.cc:2932] T 78d106d705df4089a4dfae4d5dc533cc P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9564311af29a4493831a8db658a085c4 (127.0.99.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: true } } }
I20240503 02:00:32.900213 17486 raft_consensus.cc:2932] T 78d106d705df4089a4dfae4d5dc533cc P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9564311af29a4493831a8db658a085c4 (127.0.99.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: true } } }
I20240503 02:00:32.903115 17356 raft_consensus.cc:2932] T 78d106d705df4089a4dfae4d5dc533cc P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9564311af29a4493831a8db658a085c4 (127.0.99.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: true } } }
I20240503 02:00:32.909709 17091 consensus_queue.cc:237] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:32.911378 17486 raft_consensus.cc:1261] T 81b54f3bcf34485eb11b1bc954a2bd8f P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:32.911794 17356 raft_consensus.cc:1261] T 81b54f3bcf34485eb11b1bc954a2bd8f P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:32.911769 20549 consensus_queue.cc:1035] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:32.911834 17750 raft_consensus.cc:1261] T 81b54f3bcf34485eb11b1bc954a2bd8f P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:32.912580 17619 raft_consensus.cc:1261] T 81b54f3bcf34485eb11b1bc954a2bd8f P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:32.912626 16958 raft_consensus.cc:1261] T 81b54f3bcf34485eb11b1bc954a2bd8f P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:32.912904 20474 consensus_queue.cc:1035] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:32.913286 20474 consensus_queue.cc:1035] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:32.913422 20549 consensus_queue.cc:1035] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:32.913764 20057 consensus_queue.cc:1035] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:32.915148 17222 raft_consensus.cc:1261] T 81b54f3bcf34485eb11b1bc954a2bd8f P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:32.915513 19676 consensus_queue.cc:1035] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:32.917245 19676 raft_consensus.cc:2932] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } }
I20240503 02:00:32.917820 17222 raft_consensus.cc:2932] T 81b54f3bcf34485eb11b1bc954a2bd8f P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } }
I20240503 02:00:32.919369 17486 raft_consensus.cc:2932] T 81b54f3bcf34485eb11b1bc954a2bd8f P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } }
I20240503 02:00:32.919912 17619 raft_consensus.cc:2932] T 81b54f3bcf34485eb11b1bc954a2bd8f P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } }
I20240503 02:00:32.919971 17750 raft_consensus.cc:2932] T 81b54f3bcf34485eb11b1bc954a2bd8f P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } }
I20240503 02:00:32.920245 16958 raft_consensus.cc:2932] T 81b54f3bcf34485eb11b1bc954a2bd8f P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } }
I20240503 02:00:32.921643 17356 raft_consensus.cc:2932] T 81b54f3bcf34485eb11b1bc954a2bd8f P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } }
I20240503 02:00:32.922341 16817 catalog_manager.cc:5484] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "89dd09f8aaee4b249ae3ce924a422a67" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:32.929144 17091 consensus_queue.cc:237] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } }
I20240503 02:00:32.931272 17222 raft_consensus.cc:1261] T 81b54f3bcf34485eb11b1bc954a2bd8f P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240503 02:00:32.931823 17026 consensus_peers.cc:485] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 81b54f3bcf34485eb11b1bc954a2bd8f. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:32.931851 17356 raft_consensus.cc:1261] T 81b54f3bcf34485eb11b1bc954a2bd8f P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:32.931951 20474 consensus_queue.cc:1035] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:32.932260 20057 consensus_queue.cc:1035] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:32.932312 17619 raft_consensus.cc:1261] T 81b54f3bcf34485eb11b1bc954a2bd8f P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:32.932613 17486 raft_consensus.cc:1261] T 81b54f3bcf34485eb11b1bc954a2bd8f P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:32.932987 20057 consensus_queue.cc:1035] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:32.933115 20474 consensus_queue.cc:1035] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:32.936815 16958 raft_consensus.cc:1261] T 81b54f3bcf34485eb11b1bc954a2bd8f P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:32.937213 20474 raft_consensus.cc:2932] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } }
I20240503 02:00:32.937649 20057 consensus_queue.cc:1035] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:32.938658 16958 raft_consensus.cc:2932] T 81b54f3bcf34485eb11b1bc954a2bd8f P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } }
I20240503 02:00:32.939107 17750 raft_consensus.cc:1261] T 81b54f3bcf34485eb11b1bc954a2bd8f P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:32.939661 20549 consensus_queue.cc:1035] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:32.940237 16805 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 81b54f3bcf34485eb11b1bc954a2bd8f with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 02:00:32.940320 16813 catalog_manager.cc:5484] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 reported cstate change: config changed from index 4 to 5, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New cstate: current_term: 1 leader_uuid: "89dd09f8aaee4b249ae3ce924a422a67" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:32.940531 17222 raft_consensus.cc:2932] T 81b54f3bcf34485eb11b1bc954a2bd8f P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } }
I20240503 02:00:32.940795 17486 raft_consensus.cc:2932] T 81b54f3bcf34485eb11b1bc954a2bd8f P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } }
I20240503 02:00:32.940850 17356 raft_consensus.cc:2932] T 81b54f3bcf34485eb11b1bc954a2bd8f P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } }
I20240503 02:00:32.942314 17619 raft_consensus.cc:2932] T 81b54f3bcf34485eb11b1bc954a2bd8f P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } }
I20240503 02:00:32.948632 17486 consensus_queue.cc:237] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } }
I20240503 02:00:32.949839 17750 raft_consensus.cc:2932] T 81b54f3bcf34485eb11b1bc954a2bd8f P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } }
I20240503 02:00:32.950682 16958 raft_consensus.cc:1261] T ec69ad86be3d4793b10aca98dc4c1736 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:32.950858 17751 raft_consensus.cc:1261] T ec69ad86be3d4793b10aca98dc4c1736 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:32.950871 17222 raft_consensus.cc:1261] T ec69ad86be3d4793b10aca98dc4c1736 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:32.951092 20516 consensus_queue.cc:1035] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:32.951267 19620 consensus_queue.cc:1035] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:32.951481 20517 consensus_queue.cc:1035] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:32.952874 17619 raft_consensus.cc:1261] T ec69ad86be3d4793b10aca98dc4c1736 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:32.953334 20516 consensus_queue.cc:1035] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:32.953781 17091 raft_consensus.cc:1261] T ec69ad86be3d4793b10aca98dc4c1736 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:32.954174 20515 consensus_queue.cc:1035] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:32.956094 20517 raft_consensus.cc:2932] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } }
I20240503 02:00:32.956741 17356 raft_consensus.cc:1261] T ec69ad86be3d4793b10aca98dc4c1736 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:32.957212 20515 consensus_queue.cc:1035] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:32.958601 17356 raft_consensus.cc:2932] T ec69ad86be3d4793b10aca98dc4c1736 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } }
I20240503 02:00:32.959506 17091 raft_consensus.cc:2932] T ec69ad86be3d4793b10aca98dc4c1736 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } }
I20240503 02:00:32.959815 17222 raft_consensus.cc:2932] T ec69ad86be3d4793b10aca98dc4c1736 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } }
I20240503 02:00:32.961599 16811 catalog_manager.cc:5484] T ec69ad86be3d4793b10aca98dc4c1736 P d43ed6b8663947d99a72cd922cc323aa reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "16d437fe629643948c95d628926d5ffa" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } }
I20240503 02:00:32.965607 17751 raft_consensus.cc:2932] T ec69ad86be3d4793b10aca98dc4c1736 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } }
I20240503 02:00:32.965755 16958 raft_consensus.cc:2932] T ec69ad86be3d4793b10aca98dc4c1736 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } }
I20240503 02:00:32.965749 17619 raft_consensus.cc:2932] T ec69ad86be3d4793b10aca98dc4c1736 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } }
I20240503 02:00:32.974875 17486 consensus_queue.cc:237] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } }
I20240503 02:00:32.976800 17356 raft_consensus.cc:1261] T ec69ad86be3d4793b10aca98dc4c1736 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:32.977190 16958 raft_consensus.cc:1261] T ec69ad86be3d4793b10aca98dc4c1736 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:32.977437 17091 raft_consensus.cc:1261] T ec69ad86be3d4793b10aca98dc4c1736 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:32.977511 17751 raft_consensus.cc:1261] T ec69ad86be3d4793b10aca98dc4c1736 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:32.977770 20516 consensus_queue.cc:1035] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:32.977926 20519 consensus_queue.cc:1035] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 02:00:32.978693 17422 consensus_peers.cc:485] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ec69ad86be3d4793b10aca98dc4c1736. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:32.979074 17222 raft_consensus.cc:1261] T ec69ad86be3d4793b10aca98dc4c1736 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:32.979424 19620 consensus_queue.cc:1035] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:32.980041 20519 consensus_queue.cc:1035] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:32.980369 20519 consensus_queue.cc:1035] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:32.981575 17619 raft_consensus.cc:1261] T ec69ad86be3d4793b10aca98dc4c1736 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:32.981936 20516 raft_consensus.cc:2932] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } }
I20240503 02:00:32.982160 20515 consensus_queue.cc:1035] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:32.982564 17751 raft_consensus.cc:2932] T ec69ad86be3d4793b10aca98dc4c1736 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } }
I20240503 02:00:32.983811 16802 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet ec69ad86be3d4793b10aca98dc4c1736 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 02:00:32.984413 17091 raft_consensus.cc:2932] T ec69ad86be3d4793b10aca98dc4c1736 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } }
I20240503 02:00:32.984936 17619 raft_consensus.cc:2932] T ec69ad86be3d4793b10aca98dc4c1736 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } }
I20240503 02:00:32.985626 16813 catalog_manager.cc:5484] T ec69ad86be3d4793b10aca98dc4c1736 P 002cc3411acb4fdfa0b67c6f57f687a6 reported cstate change: config changed from index 4 to 5, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New cstate: current_term: 1 leader_uuid: "16d437fe629643948c95d628926d5ffa" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } }
I20240503 02:00:32.986990 17222 raft_consensus.cc:2932] T ec69ad86be3d4793b10aca98dc4c1736 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } }
I20240503 02:00:32.989078 17356 raft_consensus.cc:2932] T ec69ad86be3d4793b10aca98dc4c1736 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } }
I20240503 02:00:32.992478 16958 raft_consensus.cc:2932] T ec69ad86be3d4793b10aca98dc4c1736 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } }
I20240503 02:00:33.233282 16958 consensus_queue.cc:237] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:33.235599 17619 raft_consensus.cc:1261] T 885fe0c3a5e04f408e5b02c3c133248a P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer 6398c8993c3c45dcaab25b42142d5671: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:33.235599 17487 raft_consensus.cc:1261] T 885fe0c3a5e04f408e5b02c3c133248a P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Refusing update from remote peer 6398c8993c3c45dcaab25b42142d5671: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:33.235821 17356 raft_consensus.cc:1261] T 885fe0c3a5e04f408e5b02c3c133248a P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Refusing update from remote peer 6398c8993c3c45dcaab25b42142d5671: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:33.235972 17222 raft_consensus.cc:1261] T 885fe0c3a5e04f408e5b02c3c133248a P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer 6398c8993c3c45dcaab25b42142d5671: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:33.236420 17091 raft_consensus.cc:1261] T 885fe0c3a5e04f408e5b02c3c133248a P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer 6398c8993c3c45dcaab25b42142d5671: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:33.236399 20480 consensus_queue.cc:1035] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:33.236460 17751 raft_consensus.cc:1261] T 885fe0c3a5e04f408e5b02c3c133248a P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer 6398c8993c3c45dcaab25b42142d5671: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:33.236708 20094 consensus_queue.cc:1035] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:33.236852 20480 consensus_queue.cc:1035] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:33.237326 20684 consensus_queue.cc:1035] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:33.237493 20094 consensus_queue.cc:1035] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:33.237658 20685 consensus_queue.cc:1035] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671 [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:33.240633 20684 raft_consensus.cc:2932] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } }
I20240503 02:00:33.241111 17222 raft_consensus.cc:2932] T 885fe0c3a5e04f408e5b02c3c133248a P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } }
I20240503 02:00:33.241667 17356 raft_consensus.cc:2932] T 885fe0c3a5e04f408e5b02c3c133248a P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } }
I20240503 02:00:33.242251 17751 raft_consensus.cc:2932] T 885fe0c3a5e04f408e5b02c3c133248a P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } }
I20240503 02:00:33.245114 16817 catalog_manager.cc:5484] T 885fe0c3a5e04f408e5b02c3c133248a P d43ed6b8663947d99a72cd922cc323aa reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "6398c8993c3c45dcaab25b42142d5671" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } }
I20240503 02:00:33.245159 17487 raft_consensus.cc:2932] T 885fe0c3a5e04f408e5b02c3c133248a P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } }
I20240503 02:00:33.245980 17619 raft_consensus.cc:2932] T 885fe0c3a5e04f408e5b02c3c133248a P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } }
I20240503 02:00:33.248781 17091 raft_consensus.cc:2932] T 885fe0c3a5e04f408e5b02c3c133248a P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } }
I20240503 02:00:33.256232 16958 consensus_queue.cc:237] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: true } }
I20240503 02:00:33.258692 17091 raft_consensus.cc:1261] T 885fe0c3a5e04f408e5b02c3c133248a P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer 6398c8993c3c45dcaab25b42142d5671: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:33.258970 17222 raft_consensus.cc:1261] T 885fe0c3a5e04f408e5b02c3c133248a P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer 6398c8993c3c45dcaab25b42142d5671: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:33.259086 20094 consensus_queue.cc:1035] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:33.259371 17356 raft_consensus.cc:1261] T 885fe0c3a5e04f408e5b02c3c133248a P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Refusing update from remote peer 6398c8993c3c45dcaab25b42142d5671: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:33.259974 20686 consensus_queue.cc:1035] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:33.260188 20480 consensus_queue.cc:1035] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:33.260993 17487 raft_consensus.cc:1261] T 885fe0c3a5e04f408e5b02c3c133248a P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Refusing update from remote peer 6398c8993c3c45dcaab25b42142d5671: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:33.261086 17751 raft_consensus.cc:1261] T 885fe0c3a5e04f408e5b02c3c133248a P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer 6398c8993c3c45dcaab25b42142d5671: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:33.261482 20684 consensus_queue.cc:1035] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:33.262477 17619 raft_consensus.cc:1261] T 885fe0c3a5e04f408e5b02c3c133248a P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer 6398c8993c3c45dcaab25b42142d5671: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:33.263622 20685 consensus_queue.cc:1035] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671 [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:33.264142 20685 raft_consensus.cc:2932] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9564311af29a4493831a8db658a085c4 (127.0.99.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: true } } }
I20240503 02:00:33.264679 17222 raft_consensus.cc:2932] T 885fe0c3a5e04f408e5b02c3c133248a P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9564311af29a4493831a8db658a085c4 (127.0.99.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: true } } }
I20240503 02:00:33.265326 20686 consensus_queue.cc:1035] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:33.265903 17487 raft_consensus.cc:2932] T 885fe0c3a5e04f408e5b02c3c133248a P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9564311af29a4493831a8db658a085c4 (127.0.99.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: true } } }
I20240503 02:00:33.267086 17356 raft_consensus.cc:2932] T 885fe0c3a5e04f408e5b02c3c133248a P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9564311af29a4493831a8db658a085c4 (127.0.99.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: true } } }
I20240503 02:00:33.269287 16817 catalog_manager.cc:5484] T 885fe0c3a5e04f408e5b02c3c133248a P 16d437fe629643948c95d628926d5ffa reported cstate change: config changed from index 4 to 5, NON_VOTER 9564311af29a4493831a8db658a085c4 (127.0.99.138) added. New cstate: current_term: 1 leader_uuid: "6398c8993c3c45dcaab25b42142d5671" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: true } } }
W20240503 02:00:33.272048 16896 consensus_peers.cc:485] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671 -> Peer 9564311af29a4493831a8db658a085c4 (127.0.99.138:33181): Couldn't send request to peer 9564311af29a4493831a8db658a085c4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 885fe0c3a5e04f408e5b02c3c133248a. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:33.272018 17091 raft_consensus.cc:2932] T 885fe0c3a5e04f408e5b02c3c133248a P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9564311af29a4493831a8db658a085c4 (127.0.99.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: true } } }
I20240503 02:00:33.273875 16804 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 885fe0c3a5e04f408e5b02c3c133248a with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 02:00:33.275209 17619 raft_consensus.cc:2932] T 885fe0c3a5e04f408e5b02c3c133248a P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9564311af29a4493831a8db658a085c4 (127.0.99.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: true } } }
I20240503 02:00:33.275894 17751 raft_consensus.cc:2932] T 885fe0c3a5e04f408e5b02c3c133248a P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9564311af29a4493831a8db658a085c4 (127.0.99.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: true } } }
I20240503 02:00:33.443279 20716 ts_tablet_manager.cc:902] T ec69ad86be3d4793b10aca98dc4c1736 P 838ef968bb45484bb27f5af6f3989bae: Initiating tablet copy from peer 16d437fe629643948c95d628926d5ffa (127.0.99.133:46581)
I20240503 02:00:33.443830 20716 tablet_copy_client.cc:308] T ec69ad86be3d4793b10aca98dc4c1736 P 838ef968bb45484bb27f5af6f3989bae: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.133:46581
I20240503 02:00:33.444443 17508 tablet_copy_service.cc:140] P 16d437fe629643948c95d628926d5ffa: Received BeginTabletCopySession request for tablet ec69ad86be3d4793b10aca98dc4c1736 from peer 838ef968bb45484bb27f5af6f3989bae ({username='slave'} at 127.0.99.137:37139)
I20240503 02:00:33.444612 17508 tablet_copy_service.cc:161] P 16d437fe629643948c95d628926d5ffa: Beginning new tablet copy session on tablet ec69ad86be3d4793b10aca98dc4c1736 from peer 838ef968bb45484bb27f5af6f3989bae at {username='slave'} at 127.0.99.137:37139: session id = 838ef968bb45484bb27f5af6f3989bae-ec69ad86be3d4793b10aca98dc4c1736
I20240503 02:00:33.445822 17508 tablet_copy_source_session.cc:205] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:33.447108 20716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec69ad86be3d4793b10aca98dc4c1736. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:33.449163 20716 tablet_copy_client.cc:783] T ec69ad86be3d4793b10aca98dc4c1736 P 838ef968bb45484bb27f5af6f3989bae: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:33.449340 20716 tablet_copy_client.cc:647] T ec69ad86be3d4793b10aca98dc4c1736 P 838ef968bb45484bb27f5af6f3989bae: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:33.450974 20716 tablet_copy_client.cc:523] T ec69ad86be3d4793b10aca98dc4c1736 P 838ef968bb45484bb27f5af6f3989bae: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:33.452011 20716 tablet_bootstrap.cc:492] T ec69ad86be3d4793b10aca98dc4c1736 P 838ef968bb45484bb27f5af6f3989bae: Bootstrap starting.
I20240503 02:00:33.470077 20716 tablet_bootstrap.cc:492] T ec69ad86be3d4793b10aca98dc4c1736 P 838ef968bb45484bb27f5af6f3989bae: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:33.470324 20716 tablet_bootstrap.cc:492] T ec69ad86be3d4793b10aca98dc4c1736 P 838ef968bb45484bb27f5af6f3989bae: Bootstrap complete.
I20240503 02:00:33.470469 20716 ts_tablet_manager.cc:1367] T ec69ad86be3d4793b10aca98dc4c1736 P 838ef968bb45484bb27f5af6f3989bae: Time spent bootstrapping tablet: real 0.018s user 0.017s sys 0.000s
I20240503 02:00:33.470870 20716 raft_consensus.cc:348] T ec69ad86be3d4793b10aca98dc4c1736 P 838ef968bb45484bb27f5af6f3989bae [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } }
I20240503 02:00:33.470996 20716 raft_consensus.cc:729] T ec69ad86be3d4793b10aca98dc4c1736 P 838ef968bb45484bb27f5af6f3989bae [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 838ef968bb45484bb27f5af6f3989bae, State: Initialized, Role: LEARNER
I20240503 02:00:33.471134 20716 consensus_queue.cc:260] T ec69ad86be3d4793b10aca98dc4c1736 P 838ef968bb45484bb27f5af6f3989bae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } }
I20240503 02:00:33.471704 20716 ts_tablet_manager.cc:1398] T ec69ad86be3d4793b10aca98dc4c1736 P 838ef968bb45484bb27f5af6f3989bae: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240503 02:00:33.472421 17508 tablet_copy_service.cc:342] P 16d437fe629643948c95d628926d5ffa: Request end of tablet copy session 838ef968bb45484bb27f5af6f3989bae-ec69ad86be3d4793b10aca98dc4c1736 received from {username='slave'} at 127.0.99.137:37139
I20240503 02:00:33.472517 17508 tablet_copy_service.cc:433] P 16d437fe629643948c95d628926d5ffa: ending tablet copy session 838ef968bb45484bb27f5af6f3989bae-ec69ad86be3d4793b10aca98dc4c1736 on tablet ec69ad86be3d4793b10aca98dc4c1736 with peer 838ef968bb45484bb27f5af6f3989bae
I20240503 02:00:33.474344 20723 ts_tablet_manager.cc:902] T 78d106d705df4089a4dfae4d5dc533cc P 9564311af29a4493831a8db658a085c4: Initiating tablet copy from peer 002cc3411acb4fdfa0b67c6f57f687a6 (127.0.99.135:38989)
I20240503 02:00:33.474920 20723 tablet_copy_client.cc:308] T 78d106d705df4089a4dfae4d5dc533cc P 9564311af29a4493831a8db658a085c4: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.135:38989
I20240503 02:00:33.475432 17772 tablet_copy_service.cc:140] P 002cc3411acb4fdfa0b67c6f57f687a6: Received BeginTabletCopySession request for tablet 78d106d705df4089a4dfae4d5dc533cc from peer 9564311af29a4493831a8db658a085c4 ({username='slave'} at 127.0.99.138:57403)
I20240503 02:00:33.475564 17772 tablet_copy_service.cc:161] P 002cc3411acb4fdfa0b67c6f57f687a6: Beginning new tablet copy session on tablet 78d106d705df4089a4dfae4d5dc533cc from peer 9564311af29a4493831a8db658a085c4 at {username='slave'} at 127.0.99.138:57403: session id = 9564311af29a4493831a8db658a085c4-78d106d705df4089a4dfae4d5dc533cc
I20240503 02:00:33.476336 17772 tablet_copy_source_session.cc:205] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:33.477238 20723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78d106d705df4089a4dfae4d5dc533cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:33.479007 20723 tablet_copy_client.cc:783] T 78d106d705df4089a4dfae4d5dc533cc P 9564311af29a4493831a8db658a085c4: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:33.479157 20723 tablet_copy_client.cc:647] T 78d106d705df4089a4dfae4d5dc533cc P 9564311af29a4493831a8db658a085c4: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:33.480298 20723 tablet_copy_client.cc:523] T 78d106d705df4089a4dfae4d5dc533cc P 9564311af29a4493831a8db658a085c4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:33.481577 20723 tablet_bootstrap.cc:492] T 78d106d705df4089a4dfae4d5dc533cc P 9564311af29a4493831a8db658a085c4: Bootstrap starting.
I20240503 02:00:33.506083 20723 tablet_bootstrap.cc:492] T 78d106d705df4089a4dfae4d5dc533cc P 9564311af29a4493831a8db658a085c4: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:33.506445 20723 tablet_bootstrap.cc:492] T 78d106d705df4089a4dfae4d5dc533cc P 9564311af29a4493831a8db658a085c4: Bootstrap complete.
I20240503 02:00:33.506649 20723 ts_tablet_manager.cc:1367] T 78d106d705df4089a4dfae4d5dc533cc P 9564311af29a4493831a8db658a085c4: Time spent bootstrapping tablet: real 0.025s user 0.021s sys 0.000s
I20240503 02:00:33.507102 20723 raft_consensus.cc:348] T 78d106d705df4089a4dfae4d5dc533cc P 9564311af29a4493831a8db658a085c4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: true } }
I20240503 02:00:33.507220 20723 raft_consensus.cc:729] T 78d106d705df4089a4dfae4d5dc533cc P 9564311af29a4493831a8db658a085c4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9564311af29a4493831a8db658a085c4, State: Initialized, Role: LEARNER
I20240503 02:00:33.507344 20723 consensus_queue.cc:260] T 78d106d705df4089a4dfae4d5dc533cc P 9564311af29a4493831a8db658a085c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: true } }
I20240503 02:00:33.507726 20723 ts_tablet_manager.cc:1398] T 78d106d705df4089a4dfae4d5dc533cc P 9564311af29a4493831a8db658a085c4: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240503 02:00:33.508154 17772 tablet_copy_service.cc:342] P 002cc3411acb4fdfa0b67c6f57f687a6: Request end of tablet copy session 9564311af29a4493831a8db658a085c4-78d106d705df4089a4dfae4d5dc533cc received from {username='slave'} at 127.0.99.138:57403
I20240503 02:00:33.508265 17772 tablet_copy_service.cc:433] P 002cc3411acb4fdfa0b67c6f57f687a6: ending tablet copy session 9564311af29a4493831a8db658a085c4-78d106d705df4089a4dfae4d5dc533cc on tablet 78d106d705df4089a4dfae4d5dc533cc with peer 9564311af29a4493831a8db658a085c4
I20240503 02:00:33.554904 20716 ts_tablet_manager.cc:902] T 81b54f3bcf34485eb11b1bc954a2bd8f P 838ef968bb45484bb27f5af6f3989bae: Initiating tablet copy from peer 89dd09f8aaee4b249ae3ce924a422a67 (127.0.99.130:38889)
I20240503 02:00:33.555567 20716 tablet_copy_client.cc:308] T 81b54f3bcf34485eb11b1bc954a2bd8f P 838ef968bb45484bb27f5af6f3989bae: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:38889
I20240503 02:00:33.556165 17111 tablet_copy_service.cc:140] P 89dd09f8aaee4b249ae3ce924a422a67: Received BeginTabletCopySession request for tablet 81b54f3bcf34485eb11b1bc954a2bd8f from peer 838ef968bb45484bb27f5af6f3989bae ({username='slave'} at 127.0.99.137:47267)
I20240503 02:00:33.556303 17111 tablet_copy_service.cc:161] P 89dd09f8aaee4b249ae3ce924a422a67: Beginning new tablet copy session on tablet 81b54f3bcf34485eb11b1bc954a2bd8f from peer 838ef968bb45484bb27f5af6f3989bae at {username='slave'} at 127.0.99.137:47267: session id = 838ef968bb45484bb27f5af6f3989bae-81b54f3bcf34485eb11b1bc954a2bd8f
I20240503 02:00:33.558571 17111 tablet_copy_source_session.cc:205] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:33.559979 20716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81b54f3bcf34485eb11b1bc954a2bd8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:33.562500 20716 tablet_copy_client.cc:783] T 81b54f3bcf34485eb11b1bc954a2bd8f P 838ef968bb45484bb27f5af6f3989bae: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:33.562702 20716 tablet_copy_client.cc:647] T 81b54f3bcf34485eb11b1bc954a2bd8f P 838ef968bb45484bb27f5af6f3989bae: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:33.564013 20716 tablet_copy_client.cc:523] T 81b54f3bcf34485eb11b1bc954a2bd8f P 838ef968bb45484bb27f5af6f3989bae: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:33.565721 20716 tablet_bootstrap.cc:492] T 81b54f3bcf34485eb11b1bc954a2bd8f P 838ef968bb45484bb27f5af6f3989bae: Bootstrap starting.
I20240503 02:00:33.583293 20716 tablet_bootstrap.cc:492] T 81b54f3bcf34485eb11b1bc954a2bd8f P 838ef968bb45484bb27f5af6f3989bae: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:33.583596 20716 tablet_bootstrap.cc:492] T 81b54f3bcf34485eb11b1bc954a2bd8f P 838ef968bb45484bb27f5af6f3989bae: Bootstrap complete.
I20240503 02:00:33.583812 20716 ts_tablet_manager.cc:1367] T 81b54f3bcf34485eb11b1bc954a2bd8f P 838ef968bb45484bb27f5af6f3989bae: Time spent bootstrapping tablet: real 0.018s user 0.016s sys 0.000s
I20240503 02:00:33.584311 20716 raft_consensus.cc:348] T 81b54f3bcf34485eb11b1bc954a2bd8f P 838ef968bb45484bb27f5af6f3989bae [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } }
I20240503 02:00:33.584471 20716 raft_consensus.cc:729] T 81b54f3bcf34485eb11b1bc954a2bd8f P 838ef968bb45484bb27f5af6f3989bae [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 838ef968bb45484bb27f5af6f3989bae, State: Initialized, Role: LEARNER
I20240503 02:00:33.584581 20716 consensus_queue.cc:260] T 81b54f3bcf34485eb11b1bc954a2bd8f P 838ef968bb45484bb27f5af6f3989bae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } }
I20240503 02:00:33.584921 20716 ts_tablet_manager.cc:1398] T 81b54f3bcf34485eb11b1bc954a2bd8f P 838ef968bb45484bb27f5af6f3989bae: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240503 02:00:33.585608 17111 tablet_copy_service.cc:342] P 89dd09f8aaee4b249ae3ce924a422a67: Request end of tablet copy session 838ef968bb45484bb27f5af6f3989bae-81b54f3bcf34485eb11b1bc954a2bd8f received from {username='slave'} at 127.0.99.137:47267
I20240503 02:00:33.585739 17111 tablet_copy_service.cc:433] P 89dd09f8aaee4b249ae3ce924a422a67: ending tablet copy session 838ef968bb45484bb27f5af6f3989bae-81b54f3bcf34485eb11b1bc954a2bd8f on tablet 81b54f3bcf34485eb11b1bc954a2bd8f with peer 838ef968bb45484bb27f5af6f3989bae
I20240503 02:00:33.671126 20723 ts_tablet_manager.cc:902] T 885fe0c3a5e04f408e5b02c3c133248a P 9564311af29a4493831a8db658a085c4: Initiating tablet copy from peer 6398c8993c3c45dcaab25b42142d5671 (127.0.99.129:36983)
I20240503 02:00:33.671689 20723 tablet_copy_client.cc:308] T 885fe0c3a5e04f408e5b02c3c133248a P 9564311af29a4493831a8db658a085c4: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:36983
I20240503 02:00:33.675652 16980 tablet_copy_service.cc:140] P 6398c8993c3c45dcaab25b42142d5671: Received BeginTabletCopySession request for tablet 885fe0c3a5e04f408e5b02c3c133248a from peer 9564311af29a4493831a8db658a085c4 ({username='slave'} at 127.0.99.138:40575)
I20240503 02:00:33.675925 16980 tablet_copy_service.cc:161] P 6398c8993c3c45dcaab25b42142d5671: Beginning new tablet copy session on tablet 885fe0c3a5e04f408e5b02c3c133248a from peer 9564311af29a4493831a8db658a085c4 at {username='slave'} at 127.0.99.138:40575: session id = 9564311af29a4493831a8db658a085c4-885fe0c3a5e04f408e5b02c3c133248a
I20240503 02:00:33.677619 16980 tablet_copy_source_session.cc:205] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:33.678915 20723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 885fe0c3a5e04f408e5b02c3c133248a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:33.680902 20723 tablet_copy_client.cc:783] T 885fe0c3a5e04f408e5b02c3c133248a P 9564311af29a4493831a8db658a085c4: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:33.681073 20723 tablet_copy_client.cc:647] T 885fe0c3a5e04f408e5b02c3c133248a P 9564311af29a4493831a8db658a085c4: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:33.682884 20723 tablet_copy_client.cc:523] T 885fe0c3a5e04f408e5b02c3c133248a P 9564311af29a4493831a8db658a085c4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:33.684144 20723 tablet_bootstrap.cc:492] T 885fe0c3a5e04f408e5b02c3c133248a P 9564311af29a4493831a8db658a085c4: Bootstrap starting.
I20240503 02:00:33.704099 20723 tablet_bootstrap.cc:492] T 885fe0c3a5e04f408e5b02c3c133248a P 9564311af29a4493831a8db658a085c4: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:33.704397 20723 tablet_bootstrap.cc:492] T 885fe0c3a5e04f408e5b02c3c133248a P 9564311af29a4493831a8db658a085c4: Bootstrap complete.
I20240503 02:00:33.704583 20723 ts_tablet_manager.cc:1367] T 885fe0c3a5e04f408e5b02c3c133248a P 9564311af29a4493831a8db658a085c4: Time spent bootstrapping tablet: real 0.020s user 0.017s sys 0.000s
I20240503 02:00:33.705034 20723 raft_consensus.cc:348] T 885fe0c3a5e04f408e5b02c3c133248a P 9564311af29a4493831a8db658a085c4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: true } }
I20240503 02:00:33.705152 20723 raft_consensus.cc:729] T 885fe0c3a5e04f408e5b02c3c133248a P 9564311af29a4493831a8db658a085c4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9564311af29a4493831a8db658a085c4, State: Initialized, Role: LEARNER
I20240503 02:00:33.705236 20723 consensus_queue.cc:260] T 885fe0c3a5e04f408e5b02c3c133248a P 9564311af29a4493831a8db658a085c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: true } }
I20240503 02:00:33.705535 20723 ts_tablet_manager.cc:1398] T 885fe0c3a5e04f408e5b02c3c133248a P 9564311af29a4493831a8db658a085c4: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240503 02:00:33.706323 16980 tablet_copy_service.cc:342] P 6398c8993c3c45dcaab25b42142d5671: Request end of tablet copy session 9564311af29a4493831a8db658a085c4-885fe0c3a5e04f408e5b02c3c133248a received from {username='slave'} at 127.0.99.138:40575
I20240503 02:00:33.706452 16980 tablet_copy_service.cc:433] P 6398c8993c3c45dcaab25b42142d5671: ending tablet copy session 9564311af29a4493831a8db658a085c4-885fe0c3a5e04f408e5b02c3c133248a on tablet 885fe0c3a5e04f408e5b02c3c133248a with peer 9564311af29a4493831a8db658a085c4
I20240503 02:00:33.862922 19238 raft_consensus.cc:1203] T 78d106d705df4089a4dfae4d5dc533cc P 9564311af29a4493831a8db658a085c4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 02:00:33.931556 19108 raft_consensus.cc:1203] T 81b54f3bcf34485eb11b1bc954a2bd8f P 838ef968bb45484bb27f5af6f3989bae [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 02:00:33.934237 19108 raft_consensus.cc:1203] T ec69ad86be3d4793b10aca98dc4c1736 P 838ef968bb45484bb27f5af6f3989bae [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 02:00:34.185567 19238 raft_consensus.cc:1203] T 885fe0c3a5e04f408e5b02c3c133248a P 9564311af29a4493831a8db658a085c4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 02:00:34.463114 20845 raft_consensus.cc:1050] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6: attempting to promote NON_VOTER 9564311af29a4493831a8db658a085c4 to VOTER
I20240503 02:00:34.463640 20845 consensus_queue.cc:237] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } }
I20240503 02:00:34.465739 19238 raft_consensus.cc:1261] T 78d106d705df4089a4dfae4d5dc533cc P 9564311af29a4493831a8db658a085c4 [term 1 LEARNER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:34.465976 17223 raft_consensus.cc:1261] T 78d106d705df4089a4dfae4d5dc533cc P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:34.466068 17619 raft_consensus.cc:1261] T 78d106d705df4089a4dfae4d5dc533cc P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:34.466604 17487 raft_consensus.cc:1261] T 78d106d705df4089a4dfae4d5dc533cc P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:34.466670 17090 raft_consensus.cc:1261] T 78d106d705df4089a4dfae4d5dc533cc P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:34.466651 17356 raft_consensus.cc:1261] T 78d106d705df4089a4dfae4d5dc533cc P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:34.466606 20847 consensus_queue.cc:1035] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:34.467099 20847 consensus_queue.cc:1035] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:34.467270 20848 consensus_queue.cc:1035] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:34.467347 16959 raft_consensus.cc:1261] T 78d106d705df4089a4dfae4d5dc533cc P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:34.467588 20851 consensus_queue.cc:1035] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:34.468530 20845 consensus_queue.cc:1035] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:34.468809 20850 consensus_queue.cc:1035] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:34.469040 20849 consensus_queue.cc:1035] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:34.471827 20847 raft_consensus.cc:2932] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9564311af29a4493831a8db658a085c4 (127.0.99.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } }
I20240503 02:00:34.472237 17487 raft_consensus.cc:2932] T 78d106d705df4089a4dfae4d5dc533cc P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9564311af29a4493831a8db658a085c4 (127.0.99.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } }
I20240503 02:00:34.472913 17090 raft_consensus.cc:2932] T 78d106d705df4089a4dfae4d5dc533cc P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9564311af29a4493831a8db658a085c4 (127.0.99.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } }
I20240503 02:00:34.474225 17223 raft_consensus.cc:2932] T 78d106d705df4089a4dfae4d5dc533cc P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9564311af29a4493831a8db658a085c4 (127.0.99.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } }
I20240503 02:00:34.475276 17356 raft_consensus.cc:2932] T 78d106d705df4089a4dfae4d5dc533cc P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9564311af29a4493831a8db658a085c4 (127.0.99.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } }
I20240503 02:00:34.475689 16959 raft_consensus.cc:2932] T 78d106d705df4089a4dfae4d5dc533cc P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9564311af29a4493831a8db658a085c4 (127.0.99.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } }
I20240503 02:00:34.478555 16817 catalog_manager.cc:5484] T 78d106d705df4089a4dfae4d5dc533cc P 16d437fe629643948c95d628926d5ffa reported cstate change: config changed from index 5 to 6, 9564311af29a4493831a8db658a085c4 (127.0.99.138) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } }
I20240503 02:00:34.479892 17619 raft_consensus.cc:2932] T 78d106d705df4089a4dfae4d5dc533cc P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9564311af29a4493831a8db658a085c4 (127.0.99.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } }
I20240503 02:00:34.481384 19238 raft_consensus.cc:2932] T 78d106d705df4089a4dfae4d5dc533cc P 9564311af29a4493831a8db658a085c4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9564311af29a4493831a8db658a085c4 (127.0.99.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } }
I20240503 02:00:34.495018 20473 raft_consensus.cc:1050] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67: attempting to promote NON_VOTER 838ef968bb45484bb27f5af6f3989bae to VOTER
I20240503 02:00:34.495678 20473 consensus_queue.cc:237] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } }
I20240503 02:00:34.496878 17751 consensus_queue.cc:237] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } }
I20240503 02:00:34.497726 17487 raft_consensus.cc:1261] T 81b54f3bcf34485eb11b1bc954a2bd8f P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:34.497928 19108 raft_consensus.cc:1261] T 81b54f3bcf34485eb11b1bc954a2bd8f P 838ef968bb45484bb27f5af6f3989bae [term 1 LEARNER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:34.498698 20057 consensus_queue.cc:1035] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:34.499025 20057 consensus_queue.cc:1035] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:34.499261 17223 raft_consensus.cc:1261] T 78d106d705df4089a4dfae4d5dc533cc P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:34.499361 17356 raft_consensus.cc:1261] T 81b54f3bcf34485eb11b1bc954a2bd8f P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:34.500054 16959 raft_consensus.cc:1261] T 81b54f3bcf34485eb11b1bc954a2bd8f P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:34.500169 17223 raft_consensus.cc:1261] T 81b54f3bcf34485eb11b1bc954a2bd8f P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:34.500622 16958 raft_consensus.cc:1261] T 78d106d705df4089a4dfae4d5dc533cc P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:34.500720 19238 raft_consensus.cc:1261] T 78d106d705df4089a4dfae4d5dc533cc P 9564311af29a4493831a8db658a085c4 [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:34.500715 20848 consensus_queue.cc:1035] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:34.501083 17090 raft_consensus.cc:1261] T 78d106d705df4089a4dfae4d5dc533cc P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:34.501457 20473 consensus_queue.cc:1035] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:34.501519 20845 consensus_queue.cc:1035] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:34.501804 20845 consensus_queue.cc:1035] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:34.501922 17620 raft_consensus.cc:1261] T 81b54f3bcf34485eb11b1bc954a2bd8f P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:34.502651 17487 raft_consensus.cc:1261] T 78d106d705df4089a4dfae4d5dc533cc P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:34.503454 20847 consensus_queue.cc:1035] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:34.503847 20847 consensus_queue.cc:1035] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:34.505349 20057 consensus_queue.cc:1035] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:34.505466 20845 raft_consensus.cc:2932] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } }
I20240503 02:00:34.505698 20473 consensus_queue.cc:1035] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:34.505826 20057 consensus_queue.cc:1035] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:34.506930 17223 raft_consensus.cc:2932] T 78d106d705df4089a4dfae4d5dc533cc P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } }
I20240503 02:00:34.507416 17751 raft_consensus.cc:1261] T 81b54f3bcf34485eb11b1bc954a2bd8f P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:34.507444 16802 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 78d106d705df4089a4dfae4d5dc533cc with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:00:34.507926 17619 raft_consensus.cc:1261] T 78d106d705df4089a4dfae4d5dc533cc P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:34.508724 20851 consensus_queue.cc:1035] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:34.509857 16815 catalog_manager.cc:5484] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 reported cstate change: config changed from index 6 to 7, VOTER d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:00:34.509869 17487 raft_consensus.cc:2932] T 78d106d705df4089a4dfae4d5dc533cc P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } }
I20240503 02:00:34.510592 16958 raft_consensus.cc:2932] T 78d106d705df4089a4dfae4d5dc533cc P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } }
I20240503 02:00:34.512836 17619 raft_consensus.cc:2932] T 78d106d705df4089a4dfae4d5dc533cc P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } }
I20240503 02:00:34.513199 19238 raft_consensus.cc:2932] T 78d106d705df4089a4dfae4d5dc533cc P 9564311af29a4493831a8db658a085c4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } }
I20240503 02:00:34.515151 17335 tablet_service.cc:1514] Processing DeleteTablet for tablet 78d106d705df4089a4dfae4d5dc533cc with delete_type TABLET_DATA_TOMBSTONED (TS d43ed6b8663947d99a72cd922cc323aa not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:44362
I20240503 02:00:34.515583 20057 consensus_queue.cc:1035] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:34.515899 17090 raft_consensus.cc:2932] T 78d106d705df4089a4dfae4d5dc533cc P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } }
I20240503 02:00:34.518780 20876 tablet_replica.cc:330] T 78d106d705df4089a4dfae4d5dc533cc P d43ed6b8663947d99a72cd922cc323aa: stopping tablet replica
I20240503 02:00:34.519013 20876 raft_consensus.cc:2229] T 78d106d705df4089a4dfae4d5dc533cc P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:34.519320 20876 raft_consensus.cc:2258] T 78d106d705df4089a4dfae4d5dc533cc P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:34.520620 20876 ts_tablet_manager.cc:1875] T 78d106d705df4089a4dfae4d5dc533cc P d43ed6b8663947d99a72cd922cc323aa: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:34.524857 20865 raft_consensus.cc:2932] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:34.525303 17222 raft_consensus.cc:2932] T 81b54f3bcf34485eb11b1bc954a2bd8f P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:34.526160 17486 raft_consensus.cc:2932] T 81b54f3bcf34485eb11b1bc954a2bd8f P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:34.527688 20876 ts_tablet_manager.cc:1888] T 78d106d705df4089a4dfae4d5dc533cc P d43ed6b8663947d99a72cd922cc323aa: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 02:00:34.527807 20876 log.cc:1196] T 78d106d705df4089a4dfae4d5dc533cc P d43ed6b8663947d99a72cd922cc323aa: Deleting WAL directory at /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/78d106d705df4089a4dfae4d5dc533cc
I20240503 02:00:34.527607 16958 raft_consensus.cc:2932] T 81b54f3bcf34485eb11b1bc954a2bd8f P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:34.528081 17353 raft_consensus.cc:2932] T 81b54f3bcf34485eb11b1bc954a2bd8f P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:34.528811 16803 catalog_manager.cc:4830] TS d43ed6b8663947d99a72cd922cc323aa (127.0.99.132:40023): tablet 78d106d705df4089a4dfae4d5dc533cc (table rebalance_test_table_5 [id=90cb01160ab843bbae459ab57f0425ed]) successfully deleted
I20240503 02:00:34.529187 17620 raft_consensus.cc:2932] T 81b54f3bcf34485eb11b1bc954a2bd8f P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:34.530959 16810 catalog_manager.cc:5484] T 81b54f3bcf34485eb11b1bc954a2bd8f P 5b8291b26a7f448b8636187d1caf86d0 reported cstate change: config changed from index 5 to 6, 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "89dd09f8aaee4b249ae3ce924a422a67" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:34.531075 19108 raft_consensus.cc:2932] T 81b54f3bcf34485eb11b1bc954a2bd8f P 838ef968bb45484bb27f5af6f3989bae [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:34.535293 17751 raft_consensus.cc:2932] T 81b54f3bcf34485eb11b1bc954a2bd8f P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:34.539528 17090 consensus_queue.cc:237] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } }
I20240503 02:00:34.541394 17486 raft_consensus.cc:1261] T 81b54f3bcf34485eb11b1bc954a2bd8f P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:34.541693 16958 raft_consensus.cc:1261] T 81b54f3bcf34485eb11b1bc954a2bd8f P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:34.541787 17353 raft_consensus.cc:1261] T 81b54f3bcf34485eb11b1bc954a2bd8f P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:34.542234 20473 consensus_queue.cc:1035] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:34.542358 17222 raft_consensus.cc:1261] T 81b54f3bcf34485eb11b1bc954a2bd8f P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:34.542413 20869 consensus_queue.cc:1035] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:34.542726 20865 consensus_queue.cc:1035] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:34.542870 20869 consensus_queue.cc:1035] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:34.544986 17751 raft_consensus.cc:1261] T 81b54f3bcf34485eb11b1bc954a2bd8f P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:34.545570 20867 consensus_queue.cc:1035] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:34.545495 20473 raft_consensus.cc:2932] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:34.546178 16958 raft_consensus.cc:2932] T 81b54f3bcf34485eb11b1bc954a2bd8f P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:34.547770 16805 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 81b54f3bcf34485eb11b1bc954a2bd8f with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:00:34.548372 19108 raft_consensus.cc:1261] T 81b54f3bcf34485eb11b1bc954a2bd8f P 838ef968bb45484bb27f5af6f3989bae [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:34.548893 16815 catalog_manager.cc:5484] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 reported cstate change: config changed from index 6 to 7, VOTER c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134) evicted. New cstate: current_term: 1 leader_uuid: "89dd09f8aaee4b249ae3ce924a422a67" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:34.549118 17353 raft_consensus.cc:2932] T 81b54f3bcf34485eb11b1bc954a2bd8f P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:34.549409 20865 consensus_queue.cc:1035] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:34.550252 17222 raft_consensus.cc:2932] T 81b54f3bcf34485eb11b1bc954a2bd8f P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:34.550446 19108 raft_consensus.cc:2932] T 81b54f3bcf34485eb11b1bc954a2bd8f P 838ef968bb45484bb27f5af6f3989bae [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:34.552585 20519 raft_consensus.cc:1050] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa: attempting to promote NON_VOTER 838ef968bb45484bb27f5af6f3989bae to VOTER
I20240503 02:00:34.553156 17751 raft_consensus.cc:2932] T 81b54f3bcf34485eb11b1bc954a2bd8f P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:34.553334 20519 consensus_queue.cc:237] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } }
I20240503 02:00:34.553740 17598 tablet_service.cc:1514] Processing DeleteTablet for tablet 81b54f3bcf34485eb11b1bc954a2bd8f with delete_type TABLET_DATA_TOMBSTONED (TS c843193b801b4f1a88f630ed3ed0f375 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:51030
I20240503 02:00:34.556794 17750 raft_consensus.cc:1261] T ec69ad86be3d4793b10aca98dc4c1736 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:34.557245 19107 raft_consensus.cc:1261] T ec69ad86be3d4793b10aca98dc4c1736 P 838ef968bb45484bb27f5af6f3989bae [term 1 LEARNER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:34.557368 17356 raft_consensus.cc:1261] T ec69ad86be3d4793b10aca98dc4c1736 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:34.557559 17223 raft_consensus.cc:1261] T ec69ad86be3d4793b10aca98dc4c1736 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:34.557904 20515 consensus_queue.cc:1035] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:34.558259 20519 consensus_queue.cc:1035] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:34.558676 20519 consensus_queue.cc:1035] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:34.558748 17620 raft_consensus.cc:1261] T ec69ad86be3d4793b10aca98dc4c1736 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:34.559087 16958 raft_consensus.cc:1261] T ec69ad86be3d4793b10aca98dc4c1736 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:34.559171 20519 consensus_queue.cc:1035] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:34.559767 20515 consensus_queue.cc:1035] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:34.560747 20519 consensus_queue.cc:1035] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:34.560858 17090 raft_consensus.cc:1261] T ec69ad86be3d4793b10aca98dc4c1736 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:34.561240 20519 consensus_queue.cc:1035] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:34.564374 17486 raft_consensus.cc:2932] T 81b54f3bcf34485eb11b1bc954a2bd8f P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:34.565234 20879 tablet_replica.cc:330] T 81b54f3bcf34485eb11b1bc954a2bd8f P c843193b801b4f1a88f630ed3ed0f375: stopping tablet replica
I20240503 02:00:34.566728 20879 raft_consensus.cc:2229] T 81b54f3bcf34485eb11b1bc954a2bd8f P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:34.567065 20519 raft_consensus.cc:2932] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:34.567520 20879 raft_consensus.cc:2258] T 81b54f3bcf34485eb11b1bc954a2bd8f P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:34.568568 20879 ts_tablet_manager.cc:1875] T 81b54f3bcf34485eb11b1bc954a2bd8f P c843193b801b4f1a88f630ed3ed0f375: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:34.569327 19108 raft_consensus.cc:2932] T ec69ad86be3d4793b10aca98dc4c1736 P 838ef968bb45484bb27f5af6f3989bae [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:34.569576 17751 raft_consensus.cc:2932] T ec69ad86be3d4793b10aca98dc4c1736 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:34.569626 17620 raft_consensus.cc:2932] T ec69ad86be3d4793b10aca98dc4c1736 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:34.570513 17090 raft_consensus.cc:2932] T ec69ad86be3d4793b10aca98dc4c1736 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:34.572273 16958 raft_consensus.cc:2932] T ec69ad86be3d4793b10aca98dc4c1736 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:34.573423 16818 catalog_manager.cc:5484] T ec69ad86be3d4793b10aca98dc4c1736 P 838ef968bb45484bb27f5af6f3989bae reported cstate change: config changed from index 5 to 6, 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "16d437fe629643948c95d628926d5ffa" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:34.573654 17356 raft_consensus.cc:2932] T ec69ad86be3d4793b10aca98dc4c1736 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:34.579752 17222 raft_consensus.cc:2932] T ec69ad86be3d4793b10aca98dc4c1736 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:34.581787 17486 consensus_queue.cc:237] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } }
I20240503 02:00:34.583175 20879 ts_tablet_manager.cc:1888] T 81b54f3bcf34485eb11b1bc954a2bd8f P c843193b801b4f1a88f630ed3ed0f375: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 02:00:34.583372 20879 log.cc:1196] T 81b54f3bcf34485eb11b1bc954a2bd8f P c843193b801b4f1a88f630ed3ed0f375: Deleting WAL directory at /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/81b54f3bcf34485eb11b1bc954a2bd8f
I20240503 02:00:34.583788 17222 raft_consensus.cc:1261] T ec69ad86be3d4793b10aca98dc4c1736 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:34.583865 19108 raft_consensus.cc:1261] T ec69ad86be3d4793b10aca98dc4c1736 P 838ef968bb45484bb27f5af6f3989bae [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:34.583993 17751 raft_consensus.cc:1261] T ec69ad86be3d4793b10aca98dc4c1736 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:34.584348 16802 catalog_manager.cc:4830] TS c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134:44343): tablet 81b54f3bcf34485eb11b1bc954a2bd8f (table rebalance_test_table_0 [id=ee1e0b20af0b4ecea04ae147701a0482]) successfully deleted
I20240503 02:00:34.584483 20519 consensus_queue.cc:1035] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:34.584619 20515 consensus_queue.cc:1035] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:34.584774 20885 consensus_queue.cc:1035] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:34.585598 17090 raft_consensus.cc:1261] T ec69ad86be3d4793b10aca98dc4c1736 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:34.585731 17620 raft_consensus.cc:1261] T ec69ad86be3d4793b10aca98dc4c1736 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:34.585877 16958 raft_consensus.cc:1261] T ec69ad86be3d4793b10aca98dc4c1736 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:34.586417 20515 consensus_queue.cc:1035] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:34.586535 20886 consensus_queue.cc:1035] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:34.586853 20515 consensus_queue.cc:1035] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:34.588933 20886 raft_consensus.cc:2932] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:34.589591 17620 raft_consensus.cc:2932] T ec69ad86be3d4793b10aca98dc4c1736 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:34.590008 16958 raft_consensus.cc:2932] T ec69ad86be3d4793b10aca98dc4c1736 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:34.591267 16802 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ec69ad86be3d4793b10aca98dc4c1736 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:00:34.591109 17222 raft_consensus.cc:2932] T ec69ad86be3d4793b10aca98dc4c1736 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:34.592036 17090 raft_consensus.cc:2932] T ec69ad86be3d4793b10aca98dc4c1736 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:34.592339 16812 catalog_manager.cc:5484] T ec69ad86be3d4793b10aca98dc4c1736 P c843193b801b4f1a88f630ed3ed0f375 reported cstate change: config changed from index 6 to 7, VOTER d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "16d437fe629643948c95d628926d5ffa" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:34.595764 20685 raft_consensus.cc:1050] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671: attempting to promote NON_VOTER 9564311af29a4493831a8db658a085c4 to VOTER
I20240503 02:00:34.596066 17335 tablet_service.cc:1514] Processing DeleteTablet for tablet ec69ad86be3d4793b10aca98dc4c1736 with delete_type TABLET_DATA_TOMBSTONED (TS d43ed6b8663947d99a72cd922cc323aa not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:44362
I20240503 02:00:34.596355 20876 tablet_replica.cc:330] T ec69ad86be3d4793b10aca98dc4c1736 P d43ed6b8663947d99a72cd922cc323aa: stopping tablet replica
I20240503 02:00:34.596508 20876 raft_consensus.cc:2229] T ec69ad86be3d4793b10aca98dc4c1736 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:34.596436 20685 consensus_queue.cc:237] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } }
I20240503 02:00:34.596793 20876 raft_consensus.cc:2258] T ec69ad86be3d4793b10aca98dc4c1736 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:34.597569 20876 ts_tablet_manager.cc:1875] T ec69ad86be3d4793b10aca98dc4c1736 P d43ed6b8663947d99a72cd922cc323aa: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:34.598485 19239 raft_consensus.cc:1261] T 885fe0c3a5e04f408e5b02c3c133248a P 9564311af29a4493831a8db658a085c4 [term 1 LEARNER]: Refusing update from remote peer 6398c8993c3c45dcaab25b42142d5671: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:34.598487 17620 raft_consensus.cc:1261] T 885fe0c3a5e04f408e5b02c3c133248a P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer 6398c8993c3c45dcaab25b42142d5671: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:34.599054 20480 consensus_queue.cc:1035] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:34.599229 20684 consensus_queue.cc:1035] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:34.599858 20876 ts_tablet_manager.cc:1888] T ec69ad86be3d4793b10aca98dc4c1736 P d43ed6b8663947d99a72cd922cc323aa: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 02:00:34.599931 20876 log.cc:1196] T ec69ad86be3d4793b10aca98dc4c1736 P d43ed6b8663947d99a72cd922cc323aa: Deleting WAL directory at /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/ec69ad86be3d4793b10aca98dc4c1736
I20240503 02:00:34.600713 16803 catalog_manager.cc:4830] TS d43ed6b8663947d99a72cd922cc323aa (127.0.99.132:40023): tablet ec69ad86be3d4793b10aca98dc4c1736 (table rebalance_test_table_5 [id=90cb01160ab843bbae459ab57f0425ed]) successfully deleted
I20240503 02:00:34.602340 17486 raft_consensus.cc:1261] T 885fe0c3a5e04f408e5b02c3c133248a P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Refusing update from remote peer 6398c8993c3c45dcaab25b42142d5671: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:34.602425 17751 raft_consensus.cc:1261] T 885fe0c3a5e04f408e5b02c3c133248a P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer 6398c8993c3c45dcaab25b42142d5671: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:34.602957 20684 consensus_queue.cc:1035] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671 [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:34.603016 17353 raft_consensus.cc:1261] T 885fe0c3a5e04f408e5b02c3c133248a P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Refusing update from remote peer 6398c8993c3c45dcaab25b42142d5671: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:34.603425 17222 raft_consensus.cc:1261] T 885fe0c3a5e04f408e5b02c3c133248a P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer 6398c8993c3c45dcaab25b42142d5671: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:34.603658 17090 raft_consensus.cc:1261] T 885fe0c3a5e04f408e5b02c3c133248a P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer 6398c8993c3c45dcaab25b42142d5671: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:34.603682 20684 consensus_queue.cc:1035] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:34.603994 20684 consensus_queue.cc:1035] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:34.604598 20684 consensus_queue.cc:1035] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:34.605082 20684 consensus_queue.cc:1035] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:34.607620 17751 raft_consensus.cc:2932] T ec69ad86be3d4793b10aca98dc4c1736 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:34.607885 19108 raft_consensus.cc:2932] T ec69ad86be3d4793b10aca98dc4c1736 P 838ef968bb45484bb27f5af6f3989bae [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:34.609059 20685 raft_consensus.cc:2932] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9564311af29a4493831a8db658a085c4 (127.0.99.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } }
I20240503 02:00:34.611449 19239 raft_consensus.cc:2932] T 885fe0c3a5e04f408e5b02c3c133248a P 9564311af29a4493831a8db658a085c4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9564311af29a4493831a8db658a085c4 (127.0.99.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } }
I20240503 02:00:34.613170 17486 raft_consensus.cc:2932] T 885fe0c3a5e04f408e5b02c3c133248a P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9564311af29a4493831a8db658a085c4 (127.0.99.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } }
I20240503 02:00:34.613337 17751 raft_consensus.cc:2932] T 885fe0c3a5e04f408e5b02c3c133248a P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9564311af29a4493831a8db658a085c4 (127.0.99.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } }
I20240503 02:00:34.614791 16813 catalog_manager.cc:5484] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671 reported cstate change: config changed from index 5 to 6, 9564311af29a4493831a8db658a085c4 (127.0.99.138) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6398c8993c3c45dcaab25b42142d5671" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:00:34.615757 17223 raft_consensus.cc:2932] T 885fe0c3a5e04f408e5b02c3c133248a P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9564311af29a4493831a8db658a085c4 (127.0.99.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } }
I20240503 02:00:34.615731 17353 raft_consensus.cc:2932] T 885fe0c3a5e04f408e5b02c3c133248a P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9564311af29a4493831a8db658a085c4 (127.0.99.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } }
I20240503 02:00:34.620801 17090 raft_consensus.cc:2932] T 885fe0c3a5e04f408e5b02c3c133248a P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9564311af29a4493831a8db658a085c4 (127.0.99.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } }
I20240503 02:00:34.622987 17620 raft_consensus.cc:2932] T 885fe0c3a5e04f408e5b02c3c133248a P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9564311af29a4493831a8db658a085c4 (127.0.99.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } }
I20240503 02:00:34.626010 16958 consensus_queue.cc:237] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } }
I20240503 02:00:34.628491 17223 raft_consensus.cc:1261] T 885fe0c3a5e04f408e5b02c3c133248a P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer 6398c8993c3c45dcaab25b42142d5671: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:34.628625 17486 raft_consensus.cc:1261] T 885fe0c3a5e04f408e5b02c3c133248a P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Refusing update from remote peer 6398c8993c3c45dcaab25b42142d5671: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:34.629089 20684 consensus_queue.cc:1035] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:34.629185 17620 raft_consensus.cc:1261] T 885fe0c3a5e04f408e5b02c3c133248a P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer 6398c8993c3c45dcaab25b42142d5671: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:34.629446 20684 consensus_queue.cc:1035] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:34.629588 19239 raft_consensus.cc:1261] T 885fe0c3a5e04f408e5b02c3c133248a P 9564311af29a4493831a8db658a085c4 [term 1 FOLLOWER]: Refusing update from remote peer 6398c8993c3c45dcaab25b42142d5671: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:34.629686 17090 raft_consensus.cc:1261] T 885fe0c3a5e04f408e5b02c3c133248a P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer 6398c8993c3c45dcaab25b42142d5671: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:34.630028 20897 consensus_queue.cc:1035] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:34.630263 20684 consensus_queue.cc:1035] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:34.632607 20480 raft_consensus.cc:2932] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } }
I20240503 02:00:34.633106 17090 raft_consensus.cc:2932] T 885fe0c3a5e04f408e5b02c3c133248a P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } }
I20240503 02:00:34.632712 20904 consensus_queue.cc:1035] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:34.635010 16804 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 885fe0c3a5e04f408e5b02c3c133248a with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:00:34.636078 17751 raft_consensus.cc:1261] T 885fe0c3a5e04f408e5b02c3c133248a P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer 6398c8993c3c45dcaab25b42142d5671: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:34.635941 16815 catalog_manager.cc:5484] T 885fe0c3a5e04f408e5b02c3c133248a P 89dd09f8aaee4b249ae3ce924a422a67 reported cstate change: config changed from index 6 to 7, VOTER d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "6398c8993c3c45dcaab25b42142d5671" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } }
I20240503 02:00:34.636287 17620 raft_consensus.cc:2932] T 885fe0c3a5e04f408e5b02c3c133248a P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } }
I20240503 02:00:34.637182 17223 raft_consensus.cc:2932] T 885fe0c3a5e04f408e5b02c3c133248a P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } }
I20240503 02:00:34.637668 19239 raft_consensus.cc:2932] T 885fe0c3a5e04f408e5b02c3c133248a P 9564311af29a4493831a8db658a085c4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } }
I20240503 02:00:34.638911 17486 raft_consensus.cc:2932] T 885fe0c3a5e04f408e5b02c3c133248a P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } }
I20240503 02:00:34.640221 20904 consensus_queue.cc:1035] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671 [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:34.642113 17751 raft_consensus.cc:2932] T 885fe0c3a5e04f408e5b02c3c133248a P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } }
I20240503 02:00:34.645650 17335 tablet_service.cc:1514] Processing DeleteTablet for tablet 885fe0c3a5e04f408e5b02c3c133248a with delete_type TABLET_DATA_TOMBSTONED (TS d43ed6b8663947d99a72cd922cc323aa not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:44362
I20240503 02:00:34.646150 20876 tablet_replica.cc:330] T 885fe0c3a5e04f408e5b02c3c133248a P d43ed6b8663947d99a72cd922cc323aa: stopping tablet replica
I20240503 02:00:34.646371 20876 raft_consensus.cc:2229] T 885fe0c3a5e04f408e5b02c3c133248a P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:34.646689 20876 raft_consensus.cc:2258] T 885fe0c3a5e04f408e5b02c3c133248a P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:34.647490 20876 ts_tablet_manager.cc:1875] T 885fe0c3a5e04f408e5b02c3c133248a P d43ed6b8663947d99a72cd922cc323aa: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:34.650091 20876 ts_tablet_manager.cc:1888] T 885fe0c3a5e04f408e5b02c3c133248a P d43ed6b8663947d99a72cd922cc323aa: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 02:00:34.650192 20876 log.cc:1196] T 885fe0c3a5e04f408e5b02c3c133248a P d43ed6b8663947d99a72cd922cc323aa: Deleting WAL directory at /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/885fe0c3a5e04f408e5b02c3c133248a
I20240503 02:00:34.650835 16803 catalog_manager.cc:4830] TS d43ed6b8663947d99a72cd922cc323aa (127.0.99.132:40023): tablet 885fe0c3a5e04f408e5b02c3c133248a (table rebalance_test_table_1 [id=38eeb06923e14f2ead405ffb039fd231]) successfully deleted
I20240503 02:00:34.883725 17620 consensus_queue.cc:237] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } }
I20240503 02:00:34.885831 17090 raft_consensus.cc:1261] T d81dcfefb4fd401b9cfd2ca36034a010 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer c843193b801b4f1a88f630ed3ed0f375: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:34.885939 17223 raft_consensus.cc:1261] T d81dcfefb4fd401b9cfd2ca36034a010 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer c843193b801b4f1a88f630ed3ed0f375: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:34.885957 17353 raft_consensus.cc:1261] T d81dcfefb4fd401b9cfd2ca36034a010 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Refusing update from remote peer c843193b801b4f1a88f630ed3ed0f375: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:34.886106 17751 raft_consensus.cc:1261] T d81dcfefb4fd401b9cfd2ca36034a010 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer c843193b801b4f1a88f630ed3ed0f375: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:34.886310 17486 raft_consensus.cc:1261] T d81dcfefb4fd401b9cfd2ca36034a010 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Refusing update from remote peer c843193b801b4f1a88f630ed3ed0f375: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:34.886565 20934 consensus_queue.cc:1035] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:34.886768 20932 consensus_queue.cc:1035] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:34.886864 16958 raft_consensus.cc:1261] T d81dcfefb4fd401b9cfd2ca36034a010 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer c843193b801b4f1a88f630ed3ed0f375: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:34.887089 20934 consensus_queue.cc:1035] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:34.887286 20193 consensus_queue.cc:1035] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:34.887955 20193 consensus_queue.cc:1035] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:34.888284 20193 consensus_queue.cc:1035] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:34.890651 20193 raft_consensus.cc:2932] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } }
I20240503 02:00:34.891165 17090 raft_consensus.cc:2932] T d81dcfefb4fd401b9cfd2ca36034a010 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } }
I20240503 02:00:34.891350 16958 raft_consensus.cc:2932] T d81dcfefb4fd401b9cfd2ca36034a010 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } }
I20240503 02:00:34.892971 17223 raft_consensus.cc:2932] T d81dcfefb4fd401b9cfd2ca36034a010 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } }
I20240503 02:00:34.895027 16810 catalog_manager.cc:5484] T d81dcfefb4fd401b9cfd2ca36034a010 P 6398c8993c3c45dcaab25b42142d5671 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "c843193b801b4f1a88f630ed3ed0f375" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } }
I20240503 02:00:34.896010 17751 raft_consensus.cc:2932] T d81dcfefb4fd401b9cfd2ca36034a010 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } }
I20240503 02:00:34.896967 17486 raft_consensus.cc:2932] T d81dcfefb4fd401b9cfd2ca36034a010 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } }
I20240503 02:00:34.898726 17353 raft_consensus.cc:2932] T d81dcfefb4fd401b9cfd2ca36034a010 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } }
I20240503 02:00:34.903754 17620 consensus_queue.cc:237] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } }
I20240503 02:00:34.905756 17090 raft_consensus.cc:1261] T d81dcfefb4fd401b9cfd2ca36034a010 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer c843193b801b4f1a88f630ed3ed0f375: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:34.906059 17751 raft_consensus.cc:1261] T d81dcfefb4fd401b9cfd2ca36034a010 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer c843193b801b4f1a88f630ed3ed0f375: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:34.906128 20934 consensus_queue.cc:1035] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:34.906373 17222 raft_consensus.cc:1261] T d81dcfefb4fd401b9cfd2ca36034a010 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer c843193b801b4f1a88f630ed3ed0f375: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:34.906486 20935 consensus_queue.cc:1035] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:34.906698 17486 raft_consensus.cc:1261] T d81dcfefb4fd401b9cfd2ca36034a010 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Refusing update from remote peer c843193b801b4f1a88f630ed3ed0f375: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:34.906759 20934 consensus_queue.cc:1035] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:34.907186 20933 consensus_queue.cc:1035] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:34.907971 16958 raft_consensus.cc:1261] T d81dcfefb4fd401b9cfd2ca36034a010 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer c843193b801b4f1a88f630ed3ed0f375: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:34.908491 20933 consensus_queue.cc:1035] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:34.909128 20934 raft_consensus.cc:2932] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:34.911199 16802 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet d81dcfefb4fd401b9cfd2ca36034a010 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 02:00:34.911190 17486 raft_consensus.cc:2932] T d81dcfefb4fd401b9cfd2ca36034a010 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:34.911166 17090 raft_consensus.cc:2932] T d81dcfefb4fd401b9cfd2ca36034a010 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:34.912101 17353 raft_consensus.cc:1261] T d81dcfefb4fd401b9cfd2ca36034a010 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Refusing update from remote peer c843193b801b4f1a88f630ed3ed0f375: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:34.913079 17751 raft_consensus.cc:2932] T d81dcfefb4fd401b9cfd2ca36034a010 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:34.913453 20932 consensus_queue.cc:1035] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:34.913637 16809 catalog_manager.cc:5484] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 reported cstate change: config changed from index 4 to 5, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "c843193b801b4f1a88f630ed3ed0f375" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:34.914304 16958 raft_consensus.cc:2932] T d81dcfefb4fd401b9cfd2ca36034a010 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:34.917555 17222 raft_consensus.cc:2932] T d81dcfefb4fd401b9cfd2ca36034a010 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:34.917687 17353 raft_consensus.cc:2932] T d81dcfefb4fd401b9cfd2ca36034a010 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
W20240503 02:00:34.920833 17554 consensus_peers.cc:485] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d81dcfefb4fd401b9cfd2ca36034a010. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:35.451695 20960 ts_tablet_manager.cc:902] T d81dcfefb4fd401b9cfd2ca36034a010 P 375d0f2ec11d4e25bd9d093bb6a5572d: Initiating tablet copy from peer c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134:44343)
I20240503 02:00:35.452244 20960 tablet_copy_client.cc:308] T d81dcfefb4fd401b9cfd2ca36034a010 P 375d0f2ec11d4e25bd9d093bb6a5572d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.134:44343
I20240503 02:00:35.457192 17640 tablet_copy_service.cc:140] P c843193b801b4f1a88f630ed3ed0f375: Received BeginTabletCopySession request for tablet d81dcfefb4fd401b9cfd2ca36034a010 from peer 375d0f2ec11d4e25bd9d093bb6a5572d ({username='slave'} at 127.0.99.136:33125)
I20240503 02:00:35.457358 17640 tablet_copy_service.cc:161] P c843193b801b4f1a88f630ed3ed0f375: Beginning new tablet copy session on tablet d81dcfefb4fd401b9cfd2ca36034a010 from peer 375d0f2ec11d4e25bd9d093bb6a5572d at {username='slave'} at 127.0.99.136:33125: session id = 375d0f2ec11d4e25bd9d093bb6a5572d-d81dcfefb4fd401b9cfd2ca36034a010
I20240503 02:00:35.458838 17640 tablet_copy_source_session.cc:205] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:35.460459 20960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d81dcfefb4fd401b9cfd2ca36034a010. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:35.463536 20960 tablet_copy_client.cc:783] T d81dcfefb4fd401b9cfd2ca36034a010 P 375d0f2ec11d4e25bd9d093bb6a5572d: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:35.463712 20960 tablet_copy_client.cc:647] T d81dcfefb4fd401b9cfd2ca36034a010 P 375d0f2ec11d4e25bd9d093bb6a5572d: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:35.466989 20960 tablet_copy_client.cc:523] T d81dcfefb4fd401b9cfd2ca36034a010 P 375d0f2ec11d4e25bd9d093bb6a5572d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:35.468405 20960 tablet_bootstrap.cc:492] T d81dcfefb4fd401b9cfd2ca36034a010 P 375d0f2ec11d4e25bd9d093bb6a5572d: Bootstrap starting.
I20240503 02:00:35.501478 20960 tablet_bootstrap.cc:492] T d81dcfefb4fd401b9cfd2ca36034a010 P 375d0f2ec11d4e25bd9d093bb6a5572d: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:35.501854 20960 tablet_bootstrap.cc:492] T d81dcfefb4fd401b9cfd2ca36034a010 P 375d0f2ec11d4e25bd9d093bb6a5572d: Bootstrap complete.
I20240503 02:00:35.502082 20960 ts_tablet_manager.cc:1367] T d81dcfefb4fd401b9cfd2ca36034a010 P 375d0f2ec11d4e25bd9d093bb6a5572d: Time spent bootstrapping tablet: real 0.034s user 0.026s sys 0.000s
I20240503 02:00:35.502641 20960 raft_consensus.cc:348] T d81dcfefb4fd401b9cfd2ca36034a010 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } }
I20240503 02:00:35.502820 20960 raft_consensus.cc:729] T d81dcfefb4fd401b9cfd2ca36034a010 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 375d0f2ec11d4e25bd9d093bb6a5572d, State: Initialized, Role: LEARNER
I20240503 02:00:35.502949 20960 consensus_queue.cc:260] T d81dcfefb4fd401b9cfd2ca36034a010 P 375d0f2ec11d4e25bd9d093bb6a5572d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } }
I20240503 02:00:35.503479 20960 ts_tablet_manager.cc:1398] T d81dcfefb4fd401b9cfd2ca36034a010 P 375d0f2ec11d4e25bd9d093bb6a5572d: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240503 02:00:35.504684 17640 tablet_copy_service.cc:342] P c843193b801b4f1a88f630ed3ed0f375: Request end of tablet copy session 375d0f2ec11d4e25bd9d093bb6a5572d-d81dcfefb4fd401b9cfd2ca36034a010 received from {username='slave'} at 127.0.99.136:33125
I20240503 02:00:35.504813 17640 tablet_copy_service.cc:433] P c843193b801b4f1a88f630ed3ed0f375: ending tablet copy session 375d0f2ec11d4e25bd9d093bb6a5572d-d81dcfefb4fd401b9cfd2ca36034a010 on tablet d81dcfefb4fd401b9cfd2ca36034a010 with peer 375d0f2ec11d4e25bd9d093bb6a5572d
I20240503 02:00:35.896562 19369 raft_consensus.cc:1203] T d81dcfefb4fd401b9cfd2ca36034a010 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 02:00:36.502408 20980 raft_consensus.cc:1050] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375: attempting to promote NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d to VOTER
I20240503 02:00:36.502952 20980 consensus_queue.cc:237] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } }
I20240503 02:00:36.505275 17090 raft_consensus.cc:1261] T d81dcfefb4fd401b9cfd2ca36034a010 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer c843193b801b4f1a88f630ed3ed0f375: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:36.505370 17750 raft_consensus.cc:1261] T d81dcfefb4fd401b9cfd2ca36034a010 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer c843193b801b4f1a88f630ed3ed0f375: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:36.505447 16959 raft_consensus.cc:1261] T d81dcfefb4fd401b9cfd2ca36034a010 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer c843193b801b4f1a88f630ed3ed0f375: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:36.505760 21005 consensus_queue.cc:1035] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:36.505873 17486 raft_consensus.cc:1261] T d81dcfefb4fd401b9cfd2ca36034a010 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Refusing update from remote peer c843193b801b4f1a88f630ed3ed0f375: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:36.506225 17223 raft_consensus.cc:1261] T d81dcfefb4fd401b9cfd2ca36034a010 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer c843193b801b4f1a88f630ed3ed0f375: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:36.506345 17353 raft_consensus.cc:1261] T d81dcfefb4fd401b9cfd2ca36034a010 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Refusing update from remote peer c843193b801b4f1a88f630ed3ed0f375: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:36.506359 19369 raft_consensus.cc:1261] T d81dcfefb4fd401b9cfd2ca36034a010 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 LEARNER]: Refusing update from remote peer c843193b801b4f1a88f630ed3ed0f375: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:36.506573 21004 consensus_queue.cc:1035] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240503 02:00:36.506780 20999 consensus_queue.cc:1035] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:36.506978 20980 consensus_queue.cc:1035] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:36.507123 21001 consensus_queue.cc:1035] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:36.507262 21002 consensus_queue.cc:1035] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:36.507527 21003 consensus_queue.cc:1035] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 [LEADER]: Connected to new peer: Peer: permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:36.511714 20999 raft_consensus.cc:2932] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:36.512355 17750 raft_consensus.cc:2932] T d81dcfefb4fd401b9cfd2ca36034a010 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:36.514211 16959 raft_consensus.cc:2932] T d81dcfefb4fd401b9cfd2ca36034a010 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:36.515517 16815 catalog_manager.cc:5484] T d81dcfefb4fd401b9cfd2ca36034a010 P 002cc3411acb4fdfa0b67c6f57f687a6 reported cstate change: config changed from index 5 to 6, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c843193b801b4f1a88f630ed3ed0f375" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:36.515517 17486 raft_consensus.cc:2932] T d81dcfefb4fd401b9cfd2ca36034a010 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:36.518003 17090 raft_consensus.cc:2932] T d81dcfefb4fd401b9cfd2ca36034a010 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:36.518306 19369 raft_consensus.cc:2932] T d81dcfefb4fd401b9cfd2ca36034a010 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:36.519282 17222 raft_consensus.cc:2932] T d81dcfefb4fd401b9cfd2ca36034a010 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:36.519716 17353 raft_consensus.cc:2932] T d81dcfefb4fd401b9cfd2ca36034a010 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:36.533126 17619 consensus_queue.cc:237] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } }
I20240503 02:00:36.534885 19369 raft_consensus.cc:1261] T d81dcfefb4fd401b9cfd2ca36034a010 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 FOLLOWER]: Refusing update from remote peer c843193b801b4f1a88f630ed3ed0f375: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:36.535040 17750 raft_consensus.cc:1261] T d81dcfefb4fd401b9cfd2ca36034a010 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer c843193b801b4f1a88f630ed3ed0f375: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:36.535465 17090 raft_consensus.cc:1261] T d81dcfefb4fd401b9cfd2ca36034a010 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer c843193b801b4f1a88f630ed3ed0f375: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:36.535482 21002 consensus_queue.cc:1035] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:36.535697 21003 consensus_queue.cc:1035] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 [LEADER]: Connected to new peer: Peer: permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:36.535995 21004 consensus_queue.cc:1035] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:36.536294 16959 raft_consensus.cc:1261] T d81dcfefb4fd401b9cfd2ca36034a010 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer c843193b801b4f1a88f630ed3ed0f375: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:36.536288 17486 raft_consensus.cc:1261] T d81dcfefb4fd401b9cfd2ca36034a010 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Refusing update from remote peer c843193b801b4f1a88f630ed3ed0f375: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:36.536305 17222 raft_consensus.cc:1261] T d81dcfefb4fd401b9cfd2ca36034a010 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer c843193b801b4f1a88f630ed3ed0f375: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:36.536818 21002 consensus_queue.cc:1035] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:36.537015 21004 consensus_queue.cc:1035] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:36.537948 21004 consensus_queue.cc:1035] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:36.540007 20999 raft_consensus.cc:2932] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:36.540338 17222 raft_consensus.cc:2932] T d81dcfefb4fd401b9cfd2ca36034a010 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:36.541790 17486 raft_consensus.cc:2932] T d81dcfefb4fd401b9cfd2ca36034a010 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:36.542228 16959 raft_consensus.cc:2932] T d81dcfefb4fd401b9cfd2ca36034a010 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:36.542668 17750 raft_consensus.cc:2932] T d81dcfefb4fd401b9cfd2ca36034a010 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:36.544013 19369 raft_consensus.cc:2932] T d81dcfefb4fd401b9cfd2ca36034a010 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:36.544860 17090 raft_consensus.cc:2932] T d81dcfefb4fd401b9cfd2ca36034a010 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:36.545931 16802 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d81dcfefb4fd401b9cfd2ca36034a010 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:00:36.547005 16818 catalog_manager.cc:5484] T d81dcfefb4fd401b9cfd2ca36034a010 P 16d437fe629643948c95d628926d5ffa reported cstate change: config changed from index 6 to 7, VOTER d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "c843193b801b4f1a88f630ed3ed0f375" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:36.556013 17335 tablet_service.cc:1514] Processing DeleteTablet for tablet d81dcfefb4fd401b9cfd2ca36034a010 with delete_type TABLET_DATA_TOMBSTONED (TS d43ed6b8663947d99a72cd922cc323aa not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:44362
I20240503 02:00:36.556866 21019 tablet_replica.cc:330] T d81dcfefb4fd401b9cfd2ca36034a010 P d43ed6b8663947d99a72cd922cc323aa: stopping tablet replica
I20240503 02:00:36.557248 21019 raft_consensus.cc:2229] T d81dcfefb4fd401b9cfd2ca36034a010 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:36.557714 21019 raft_consensus.cc:2258] T d81dcfefb4fd401b9cfd2ca36034a010 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:36.558579 21019 ts_tablet_manager.cc:1875] T d81dcfefb4fd401b9cfd2ca36034a010 P d43ed6b8663947d99a72cd922cc323aa: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:36.560740 21019 ts_tablet_manager.cc:1888] T d81dcfefb4fd401b9cfd2ca36034a010 P d43ed6b8663947d99a72cd922cc323aa: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 02:00:36.560927 21019 log.cc:1196] T d81dcfefb4fd401b9cfd2ca36034a010 P d43ed6b8663947d99a72cd922cc323aa: Deleting WAL directory at /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/d81dcfefb4fd401b9cfd2ca36034a010
I20240503 02:00:36.561540 16803 catalog_manager.cc:4830] TS d43ed6b8663947d99a72cd922cc323aa (127.0.99.132:40023): tablet d81dcfefb4fd401b9cfd2ca36034a010 (table rebalance_test_table_4 [id=421dd60555d242a78638e28180581a9c]) successfully deleted
I20240503 02:00:36.869168 17730 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:00:36.870503 17202 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:36.870820 19350 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:36.876792 17070 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:00:36.881721 19219 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:36.886613 16938 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:36.890594 17467 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:00:36.898437 17335 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:00:36.898538 19088 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:36.909451 17598 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:37.058686 19350 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:37.060448 16938 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:37.060809 17070 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:00:37.063331 17202 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:37.063874 17730 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:00:37.082875 19219 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:37.085789 17598 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:37.092516 19088 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:37.097848 17467 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:00:37.099892 17335 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:00:37.165127 17486 consensus_queue.cc:237] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } }
I20240503 02:00:37.167284 17750 raft_consensus.cc:1261] T d52afe1281064d6480427554ca4bb2f6 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:37.167727 16959 raft_consensus.cc:1261] T d52afe1281064d6480427554ca4bb2f6 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:37.167773 21120 consensus_queue.cc:1035] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:37.167984 17619 raft_consensus.cc:1261] T d52afe1281064d6480427554ca4bb2f6 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:37.168505 19239 raft_consensus.cc:1261] T d52afe1281064d6480427554ca4bb2f6 P 9564311af29a4493831a8db658a085c4 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:37.168475 21120 consensus_queue.cc:1035] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:37.168650 17090 raft_consensus.cc:1261] T d52afe1281064d6480427554ca4bb2f6 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:37.169286 21120 consensus_queue.cc:1035] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240503 02:00:37.169814 17422 consensus_peers.cc:485] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d52afe1281064d6480427554ca4bb2f6. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:37.169868 17222 raft_consensus.cc:1261] T d52afe1281064d6480427554ca4bb2f6 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:37.170424 21120 consensus_queue.cc:1035] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:37.170612 20519 consensus_queue.cc:1035] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:37.171044 20519 consensus_queue.cc:1035] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:37.171630 21120 raft_consensus.cc:2932] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:37.172125 17090 raft_consensus.cc:2932] T d52afe1281064d6480427554ca4bb2f6 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:37.174892 16959 raft_consensus.cc:2932] T d52afe1281064d6480427554ca4bb2f6 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:37.175077 16817 catalog_manager.cc:5484] T d52afe1281064d6480427554ca4bb2f6 P 89dd09f8aaee4b249ae3ce924a422a67 reported cstate change: config changed from index 7 to 8, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "16d437fe629643948c95d628926d5ffa" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:37.176056 17750 raft_consensus.cc:2932] T d52afe1281064d6480427554ca4bb2f6 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:37.179580 17619 raft_consensus.cc:2932] T d52afe1281064d6480427554ca4bb2f6 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:37.183785 17222 raft_consensus.cc:2932] T d52afe1281064d6480427554ca4bb2f6 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:37.183760 19239 raft_consensus.cc:2932] T d52afe1281064d6480427554ca4bb2f6 P 9564311af29a4493831a8db658a085c4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:37.190333 17750 consensus_queue.cc:237] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } }
I20240503 02:00:37.192764 19239 raft_consensus.cc:1261] T 78d106d705df4089a4dfae4d5dc533cc P 9564311af29a4493831a8db658a085c4 [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:37.192971 17222 raft_consensus.cc:1261] T 78d106d705df4089a4dfae4d5dc533cc P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:37.193090 16959 raft_consensus.cc:1261] T 78d106d705df4089a4dfae4d5dc533cc P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:37.193173 17619 raft_consensus.cc:1261] T 78d106d705df4089a4dfae4d5dc533cc P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:37.193431 17090 raft_consensus.cc:1261] T 78d106d705df4089a4dfae4d5dc533cc P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:37.193720 21141 consensus_queue.cc:1035] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:37.193938 21142 consensus_queue.cc:1035] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:37.194028 17486 raft_consensus.cc:1261] T 78d106d705df4089a4dfae4d5dc533cc P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:37.194252 20851 consensus_queue.cc:1035] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:37.194602 20851 consensus_queue.cc:1035] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:37.195076 20851 consensus_queue.cc:1035] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:37.196045 20851 consensus_queue.cc:1035] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:37.197399 21144 raft_consensus.cc:2932] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } }
I20240503 02:00:37.197839 19239 raft_consensus.cc:2932] T 78d106d705df4089a4dfae4d5dc533cc P 9564311af29a4493831a8db658a085c4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } }
I20240503 02:00:37.197988 17090 raft_consensus.cc:2932] T 78d106d705df4089a4dfae4d5dc533cc P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } }
I20240503 02:00:37.198576 17222 raft_consensus.cc:2932] T 78d106d705df4089a4dfae4d5dc533cc P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } }
I20240503 02:00:37.200872 16959 raft_consensus.cc:2932] T 78d106d705df4089a4dfae4d5dc533cc P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } }
I20240503 02:00:37.201746 16809 catalog_manager.cc:5484] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 reported cstate change: config changed from index 7 to 8, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New cstate: current_term: 1 leader_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:37.202340 17619 raft_consensus.cc:2932] T 78d106d705df4089a4dfae4d5dc533cc P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } }
W20240503 02:00:37.207686 17686 consensus_peers.cc:485] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 78d106d705df4089a4dfae4d5dc533cc. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:37.209295 17486 raft_consensus.cc:2932] T 78d106d705df4089a4dfae4d5dc533cc P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } }
I20240503 02:00:37.214953 17750 consensus_queue.cc:237] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } attrs { replace: true } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } }
I20240503 02:00:37.217204 17619 raft_consensus.cc:1261] T 5d2104698d5f452c8350cce10afd3898 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:37.217254 17353 raft_consensus.cc:1261] T 5d2104698d5f452c8350cce10afd3898 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:37.217375 17090 raft_consensus.cc:1261] T 5d2104698d5f452c8350cce10afd3898 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:37.217650 21143 consensus_queue.cc:1035] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:37.217984 16959 raft_consensus.cc:1261] T 5d2104698d5f452c8350cce10afd3898 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:37.217995 21143 consensus_queue.cc:1035] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:37.218534 21144 consensus_queue.cc:1035] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240503 02:00:37.218688 17686 consensus_peers.cc:485] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5d2104698d5f452c8350cce10afd3898. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:37.218905 21144 consensus_queue.cc:1035] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:37.219314 17222 raft_consensus.cc:1261] T 5d2104698d5f452c8350cce10afd3898 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:37.219764 21143 consensus_queue.cc:1035] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:37.221032 19239 raft_consensus.cc:1261] T 5d2104698d5f452c8350cce10afd3898 P 9564311af29a4493831a8db658a085c4 [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:37.221469 21143 consensus_queue.cc:1035] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:37.222349 21144 raft_consensus.cc:2932] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } attrs { replace: true } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:37.222988 16959 raft_consensus.cc:2932] T 5d2104698d5f452c8350cce10afd3898 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } attrs { replace: true } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:37.224020 19239 raft_consensus.cc:2932] T 5d2104698d5f452c8350cce10afd3898 P 9564311af29a4493831a8db658a085c4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } attrs { replace: true } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:37.224623 17222 raft_consensus.cc:2932] T 5d2104698d5f452c8350cce10afd3898 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } attrs { replace: true } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:37.225066 17353 raft_consensus.cc:2932] T 5d2104698d5f452c8350cce10afd3898 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } attrs { replace: true } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:37.227620 16817 catalog_manager.cc:5484] T 5d2104698d5f452c8350cce10afd3898 P 6398c8993c3c45dcaab25b42142d5671 reported cstate change: config changed from index 7 to 8, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } attrs { replace: true } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:37.230684 17090 raft_consensus.cc:2932] T 5d2104698d5f452c8350cce10afd3898 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } attrs { replace: true } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:37.233227 17619 raft_consensus.cc:2932] T 5d2104698d5f452c8350cce10afd3898 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } attrs { replace: true } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:37.238472 17486 consensus_queue.cc:237] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } }
I20240503 02:00:37.240533 17353 raft_consensus.cc:1261] T fbec07350ad04c49afd42e5d88dda6ad P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:37.240530 19239 raft_consensus.cc:1261] T fbec07350ad04c49afd42e5d88dda6ad P 9564311af29a4493831a8db658a085c4 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240503 02:00:37.241048 17422 consensus_peers.cc:485] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fbec07350ad04c49afd42e5d88dda6ad. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:37.241142 21123 consensus_queue.cc:1035] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:37.241217 17222 raft_consensus.cc:1261] T fbec07350ad04c49afd42e5d88dda6ad P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:37.241361 21119 consensus_queue.cc:1035] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:37.241585 21121 consensus_queue.cc:1035] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:37.241688 17750 raft_consensus.cc:1261] T fbec07350ad04c49afd42e5d88dda6ad P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:37.242038 21121 consensus_queue.cc:1035] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:37.243021 16959 raft_consensus.cc:1261] T fbec07350ad04c49afd42e5d88dda6ad P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:37.243363 17090 raft_consensus.cc:1261] T fbec07350ad04c49afd42e5d88dda6ad P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:37.243391 21119 consensus_queue.cc:1035] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:37.244311 21123 consensus_queue.cc:1035] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:37.245144 21121 raft_consensus.cc:2932] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } }
I20240503 02:00:37.246358 16959 raft_consensus.cc:2932] T fbec07350ad04c49afd42e5d88dda6ad P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } }
I20240503 02:00:37.246434 17750 raft_consensus.cc:2932] T fbec07350ad04c49afd42e5d88dda6ad P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } }
I20240503 02:00:37.247293 17353 raft_consensus.cc:2932] T fbec07350ad04c49afd42e5d88dda6ad P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } }
I20240503 02:00:37.250006 16810 catalog_manager.cc:5484] T fbec07350ad04c49afd42e5d88dda6ad P 6398c8993c3c45dcaab25b42142d5671 reported cstate change: config changed from index 7 to 8, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New cstate: current_term: 1 leader_uuid: "16d437fe629643948c95d628926d5ffa" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } }
I20240503 02:00:37.252354 17090 raft_consensus.cc:2932] T fbec07350ad04c49afd42e5d88dda6ad P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } }
I20240503 02:00:37.253063 17223 raft_consensus.cc:2932] T fbec07350ad04c49afd42e5d88dda6ad P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } }
I20240503 02:00:37.260298 19239 raft_consensus.cc:2932] T fbec07350ad04c49afd42e5d88dda6ad P 9564311af29a4493831a8db658a085c4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } }
I20240503 02:00:37.265146 17090 consensus_queue.cc:237] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } }
I20240503 02:00:37.266863 19108 raft_consensus.cc:1261] T 5833b4f9a5434734817f154d660aa9f7 P 838ef968bb45484bb27f5af6f3989bae [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:37.267306 21185 consensus_queue.cc:1035] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:37.267498 17223 raft_consensus.cc:1261] T 5833b4f9a5434734817f154d660aa9f7 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:37.267561 17353 raft_consensus.cc:1261] T 5833b4f9a5434734817f154d660aa9f7 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:37.267864 17486 raft_consensus.cc:1261] T 5833b4f9a5434734817f154d660aa9f7 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:37.268030 21185 consensus_queue.cc:1035] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:37.268412 16959 raft_consensus.cc:1261] T 5833b4f9a5434734817f154d660aa9f7 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:37.268885 21186 consensus_queue.cc:1035] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:37.268976 17750 raft_consensus.cc:1261] T 5833b4f9a5434734817f154d660aa9f7 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:37.269462 21187 consensus_queue.cc:1035] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:37.269807 21187 consensus_queue.cc:1035] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:37.270021 21186 consensus_queue.cc:1035] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240503 02:00:37.270323 17026 consensus_peers.cc:485] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5833b4f9a5434734817f154d660aa9f7. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:37.272609 20473 raft_consensus.cc:2932] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:37.273118 17486 raft_consensus.cc:2932] T 5833b4f9a5434734817f154d660aa9f7 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:37.273151 16959 raft_consensus.cc:2932] T 5833b4f9a5434734817f154d660aa9f7 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:37.274740 17223 raft_consensus.cc:2932] T 5833b4f9a5434734817f154d660aa9f7 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:37.276160 16813 catalog_manager.cc:5484] T 5833b4f9a5434734817f154d660aa9f7 P 16d437fe629643948c95d628926d5ffa reported cstate change: config changed from index 7 to 8, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "89dd09f8aaee4b249ae3ce924a422a67" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:37.277053 17353 raft_consensus.cc:2932] T 5833b4f9a5434734817f154d660aa9f7 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:37.280746 17750 raft_consensus.cc:2932] T 5833b4f9a5434734817f154d660aa9f7 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:37.287019 19108 raft_consensus.cc:2932] T 5833b4f9a5434734817f154d660aa9f7 P 838ef968bb45484bb27f5af6f3989bae [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:37.300161 17223 consensus_queue.cc:237] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } attrs { replace: true } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } }
I20240503 02:00:37.302711 17090 raft_consensus.cc:1261] T 4782de9826d6460e9e993e01327034e4 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer 5b8291b26a7f448b8636187d1caf86d0: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:37.303359 19369 raft_consensus.cc:1261] T 4782de9826d6460e9e993e01327034e4 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 FOLLOWER]: Refusing update from remote peer 5b8291b26a7f448b8636187d1caf86d0: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:37.303802 17750 raft_consensus.cc:1261] T 4782de9826d6460e9e993e01327034e4 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer 5b8291b26a7f448b8636187d1caf86d0: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:37.304023 17486 raft_consensus.cc:1261] T 4782de9826d6460e9e993e01327034e4 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Refusing update from remote peer 5b8291b26a7f448b8636187d1caf86d0: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:37.304420 17619 raft_consensus.cc:1261] T 4782de9826d6460e9e993e01327034e4 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer 5b8291b26a7f448b8636187d1caf86d0: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:37.304848 21202 consensus_queue.cc:1035] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:37.305197 21202 consensus_queue.cc:1035] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:37.305356 20734 consensus_queue.cc:1035] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:37.305532 21201 consensus_queue.cc:1035] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:37.305847 20734 consensus_queue.cc:1035] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:37.306115 16959 raft_consensus.cc:1261] T 4782de9826d6460e9e993e01327034e4 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 5b8291b26a7f448b8636187d1caf86d0: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240503 02:00:37.311617 17158 consensus_peers.cc:485] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4782de9826d6460e9e993e01327034e4. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:37.311573 21201 raft_consensus.cc:2932] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } attrs { replace: true } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } }
I20240503 02:00:37.312402 21204 consensus_queue.cc:1035] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:37.313252 17486 raft_consensus.cc:2932] T 4782de9826d6460e9e993e01327034e4 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } attrs { replace: true } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } }
I20240503 02:00:37.314081 16959 raft_consensus.cc:2932] T 4782de9826d6460e9e993e01327034e4 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } attrs { replace: true } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } }
I20240503 02:00:37.314208 17090 raft_consensus.cc:2932] T 4782de9826d6460e9e993e01327034e4 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } attrs { replace: true } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } }
I20240503 02:00:37.314997 19369 raft_consensus.cc:2932] T 4782de9826d6460e9e993e01327034e4 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } attrs { replace: true } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } }
I20240503 02:00:37.316584 17750 raft_consensus.cc:2932] T 4782de9826d6460e9e993e01327034e4 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } attrs { replace: true } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } }
I20240503 02:00:37.318161 16813 catalog_manager.cc:5484] T 4782de9826d6460e9e993e01327034e4 P 89dd09f8aaee4b249ae3ce924a422a67 reported cstate change: config changed from index 7 to 8, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New cstate: current_term: 1 leader_uuid: "5b8291b26a7f448b8636187d1caf86d0" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } attrs { replace: true } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } }
I20240503 02:00:37.319257 17619 raft_consensus.cc:2932] T 4782de9826d6460e9e993e01327034e4 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } attrs { replace: true } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } }
I20240503 02:00:37.334658 17353 consensus_queue.cc:237] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } }
I20240503 02:00:37.337256 16959 raft_consensus.cc:1261] T 63962b169caf4f14888871e246c00075 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:37.337349 19239 raft_consensus.cc:1261] T 63962b169caf4f14888871e246c00075 P 9564311af29a4493831a8db658a085c4 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:37.337725 21016 consensus_queue.cc:1035] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:37.337967 17223 raft_consensus.cc:1261] T 63962b169caf4f14888871e246c00075 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:37.338418 21016 consensus_queue.cc:1035] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:37.338625 17750 raft_consensus.cc:1261] T 63962b169caf4f14888871e246c00075 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:37.339190 21223 consensus_queue.cc:1035] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:37.339263 17090 raft_consensus.cc:1261] T 63962b169caf4f14888871e246c00075 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:37.339324 17487 raft_consensus.cc:1261] T 63962b169caf4f14888871e246c00075 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:37.340035 21224 consensus_queue.cc:1035] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:37.340399 21224 consensus_queue.cc:1035] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:37.340740 21224 consensus_queue.cc:1035] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:37.343206 21219 raft_consensus.cc:2932] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:37.344055 17223 raft_consensus.cc:2932] T 63962b169caf4f14888871e246c00075 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:37.346307 16959 raft_consensus.cc:2932] T 63962b169caf4f14888871e246c00075 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:37.346423 17090 raft_consensus.cc:2932] T 63962b169caf4f14888871e246c00075 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:37.346436 19239 raft_consensus.cc:2932] T 63962b169caf4f14888871e246c00075 P 9564311af29a4493831a8db658a085c4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:37.349529 16817 catalog_manager.cc:5484] T 63962b169caf4f14888871e246c00075 P 6398c8993c3c45dcaab25b42142d5671 reported cstate change: config changed from index 7 to 8, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "d43ed6b8663947d99a72cd922cc323aa" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:37.350466 17487 raft_consensus.cc:2932] T 63962b169caf4f14888871e246c00075 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
W20240503 02:00:37.355677 17290 consensus_peers.cc:485] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 63962b169caf4f14888871e246c00075. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:37.356798 17750 raft_consensus.cc:2932] T 63962b169caf4f14888871e246c00075 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:37.364373 17619 consensus_queue.cc:237] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } }
I20240503 02:00:37.366412 17090 raft_consensus.cc:1261] T d81dcfefb4fd401b9cfd2ca36034a010 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer c843193b801b4f1a88f630ed3ed0f375: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:37.366413 17487 raft_consensus.cc:1261] T d81dcfefb4fd401b9cfd2ca36034a010 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Refusing update from remote peer c843193b801b4f1a88f630ed3ed0f375: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:37.366860 16959 raft_consensus.cc:1261] T d81dcfefb4fd401b9cfd2ca36034a010 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer c843193b801b4f1a88f630ed3ed0f375: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:37.366837 21239 consensus_queue.cc:1035] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:37.367049 17750 raft_consensus.cc:1261] T d81dcfefb4fd401b9cfd2ca36034a010 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer c843193b801b4f1a88f630ed3ed0f375: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:37.367475 20999 consensus_queue.cc:1035] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:37.367708 17223 raft_consensus.cc:1261] T d81dcfefb4fd401b9cfd2ca36034a010 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer c843193b801b4f1a88f630ed3ed0f375: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:37.367736 19369 raft_consensus.cc:1261] T d81dcfefb4fd401b9cfd2ca36034a010 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 FOLLOWER]: Refusing update from remote peer c843193b801b4f1a88f630ed3ed0f375: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:37.367923 20999 consensus_queue.cc:1035] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:37.368211 21237 consensus_queue.cc:1035] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 [LEADER]: Connected to new peer: Peer: permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:37.368583 21237 consensus_queue.cc:1035] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:37.370538 20999 consensus_queue.cc:1035] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:37.371044 21237 raft_consensus.cc:2932] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } }
I20240503 02:00:37.371614 16959 raft_consensus.cc:2932] T d81dcfefb4fd401b9cfd2ca36034a010 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } }
I20240503 02:00:37.371662 19369 raft_consensus.cc:2932] T d81dcfefb4fd401b9cfd2ca36034a010 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } }
W20240503 02:00:37.373565 17554 consensus_peers.cc:485] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d81dcfefb4fd401b9cfd2ca36034a010. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:37.373860 17223 raft_consensus.cc:2932] T d81dcfefb4fd401b9cfd2ca36034a010 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } }
I20240503 02:00:37.374872 17487 raft_consensus.cc:2932] T d81dcfefb4fd401b9cfd2ca36034a010 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } }
I20240503 02:00:37.375090 16811 catalog_manager.cc:5484] T d81dcfefb4fd401b9cfd2ca36034a010 P 375d0f2ec11d4e25bd9d093bb6a5572d reported cstate change: config changed from index 7 to 8, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New cstate: current_term: 1 leader_uuid: "c843193b801b4f1a88f630ed3ed0f375" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } }
I20240503 02:00:37.379633 17750 raft_consensus.cc:2932] T d81dcfefb4fd401b9cfd2ca36034a010 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } }
I20240503 02:00:37.379817 17090 raft_consensus.cc:2932] T d81dcfefb4fd401b9cfd2ca36034a010 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } }
I20240503 02:00:37.389925 17487 consensus_queue.cc:237] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } attrs { replace: true } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } }
I20240503 02:00:37.392602 17750 raft_consensus.cc:1261] T ec69ad86be3d4793b10aca98dc4c1736 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:37.392602 16959 raft_consensus.cc:1261] T ec69ad86be3d4793b10aca98dc4c1736 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:37.392634 19108 raft_consensus.cc:1261] T ec69ad86be3d4793b10aca98dc4c1736 P 838ef968bb45484bb27f5af6f3989bae [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:37.392690 17090 raft_consensus.cc:1261] T ec69ad86be3d4793b10aca98dc4c1736 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:37.393213 17223 raft_consensus.cc:1261] T ec69ad86be3d4793b10aca98dc4c1736 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:37.393188 21130 consensus_queue.cc:1035] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:37.393239 17619 raft_consensus.cc:1261] T ec69ad86be3d4793b10aca98dc4c1736 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240503 02:00:37.393452 17422 consensus_peers.cc:485] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ec69ad86be3d4793b10aca98dc4c1736. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:37.393677 21130 consensus_queue.cc:1035] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:37.393836 21121 consensus_queue.cc:1035] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:37.394224 21121 consensus_queue.cc:1035] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:37.394353 21120 consensus_queue.cc:1035] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:37.397652 21130 consensus_queue.cc:1035] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:37.398073 21130 raft_consensus.cc:2932] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } attrs { replace: true } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:37.398569 16959 raft_consensus.cc:2932] T ec69ad86be3d4793b10aca98dc4c1736 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } attrs { replace: true } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:37.400523 17090 raft_consensus.cc:2932] T ec69ad86be3d4793b10aca98dc4c1736 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } attrs { replace: true } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:37.400977 17619 raft_consensus.cc:2932] T ec69ad86be3d4793b10aca98dc4c1736 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } attrs { replace: true } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:37.402071 16810 catalog_manager.cc:5484] T ec69ad86be3d4793b10aca98dc4c1736 P 6398c8993c3c45dcaab25b42142d5671 reported cstate change: config changed from index 7 to 8, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "16d437fe629643948c95d628926d5ffa" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } attrs { replace: true } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:37.402806 19108 raft_consensus.cc:2932] T ec69ad86be3d4793b10aca98dc4c1736 P 838ef968bb45484bb27f5af6f3989bae [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } attrs { replace: true } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:37.403952 17750 raft_consensus.cc:2932] T ec69ad86be3d4793b10aca98dc4c1736 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } attrs { replace: true } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:37.406399 17223 raft_consensus.cc:2932] T ec69ad86be3d4793b10aca98dc4c1736 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } attrs { replace: true } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:37.418291 17353 consensus_queue.cc:237] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } }
I20240503 02:00:37.420295 17223 raft_consensus.cc:1261] T ab536115ac7a4ba193f718d421f4000d P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:37.420439 16959 raft_consensus.cc:1261] T ab536115ac7a4ba193f718d421f4000d P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:37.420461 19239 raft_consensus.cc:1261] T ab536115ac7a4ba193f718d421f4000d P 9564311af29a4493831a8db658a085c4 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:37.420764 17619 raft_consensus.cc:1261] T ab536115ac7a4ba193f718d421f4000d P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:37.420795 21221 consensus_queue.cc:1035] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:37.420884 17750 raft_consensus.cc:1261] T ab536115ac7a4ba193f718d421f4000d P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:37.421216 17090 raft_consensus.cc:1261] T ab536115ac7a4ba193f718d421f4000d P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240503 02:00:37.421366 17290 consensus_peers.cc:485] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ab536115ac7a4ba193f718d421f4000d. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:37.421772 21220 consensus_queue.cc:1035] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:37.421975 21222 consensus_queue.cc:1035] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:37.422235 21016 consensus_queue.cc:1035] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:37.422361 21223 consensus_queue.cc:1035] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:37.422479 21220 consensus_queue.cc:1035] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:37.424886 21223 raft_consensus.cc:2932] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } }
I20240503 02:00:37.426008 17750 raft_consensus.cc:2932] T ab536115ac7a4ba193f718d421f4000d P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } }
I20240503 02:00:37.427214 17223 raft_consensus.cc:2932] T ab536115ac7a4ba193f718d421f4000d P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } }
I20240503 02:00:37.428766 16810 catalog_manager.cc:5484] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa reported cstate change: config changed from index 7 to 8, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New cstate: current_term: 1 leader_uuid: "d43ed6b8663947d99a72cd922cc323aa" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:37.429790 16959 raft_consensus.cc:2932] T ab536115ac7a4ba193f718d421f4000d P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } }
I20240503 02:00:37.430213 17619 raft_consensus.cc:2932] T ab536115ac7a4ba193f718d421f4000d P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } }
I20240503 02:00:37.433709 17090 raft_consensus.cc:2932] T ab536115ac7a4ba193f718d421f4000d P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } }
I20240503 02:00:37.436782 19239 raft_consensus.cc:2932] T ab536115ac7a4ba193f718d421f4000d P 9564311af29a4493831a8db658a085c4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } }
I20240503 02:00:37.627450 21317 ts_tablet_manager.cc:902] T 5d2104698d5f452c8350cce10afd3898 P 375d0f2ec11d4e25bd9d093bb6a5572d: Initiating tablet copy from peer 002cc3411acb4fdfa0b67c6f57f687a6 (127.0.99.135:38989)
I20240503 02:00:37.628204 21317 tablet_copy_client.cc:308] T 5d2104698d5f452c8350cce10afd3898 P 375d0f2ec11d4e25bd9d093bb6a5572d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.135:38989
I20240503 02:00:37.628875 17772 tablet_copy_service.cc:140] P 002cc3411acb4fdfa0b67c6f57f687a6: Received BeginTabletCopySession request for tablet 5d2104698d5f452c8350cce10afd3898 from peer 375d0f2ec11d4e25bd9d093bb6a5572d ({username='slave'} at 127.0.99.136:48307)
I20240503 02:00:37.628984 17772 tablet_copy_service.cc:161] P 002cc3411acb4fdfa0b67c6f57f687a6: Beginning new tablet copy session on tablet 5d2104698d5f452c8350cce10afd3898 from peer 375d0f2ec11d4e25bd9d093bb6a5572d at {username='slave'} at 127.0.99.136:48307: session id = 375d0f2ec11d4e25bd9d093bb6a5572d-5d2104698d5f452c8350cce10afd3898
I20240503 02:00:37.629714 17772 tablet_copy_source_session.cc:205] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:37.630698 21317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d2104698d5f452c8350cce10afd3898. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:37.632735 21317 tablet_copy_client.cc:783] T 5d2104698d5f452c8350cce10afd3898 P 375d0f2ec11d4e25bd9d093bb6a5572d: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:37.633018 21317 tablet_copy_client.cc:647] T 5d2104698d5f452c8350cce10afd3898 P 375d0f2ec11d4e25bd9d093bb6a5572d: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:37.634409 21317 tablet_copy_client.cc:523] T 5d2104698d5f452c8350cce10afd3898 P 375d0f2ec11d4e25bd9d093bb6a5572d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:37.636292 21317 tablet_bootstrap.cc:492] T 5d2104698d5f452c8350cce10afd3898 P 375d0f2ec11d4e25bd9d093bb6a5572d: Bootstrap starting.
I20240503 02:00:37.656538 21317 tablet_bootstrap.cc:492] T 5d2104698d5f452c8350cce10afd3898 P 375d0f2ec11d4e25bd9d093bb6a5572d: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:37.656920 21317 tablet_bootstrap.cc:492] T 5d2104698d5f452c8350cce10afd3898 P 375d0f2ec11d4e25bd9d093bb6a5572d: Bootstrap complete.
I20240503 02:00:37.657114 21317 ts_tablet_manager.cc:1367] T 5d2104698d5f452c8350cce10afd3898 P 375d0f2ec11d4e25bd9d093bb6a5572d: Time spent bootstrapping tablet: real 0.021s user 0.017s sys 0.004s
I20240503 02:00:37.657588 21317 raft_consensus.cc:348] T 5d2104698d5f452c8350cce10afd3898 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } attrs { replace: true } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } }
I20240503 02:00:37.657724 21317 raft_consensus.cc:729] T 5d2104698d5f452c8350cce10afd3898 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 375d0f2ec11d4e25bd9d093bb6a5572d, State: Initialized, Role: LEARNER
I20240503 02:00:37.657829 21317 consensus_queue.cc:260] T 5d2104698d5f452c8350cce10afd3898 P 375d0f2ec11d4e25bd9d093bb6a5572d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } attrs { replace: true } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } }
I20240503 02:00:37.658093 21317 ts_tablet_manager.cc:1398] T 5d2104698d5f452c8350cce10afd3898 P 375d0f2ec11d4e25bd9d093bb6a5572d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:37.658622 17772 tablet_copy_service.cc:342] P 002cc3411acb4fdfa0b67c6f57f687a6: Request end of tablet copy session 375d0f2ec11d4e25bd9d093bb6a5572d-5d2104698d5f452c8350cce10afd3898 received from {username='slave'} at 127.0.99.136:48307
I20240503 02:00:37.658732 17772 tablet_copy_service.cc:433] P 002cc3411acb4fdfa0b67c6f57f687a6: ending tablet copy session 375d0f2ec11d4e25bd9d093bb6a5572d-5d2104698d5f452c8350cce10afd3898 on tablet 5d2104698d5f452c8350cce10afd3898 with peer 375d0f2ec11d4e25bd9d093bb6a5572d
I20240503 02:00:37.760818 21317 ts_tablet_manager.cc:902] T d52afe1281064d6480427554ca4bb2f6 P 375d0f2ec11d4e25bd9d093bb6a5572d: Initiating tablet copy from peer 16d437fe629643948c95d628926d5ffa (127.0.99.133:46581)
I20240503 02:00:37.761332 21317 tablet_copy_client.cc:308] T d52afe1281064d6480427554ca4bb2f6 P 375d0f2ec11d4e25bd9d093bb6a5572d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.133:46581
I20240503 02:00:37.762135 17508 tablet_copy_service.cc:140] P 16d437fe629643948c95d628926d5ffa: Received BeginTabletCopySession request for tablet d52afe1281064d6480427554ca4bb2f6 from peer 375d0f2ec11d4e25bd9d093bb6a5572d ({username='slave'} at 127.0.99.136:36855)
I20240503 02:00:37.762270 17508 tablet_copy_service.cc:161] P 16d437fe629643948c95d628926d5ffa: Beginning new tablet copy session on tablet d52afe1281064d6480427554ca4bb2f6 from peer 375d0f2ec11d4e25bd9d093bb6a5572d at {username='slave'} at 127.0.99.136:36855: session id = 375d0f2ec11d4e25bd9d093bb6a5572d-d52afe1281064d6480427554ca4bb2f6
I20240503 02:00:37.763355 17508 tablet_copy_source_session.cc:205] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:37.764508 21317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d52afe1281064d6480427554ca4bb2f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:37.766911 21317 tablet_copy_client.cc:783] T d52afe1281064d6480427554ca4bb2f6 P 375d0f2ec11d4e25bd9d093bb6a5572d: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:37.767068 21317 tablet_copy_client.cc:647] T d52afe1281064d6480427554ca4bb2f6 P 375d0f2ec11d4e25bd9d093bb6a5572d: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:37.769418 21317 tablet_copy_client.cc:523] T d52afe1281064d6480427554ca4bb2f6 P 375d0f2ec11d4e25bd9d093bb6a5572d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:37.770917 21317 tablet_bootstrap.cc:492] T d52afe1281064d6480427554ca4bb2f6 P 375d0f2ec11d4e25bd9d093bb6a5572d: Bootstrap starting.
I20240503 02:00:37.772851 21377 ts_tablet_manager.cc:902] T 78d106d705df4089a4dfae4d5dc533cc P 838ef968bb45484bb27f5af6f3989bae: Initiating tablet copy from peer 002cc3411acb4fdfa0b67c6f57f687a6 (127.0.99.135:38989)
I20240503 02:00:37.772907 21378 ts_tablet_manager.cc:902] T 4782de9826d6460e9e993e01327034e4 P 838ef968bb45484bb27f5af6f3989bae: Initiating tablet copy from peer 5b8291b26a7f448b8636187d1caf86d0 (127.0.99.131:44479)
I20240503 02:00:37.773551 21377 tablet_copy_client.cc:308] T 78d106d705df4089a4dfae4d5dc533cc P 838ef968bb45484bb27f5af6f3989bae: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.135:38989
I20240503 02:00:37.773659 21378 tablet_copy_client.cc:308] T 4782de9826d6460e9e993e01327034e4 P 838ef968bb45484bb27f5af6f3989bae: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:44479
I20240503 02:00:37.778661 21390 ts_tablet_manager.cc:902] T d81dcfefb4fd401b9cfd2ca36034a010 P 838ef968bb45484bb27f5af6f3989bae: Initiating tablet copy from peer c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134:44343)
I20240503 02:00:37.779088 17772 tablet_copy_service.cc:140] P 002cc3411acb4fdfa0b67c6f57f687a6: Received BeginTabletCopySession request for tablet 78d106d705df4089a4dfae4d5dc533cc from peer 838ef968bb45484bb27f5af6f3989bae ({username='slave'} at 127.0.99.137:44593)
I20240503 02:00:37.779206 17772 tablet_copy_service.cc:161] P 002cc3411acb4fdfa0b67c6f57f687a6: Beginning new tablet copy session on tablet 78d106d705df4089a4dfae4d5dc533cc from peer 838ef968bb45484bb27f5af6f3989bae at {username='slave'} at 127.0.99.137:44593: session id = 838ef968bb45484bb27f5af6f3989bae-78d106d705df4089a4dfae4d5dc533cc
I20240503 02:00:37.779284 21390 tablet_copy_client.cc:308] T d81dcfefb4fd401b9cfd2ca36034a010 P 838ef968bb45484bb27f5af6f3989bae: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.134:44343
I20240503 02:00:37.780386 17772 tablet_copy_source_session.cc:205] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:37.781577 21377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78d106d705df4089a4dfae4d5dc533cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:37.783999 21377 tablet_copy_client.cc:783] T 78d106d705df4089a4dfae4d5dc533cc P 838ef968bb45484bb27f5af6f3989bae: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:37.784180 21377 tablet_copy_client.cc:647] T 78d106d705df4089a4dfae4d5dc533cc P 838ef968bb45484bb27f5af6f3989bae: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:37.785632 17640 tablet_copy_service.cc:140] P c843193b801b4f1a88f630ed3ed0f375: Received BeginTabletCopySession request for tablet d81dcfefb4fd401b9cfd2ca36034a010 from peer 838ef968bb45484bb27f5af6f3989bae ({username='slave'} at 127.0.99.137:38823)
I20240503 02:00:37.785773 17640 tablet_copy_service.cc:161] P c843193b801b4f1a88f630ed3ed0f375: Beginning new tablet copy session on tablet d81dcfefb4fd401b9cfd2ca36034a010 from peer 838ef968bb45484bb27f5af6f3989bae at {username='slave'} at 127.0.99.137:38823: session id = 838ef968bb45484bb27f5af6f3989bae-d81dcfefb4fd401b9cfd2ca36034a010
I20240503 02:00:37.786818 17640 tablet_copy_source_session.cc:205] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:37.787819 21395 ts_tablet_manager.cc:902] T fbec07350ad04c49afd42e5d88dda6ad P 838ef968bb45484bb27f5af6f3989bae: Initiating tablet copy from peer 16d437fe629643948c95d628926d5ffa (127.0.99.133:46581)
I20240503 02:00:37.788012 17244 tablet_copy_service.cc:140] P 5b8291b26a7f448b8636187d1caf86d0: Received BeginTabletCopySession request for tablet 4782de9826d6460e9e993e01327034e4 from peer 838ef968bb45484bb27f5af6f3989bae ({username='slave'} at 127.0.99.137:39307)
I20240503 02:00:37.788148 17244 tablet_copy_service.cc:161] P 5b8291b26a7f448b8636187d1caf86d0: Beginning new tablet copy session on tablet 4782de9826d6460e9e993e01327034e4 from peer 838ef968bb45484bb27f5af6f3989bae at {username='slave'} at 127.0.99.137:39307: session id = 838ef968bb45484bb27f5af6f3989bae-4782de9826d6460e9e993e01327034e4
I20240503 02:00:37.788322 21390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d81dcfefb4fd401b9cfd2ca36034a010. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:37.789180 17244 tablet_copy_source_session.cc:205] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:37.789965 21377 tablet_copy_client.cc:523] T 78d106d705df4089a4dfae4d5dc533cc P 838ef968bb45484bb27f5af6f3989bae: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:37.790568 21390 tablet_copy_client.cc:783] T d81dcfefb4fd401b9cfd2ca36034a010 P 838ef968bb45484bb27f5af6f3989bae: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:37.790592 21378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4782de9826d6460e9e993e01327034e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:37.790767 21390 tablet_copy_client.cc:647] T d81dcfefb4fd401b9cfd2ca36034a010 P 838ef968bb45484bb27f5af6f3989bae: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:37.791409 21377 tablet_bootstrap.cc:492] T 78d106d705df4089a4dfae4d5dc533cc P 838ef968bb45484bb27f5af6f3989bae: Bootstrap starting.
I20240503 02:00:37.792233 21390 tablet_copy_client.cc:523] T d81dcfefb4fd401b9cfd2ca36034a010 P 838ef968bb45484bb27f5af6f3989bae: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:37.793550 21390 tablet_bootstrap.cc:492] T d81dcfefb4fd401b9cfd2ca36034a010 P 838ef968bb45484bb27f5af6f3989bae: Bootstrap starting.
I20240503 02:00:37.793823 21378 tablet_copy_client.cc:783] T 4782de9826d6460e9e993e01327034e4 P 838ef968bb45484bb27f5af6f3989bae: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:37.794108 21378 tablet_copy_client.cc:647] T 4782de9826d6460e9e993e01327034e4 P 838ef968bb45484bb27f5af6f3989bae: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:37.796362 21378 tablet_copy_client.cc:523] T 4782de9826d6460e9e993e01327034e4 P 838ef968bb45484bb27f5af6f3989bae: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:37.797793 21395 tablet_copy_client.cc:308] T fbec07350ad04c49afd42e5d88dda6ad P 838ef968bb45484bb27f5af6f3989bae: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.133:46581
I20240503 02:00:37.797947 21378 tablet_bootstrap.cc:492] T 4782de9826d6460e9e993e01327034e4 P 838ef968bb45484bb27f5af6f3989bae: Bootstrap starting.
I20240503 02:00:37.799480 17508 tablet_copy_service.cc:140] P 16d437fe629643948c95d628926d5ffa: Received BeginTabletCopySession request for tablet fbec07350ad04c49afd42e5d88dda6ad from peer 838ef968bb45484bb27f5af6f3989bae ({username='slave'} at 127.0.99.137:37139)
I20240503 02:00:37.799600 17508 tablet_copy_service.cc:161] P 16d437fe629643948c95d628926d5ffa: Beginning new tablet copy session on tablet fbec07350ad04c49afd42e5d88dda6ad from peer 838ef968bb45484bb27f5af6f3989bae at {username='slave'} at 127.0.99.137:37139: session id = 838ef968bb45484bb27f5af6f3989bae-fbec07350ad04c49afd42e5d88dda6ad
I20240503 02:00:37.800674 17508 tablet_copy_source_session.cc:205] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:37.801790 21395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbec07350ad04c49afd42e5d88dda6ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:37.804224 21395 tablet_copy_client.cc:783] T fbec07350ad04c49afd42e5d88dda6ad P 838ef968bb45484bb27f5af6f3989bae: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:37.804436 21395 tablet_copy_client.cc:647] T fbec07350ad04c49afd42e5d88dda6ad P 838ef968bb45484bb27f5af6f3989bae: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:37.806473 21395 tablet_copy_client.cc:523] T fbec07350ad04c49afd42e5d88dda6ad P 838ef968bb45484bb27f5af6f3989bae: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:37.806914 21317 tablet_bootstrap.cc:492] T d52afe1281064d6480427554ca4bb2f6 P 375d0f2ec11d4e25bd9d093bb6a5572d: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:37.807216 21317 tablet_bootstrap.cc:492] T d52afe1281064d6480427554ca4bb2f6 P 375d0f2ec11d4e25bd9d093bb6a5572d: Bootstrap complete.
I20240503 02:00:37.807430 21317 ts_tablet_manager.cc:1367] T d52afe1281064d6480427554ca4bb2f6 P 375d0f2ec11d4e25bd9d093bb6a5572d: Time spent bootstrapping tablet: real 0.037s user 0.028s sys 0.000s
I20240503 02:00:37.807785 21395 tablet_bootstrap.cc:492] T fbec07350ad04c49afd42e5d88dda6ad P 838ef968bb45484bb27f5af6f3989bae: Bootstrap starting.
I20240503 02:00:37.808598 21317 raft_consensus.cc:348] T d52afe1281064d6480427554ca4bb2f6 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } }
I20240503 02:00:37.808799 21317 raft_consensus.cc:729] T d52afe1281064d6480427554ca4bb2f6 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 375d0f2ec11d4e25bd9d093bb6a5572d, State: Initialized, Role: LEARNER
I20240503 02:00:37.808919 21317 consensus_queue.cc:260] T d52afe1281064d6480427554ca4bb2f6 P 375d0f2ec11d4e25bd9d093bb6a5572d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } }
I20240503 02:00:37.809283 21317 ts_tablet_manager.cc:1398] T d52afe1281064d6480427554ca4bb2f6 P 375d0f2ec11d4e25bd9d093bb6a5572d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:37.809800 17508 tablet_copy_service.cc:342] P 16d437fe629643948c95d628926d5ffa: Request end of tablet copy session 375d0f2ec11d4e25bd9d093bb6a5572d-d52afe1281064d6480427554ca4bb2f6 received from {username='slave'} at 127.0.99.136:36855
I20240503 02:00:37.809890 17508 tablet_copy_service.cc:433] P 16d437fe629643948c95d628926d5ffa: ending tablet copy session 375d0f2ec11d4e25bd9d093bb6a5572d-d52afe1281064d6480427554ca4bb2f6 on tablet d52afe1281064d6480427554ca4bb2f6 with peer 375d0f2ec11d4e25bd9d093bb6a5572d
I20240503 02:00:37.816388 21317 ts_tablet_manager.cc:902] T 63962b169caf4f14888871e246c00075 P 375d0f2ec11d4e25bd9d093bb6a5572d: Initiating tablet copy from peer d43ed6b8663947d99a72cd922cc323aa (127.0.99.132:40023)
I20240503 02:00:37.817013 21317 tablet_copy_client.cc:308] T 63962b169caf4f14888871e246c00075 P 375d0f2ec11d4e25bd9d093bb6a5572d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:40023
I20240503 02:00:37.823922 17376 tablet_copy_service.cc:140] P d43ed6b8663947d99a72cd922cc323aa: Received BeginTabletCopySession request for tablet 63962b169caf4f14888871e246c00075 from peer 375d0f2ec11d4e25bd9d093bb6a5572d ({username='slave'} at 127.0.99.136:54399)
I20240503 02:00:37.824065 17376 tablet_copy_service.cc:161] P d43ed6b8663947d99a72cd922cc323aa: Beginning new tablet copy session on tablet 63962b169caf4f14888871e246c00075 from peer 375d0f2ec11d4e25bd9d093bb6a5572d at {username='slave'} at 127.0.99.136:54399: session id = 375d0f2ec11d4e25bd9d093bb6a5572d-63962b169caf4f14888871e246c00075
I20240503 02:00:37.825189 17376 tablet_copy_source_session.cc:205] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:37.826078 21390 tablet_bootstrap.cc:492] T d81dcfefb4fd401b9cfd2ca36034a010 P 838ef968bb45484bb27f5af6f3989bae: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:37.826293 21317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63962b169caf4f14888871e246c00075. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:37.828133 21317 tablet_copy_client.cc:783] T 63962b169caf4f14888871e246c00075 P 375d0f2ec11d4e25bd9d093bb6a5572d: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:37.828311 21317 tablet_copy_client.cc:647] T 63962b169caf4f14888871e246c00075 P 375d0f2ec11d4e25bd9d093bb6a5572d: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:37.829847 21317 tablet_copy_client.cc:523] T 63962b169caf4f14888871e246c00075 P 375d0f2ec11d4e25bd9d093bb6a5572d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:37.830823 21317 tablet_bootstrap.cc:492] T 63962b169caf4f14888871e246c00075 P 375d0f2ec11d4e25bd9d093bb6a5572d: Bootstrap starting.
I20240503 02:00:37.832979 21390 tablet_bootstrap.cc:492] T d81dcfefb4fd401b9cfd2ca36034a010 P 838ef968bb45484bb27f5af6f3989bae: Bootstrap complete.
I20240503 02:00:37.833315 21390 ts_tablet_manager.cc:1367] T d81dcfefb4fd401b9cfd2ca36034a010 P 838ef968bb45484bb27f5af6f3989bae: Time spent bootstrapping tablet: real 0.040s user 0.026s sys 0.005s
I20240503 02:00:37.834092 21390 raft_consensus.cc:348] T d81dcfefb4fd401b9cfd2ca36034a010 P 838ef968bb45484bb27f5af6f3989bae [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } }
I20240503 02:00:37.834278 21390 raft_consensus.cc:729] T d81dcfefb4fd401b9cfd2ca36034a010 P 838ef968bb45484bb27f5af6f3989bae [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 838ef968bb45484bb27f5af6f3989bae, State: Initialized, Role: LEARNER
I20240503 02:00:37.834395 21390 consensus_queue.cc:260] T d81dcfefb4fd401b9cfd2ca36034a010 P 838ef968bb45484bb27f5af6f3989bae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } }
I20240503 02:00:37.834689 21390 ts_tablet_manager.cc:1398] T d81dcfefb4fd401b9cfd2ca36034a010 P 838ef968bb45484bb27f5af6f3989bae: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:37.835326 17640 tablet_copy_service.cc:342] P c843193b801b4f1a88f630ed3ed0f375: Request end of tablet copy session 838ef968bb45484bb27f5af6f3989bae-d81dcfefb4fd401b9cfd2ca36034a010 received from {username='slave'} at 127.0.99.137:38823
I20240503 02:00:37.835417 17640 tablet_copy_service.cc:433] P c843193b801b4f1a88f630ed3ed0f375: ending tablet copy session 838ef968bb45484bb27f5af6f3989bae-d81dcfefb4fd401b9cfd2ca36034a010 on tablet d81dcfefb4fd401b9cfd2ca36034a010 with peer 838ef968bb45484bb27f5af6f3989bae
I20240503 02:00:37.848835 21410 ts_tablet_manager.cc:902] T ab536115ac7a4ba193f718d421f4000d P 838ef968bb45484bb27f5af6f3989bae: Initiating tablet copy from peer d43ed6b8663947d99a72cd922cc323aa (127.0.99.132:40023)
I20240503 02:00:37.850981 21410 tablet_copy_client.cc:308] T ab536115ac7a4ba193f718d421f4000d P 838ef968bb45484bb27f5af6f3989bae: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:40023
I20240503 02:00:37.851637 17376 tablet_copy_service.cc:140] P d43ed6b8663947d99a72cd922cc323aa: Received BeginTabletCopySession request for tablet ab536115ac7a4ba193f718d421f4000d from peer 838ef968bb45484bb27f5af6f3989bae ({username='slave'} at 127.0.99.137:39379)
I20240503 02:00:37.851763 17376 tablet_copy_service.cc:161] P d43ed6b8663947d99a72cd922cc323aa: Beginning new tablet copy session on tablet ab536115ac7a4ba193f718d421f4000d from peer 838ef968bb45484bb27f5af6f3989bae at {username='slave'} at 127.0.99.137:39379: session id = 838ef968bb45484bb27f5af6f3989bae-ab536115ac7a4ba193f718d421f4000d
I20240503 02:00:37.852897 17376 tablet_copy_source_session.cc:205] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:37.855226 21410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab536115ac7a4ba193f718d421f4000d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:37.862849 21377 tablet_bootstrap.cc:492] T 78d106d705df4089a4dfae4d5dc533cc P 838ef968bb45484bb27f5af6f3989bae: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:37.863157 21377 tablet_bootstrap.cc:492] T 78d106d705df4089a4dfae4d5dc533cc P 838ef968bb45484bb27f5af6f3989bae: Bootstrap complete.
I20240503 02:00:37.863335 21377 ts_tablet_manager.cc:1367] T 78d106d705df4089a4dfae4d5dc533cc P 838ef968bb45484bb27f5af6f3989bae: Time spent bootstrapping tablet: real 0.072s user 0.017s sys 0.010s
I20240503 02:00:37.863775 21377 raft_consensus.cc:348] T 78d106d705df4089a4dfae4d5dc533cc P 838ef968bb45484bb27f5af6f3989bae [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } }
I20240503 02:00:37.863922 21377 raft_consensus.cc:729] T 78d106d705df4089a4dfae4d5dc533cc P 838ef968bb45484bb27f5af6f3989bae [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 838ef968bb45484bb27f5af6f3989bae, State: Initialized, Role: LEARNER
I20240503 02:00:37.864044 21377 consensus_queue.cc:260] T 78d106d705df4089a4dfae4d5dc533cc P 838ef968bb45484bb27f5af6f3989bae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } }
I20240503 02:00:37.864337 21377 ts_tablet_manager.cc:1398] T 78d106d705df4089a4dfae4d5dc533cc P 838ef968bb45484bb27f5af6f3989bae: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:37.864578 21410 tablet_copy_client.cc:783] T ab536115ac7a4ba193f718d421f4000d P 838ef968bb45484bb27f5af6f3989bae: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:37.864746 21410 tablet_copy_client.cc:647] T ab536115ac7a4ba193f718d421f4000d P 838ef968bb45484bb27f5af6f3989bae: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:37.865199 17772 tablet_copy_service.cc:342] P 002cc3411acb4fdfa0b67c6f57f687a6: Request end of tablet copy session 838ef968bb45484bb27f5af6f3989bae-78d106d705df4089a4dfae4d5dc533cc received from {username='slave'} at 127.0.99.137:44593
I20240503 02:00:37.865337 17772 tablet_copy_service.cc:433] P 002cc3411acb4fdfa0b67c6f57f687a6: ending tablet copy session 838ef968bb45484bb27f5af6f3989bae-78d106d705df4089a4dfae4d5dc533cc on tablet 78d106d705df4089a4dfae4d5dc533cc with peer 838ef968bb45484bb27f5af6f3989bae
I20240503 02:00:37.866037 21410 tablet_copy_client.cc:523] T ab536115ac7a4ba193f718d421f4000d P 838ef968bb45484bb27f5af6f3989bae: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:37.868497 21410 tablet_bootstrap.cc:492] T ab536115ac7a4ba193f718d421f4000d P 838ef968bb45484bb27f5af6f3989bae: Bootstrap starting.
I20240503 02:00:37.872005 21378 tablet_bootstrap.cc:492] T 4782de9826d6460e9e993e01327034e4 P 838ef968bb45484bb27f5af6f3989bae: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:37.872422 21378 tablet_bootstrap.cc:492] T 4782de9826d6460e9e993e01327034e4 P 838ef968bb45484bb27f5af6f3989bae: Bootstrap complete.
I20240503 02:00:37.872624 21378 ts_tablet_manager.cc:1367] T 4782de9826d6460e9e993e01327034e4 P 838ef968bb45484bb27f5af6f3989bae: Time spent bootstrapping tablet: real 0.075s user 0.031s sys 0.000s
I20240503 02:00:37.873123 21378 raft_consensus.cc:348] T 4782de9826d6460e9e993e01327034e4 P 838ef968bb45484bb27f5af6f3989bae [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } attrs { replace: true } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } }
I20240503 02:00:37.873297 21378 raft_consensus.cc:729] T 4782de9826d6460e9e993e01327034e4 P 838ef968bb45484bb27f5af6f3989bae [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 838ef968bb45484bb27f5af6f3989bae, State: Initialized, Role: LEARNER
I20240503 02:00:37.873414 21378 consensus_queue.cc:260] T 4782de9826d6460e9e993e01327034e4 P 838ef968bb45484bb27f5af6f3989bae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } attrs { replace: true } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } }
I20240503 02:00:37.873776 21378 ts_tablet_manager.cc:1398] T 4782de9826d6460e9e993e01327034e4 P 838ef968bb45484bb27f5af6f3989bae: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:37.874445 17244 tablet_copy_service.cc:342] P 5b8291b26a7f448b8636187d1caf86d0: Request end of tablet copy session 838ef968bb45484bb27f5af6f3989bae-4782de9826d6460e9e993e01327034e4 received from {username='slave'} at 127.0.99.137:39307
I20240503 02:00:37.874555 17244 tablet_copy_service.cc:433] P 5b8291b26a7f448b8636187d1caf86d0: ending tablet copy session 838ef968bb45484bb27f5af6f3989bae-4782de9826d6460e9e993e01327034e4 on tablet 4782de9826d6460e9e993e01327034e4 with peer 838ef968bb45484bb27f5af6f3989bae
I20240503 02:00:37.877375 21395 tablet_bootstrap.cc:492] T fbec07350ad04c49afd42e5d88dda6ad P 838ef968bb45484bb27f5af6f3989bae: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:37.877692 21395 tablet_bootstrap.cc:492] T fbec07350ad04c49afd42e5d88dda6ad P 838ef968bb45484bb27f5af6f3989bae: Bootstrap complete.
I20240503 02:00:37.877880 21395 ts_tablet_manager.cc:1367] T fbec07350ad04c49afd42e5d88dda6ad P 838ef968bb45484bb27f5af6f3989bae: Time spent bootstrapping tablet: real 0.070s user 0.025s sys 0.004s
I20240503 02:00:37.878352 21395 raft_consensus.cc:348] T fbec07350ad04c49afd42e5d88dda6ad P 838ef968bb45484bb27f5af6f3989bae [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } }
I20240503 02:00:37.878507 21395 raft_consensus.cc:729] T fbec07350ad04c49afd42e5d88dda6ad P 838ef968bb45484bb27f5af6f3989bae [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 838ef968bb45484bb27f5af6f3989bae, State: Initialized, Role: LEARNER
I20240503 02:00:37.878605 21395 consensus_queue.cc:260] T fbec07350ad04c49afd42e5d88dda6ad P 838ef968bb45484bb27f5af6f3989bae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } }
I20240503 02:00:37.878907 21395 ts_tablet_manager.cc:1398] T fbec07350ad04c49afd42e5d88dda6ad P 838ef968bb45484bb27f5af6f3989bae: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:37.879580 17508 tablet_copy_service.cc:342] P 16d437fe629643948c95d628926d5ffa: Request end of tablet copy session 838ef968bb45484bb27f5af6f3989bae-fbec07350ad04c49afd42e5d88dda6ad received from {username='slave'} at 127.0.99.137:37139
I20240503 02:00:37.879691 17508 tablet_copy_service.cc:433] P 16d437fe629643948c95d628926d5ffa: ending tablet copy session 838ef968bb45484bb27f5af6f3989bae-fbec07350ad04c49afd42e5d88dda6ad on tablet fbec07350ad04c49afd42e5d88dda6ad with peer 838ef968bb45484bb27f5af6f3989bae
I20240503 02:00:37.881400 21416 ts_tablet_manager.cc:902] T 5833b4f9a5434734817f154d660aa9f7 P 375d0f2ec11d4e25bd9d093bb6a5572d: Initiating tablet copy from peer 89dd09f8aaee4b249ae3ce924a422a67 (127.0.99.130:38889)
I20240503 02:00:37.882225 21416 tablet_copy_client.cc:308] T 5833b4f9a5434734817f154d660aa9f7 P 375d0f2ec11d4e25bd9d093bb6a5572d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:38889
I20240503 02:00:37.882805 17111 tablet_copy_service.cc:140] P 89dd09f8aaee4b249ae3ce924a422a67: Received BeginTabletCopySession request for tablet 5833b4f9a5434734817f154d660aa9f7 from peer 375d0f2ec11d4e25bd9d093bb6a5572d ({username='slave'} at 127.0.99.136:56011)
I20240503 02:00:37.882932 17111 tablet_copy_service.cc:161] P 89dd09f8aaee4b249ae3ce924a422a67: Beginning new tablet copy session on tablet 5833b4f9a5434734817f154d660aa9f7 from peer 375d0f2ec11d4e25bd9d093bb6a5572d at {username='slave'} at 127.0.99.136:56011: session id = 375d0f2ec11d4e25bd9d093bb6a5572d-5833b4f9a5434734817f154d660aa9f7
I20240503 02:00:37.884006 17111 tablet_copy_source_session.cc:205] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:37.885089 21416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5833b4f9a5434734817f154d660aa9f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:37.887826 21416 tablet_copy_client.cc:783] T 5833b4f9a5434734817f154d660aa9f7 P 375d0f2ec11d4e25bd9d093bb6a5572d: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:37.888044 21416 tablet_copy_client.cc:647] T 5833b4f9a5434734817f154d660aa9f7 P 375d0f2ec11d4e25bd9d093bb6a5572d: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:37.889632 21416 tablet_copy_client.cc:523] T 5833b4f9a5434734817f154d660aa9f7 P 375d0f2ec11d4e25bd9d093bb6a5572d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:37.891108 21416 tablet_bootstrap.cc:492] T 5833b4f9a5434734817f154d660aa9f7 P 375d0f2ec11d4e25bd9d093bb6a5572d: Bootstrap starting.
I20240503 02:00:37.896914 21410 tablet_bootstrap.cc:492] T ab536115ac7a4ba193f718d421f4000d P 838ef968bb45484bb27f5af6f3989bae: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:37.897281 21410 tablet_bootstrap.cc:492] T ab536115ac7a4ba193f718d421f4000d P 838ef968bb45484bb27f5af6f3989bae: Bootstrap complete.
I20240503 02:00:37.897485 21410 ts_tablet_manager.cc:1367] T ab536115ac7a4ba193f718d421f4000d P 838ef968bb45484bb27f5af6f3989bae: Time spent bootstrapping tablet: real 0.029s user 0.024s sys 0.001s
I20240503 02:00:37.897972 21410 raft_consensus.cc:348] T ab536115ac7a4ba193f718d421f4000d P 838ef968bb45484bb27f5af6f3989bae [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } }
I20240503 02:00:37.898131 21410 raft_consensus.cc:729] T ab536115ac7a4ba193f718d421f4000d P 838ef968bb45484bb27f5af6f3989bae [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 838ef968bb45484bb27f5af6f3989bae, State: Initialized, Role: LEARNER
I20240503 02:00:37.898239 21410 consensus_queue.cc:260] T ab536115ac7a4ba193f718d421f4000d P 838ef968bb45484bb27f5af6f3989bae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } }
I20240503 02:00:37.898646 21410 ts_tablet_manager.cc:1398] T ab536115ac7a4ba193f718d421f4000d P 838ef968bb45484bb27f5af6f3989bae: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 02:00:37.899359 17376 tablet_copy_service.cc:342] P d43ed6b8663947d99a72cd922cc323aa: Request end of tablet copy session 838ef968bb45484bb27f5af6f3989bae-ab536115ac7a4ba193f718d421f4000d received from {username='slave'} at 127.0.99.137:39379
I20240503 02:00:37.899459 17376 tablet_copy_service.cc:433] P d43ed6b8663947d99a72cd922cc323aa: ending tablet copy session 838ef968bb45484bb27f5af6f3989bae-ab536115ac7a4ba193f718d421f4000d on tablet ab536115ac7a4ba193f718d421f4000d with peer 838ef968bb45484bb27f5af6f3989bae
I20240503 02:00:37.909076 21317 tablet_bootstrap.cc:492] T 63962b169caf4f14888871e246c00075 P 375d0f2ec11d4e25bd9d093bb6a5572d: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:37.909418 21317 tablet_bootstrap.cc:492] T 63962b169caf4f14888871e246c00075 P 375d0f2ec11d4e25bd9d093bb6a5572d: Bootstrap complete.
I20240503 02:00:37.909615 21317 ts_tablet_manager.cc:1367] T 63962b169caf4f14888871e246c00075 P 375d0f2ec11d4e25bd9d093bb6a5572d: Time spent bootstrapping tablet: real 0.079s user 0.026s sys 0.000s
I20240503 02:00:37.910112 21317 raft_consensus.cc:348] T 63962b169caf4f14888871e246c00075 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } }
I20240503 02:00:37.910254 21317 raft_consensus.cc:729] T 63962b169caf4f14888871e246c00075 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 375d0f2ec11d4e25bd9d093bb6a5572d, State: Initialized, Role: LEARNER
I20240503 02:00:37.910382 21317 consensus_queue.cc:260] T 63962b169caf4f14888871e246c00075 P 375d0f2ec11d4e25bd9d093bb6a5572d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } }
I20240503 02:00:37.910691 21317 ts_tablet_manager.cc:1398] T 63962b169caf4f14888871e246c00075 P 375d0f2ec11d4e25bd9d093bb6a5572d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:37.911329 17376 tablet_copy_service.cc:342] P d43ed6b8663947d99a72cd922cc323aa: Request end of tablet copy session 375d0f2ec11d4e25bd9d093bb6a5572d-63962b169caf4f14888871e246c00075 received from {username='slave'} at 127.0.99.136:54399
I20240503 02:00:37.911455 17376 tablet_copy_service.cc:433] P d43ed6b8663947d99a72cd922cc323aa: ending tablet copy session 375d0f2ec11d4e25bd9d093bb6a5572d-63962b169caf4f14888871e246c00075 on tablet 63962b169caf4f14888871e246c00075 with peer 375d0f2ec11d4e25bd9d093bb6a5572d
I20240503 02:00:37.916612 21416 tablet_bootstrap.cc:492] T 5833b4f9a5434734817f154d660aa9f7 P 375d0f2ec11d4e25bd9d093bb6a5572d: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:37.917001 21416 tablet_bootstrap.cc:492] T 5833b4f9a5434734817f154d660aa9f7 P 375d0f2ec11d4e25bd9d093bb6a5572d: Bootstrap complete.
I20240503 02:00:37.917213 21416 ts_tablet_manager.cc:1367] T 5833b4f9a5434734817f154d660aa9f7 P 375d0f2ec11d4e25bd9d093bb6a5572d: Time spent bootstrapping tablet: real 0.026s user 0.021s sys 0.004s
I20240503 02:00:37.917646 21416 raft_consensus.cc:348] T 5833b4f9a5434734817f154d660aa9f7 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } }
I20240503 02:00:37.917766 21416 raft_consensus.cc:729] T 5833b4f9a5434734817f154d660aa9f7 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 375d0f2ec11d4e25bd9d093bb6a5572d, State: Initialized, Role: LEARNER
I20240503 02:00:37.917841 21416 consensus_queue.cc:260] T 5833b4f9a5434734817f154d660aa9f7 P 375d0f2ec11d4e25bd9d093bb6a5572d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } }
I20240503 02:00:37.918129 21416 ts_tablet_manager.cc:1398] T 5833b4f9a5434734817f154d660aa9f7 P 375d0f2ec11d4e25bd9d093bb6a5572d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:37.918793 17111 tablet_copy_service.cc:342] P 89dd09f8aaee4b249ae3ce924a422a67: Request end of tablet copy session 375d0f2ec11d4e25bd9d093bb6a5572d-5833b4f9a5434734817f154d660aa9f7 received from {username='slave'} at 127.0.99.136:56011
I20240503 02:00:37.918892 17111 tablet_copy_service.cc:433] P 89dd09f8aaee4b249ae3ce924a422a67: ending tablet copy session 375d0f2ec11d4e25bd9d093bb6a5572d-5833b4f9a5434734817f154d660aa9f7 on tablet 5833b4f9a5434734817f154d660aa9f7 with peer 375d0f2ec11d4e25bd9d093bb6a5572d
I20240503 02:00:37.951212 21416 ts_tablet_manager.cc:902] T ec69ad86be3d4793b10aca98dc4c1736 P 375d0f2ec11d4e25bd9d093bb6a5572d: Initiating tablet copy from peer 16d437fe629643948c95d628926d5ffa (127.0.99.133:46581)
I20240503 02:00:37.951762 21416 tablet_copy_client.cc:308] T ec69ad86be3d4793b10aca98dc4c1736 P 375d0f2ec11d4e25bd9d093bb6a5572d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.133:46581
I20240503 02:00:37.952343 17508 tablet_copy_service.cc:140] P 16d437fe629643948c95d628926d5ffa: Received BeginTabletCopySession request for tablet ec69ad86be3d4793b10aca98dc4c1736 from peer 375d0f2ec11d4e25bd9d093bb6a5572d ({username='slave'} at 127.0.99.136:36855)
I20240503 02:00:37.952455 17508 tablet_copy_service.cc:161] P 16d437fe629643948c95d628926d5ffa: Beginning new tablet copy session on tablet ec69ad86be3d4793b10aca98dc4c1736 from peer 375d0f2ec11d4e25bd9d093bb6a5572d at {username='slave'} at 127.0.99.136:36855: session id = 375d0f2ec11d4e25bd9d093bb6a5572d-ec69ad86be3d4793b10aca98dc4c1736
I20240503 02:00:37.953231 17508 tablet_copy_source_session.cc:205] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:37.954203 21416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec69ad86be3d4793b10aca98dc4c1736. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:37.956195 21416 tablet_copy_client.cc:783] T ec69ad86be3d4793b10aca98dc4c1736 P 375d0f2ec11d4e25bd9d093bb6a5572d: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:37.956377 21416 tablet_copy_client.cc:647] T ec69ad86be3d4793b10aca98dc4c1736 P 375d0f2ec11d4e25bd9d093bb6a5572d: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:37.958073 21416 tablet_copy_client.cc:523] T ec69ad86be3d4793b10aca98dc4c1736 P 375d0f2ec11d4e25bd9d093bb6a5572d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:37.959079 21416 tablet_bootstrap.cc:492] T ec69ad86be3d4793b10aca98dc4c1736 P 375d0f2ec11d4e25bd9d093bb6a5572d: Bootstrap starting.
I20240503 02:00:37.980212 21416 tablet_bootstrap.cc:492] T ec69ad86be3d4793b10aca98dc4c1736 P 375d0f2ec11d4e25bd9d093bb6a5572d: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:37.980584 21416 tablet_bootstrap.cc:492] T ec69ad86be3d4793b10aca98dc4c1736 P 375d0f2ec11d4e25bd9d093bb6a5572d: Bootstrap complete.
I20240503 02:00:37.980814 21416 ts_tablet_manager.cc:1367] T ec69ad86be3d4793b10aca98dc4c1736 P 375d0f2ec11d4e25bd9d093bb6a5572d: Time spent bootstrapping tablet: real 0.022s user 0.022s sys 0.000s
I20240503 02:00:37.981349 21416 raft_consensus.cc:348] T ec69ad86be3d4793b10aca98dc4c1736 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } attrs { replace: true } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } }
I20240503 02:00:37.981495 21416 raft_consensus.cc:729] T ec69ad86be3d4793b10aca98dc4c1736 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 375d0f2ec11d4e25bd9d093bb6a5572d, State: Initialized, Role: LEARNER
I20240503 02:00:37.981612 21416 consensus_queue.cc:260] T ec69ad86be3d4793b10aca98dc4c1736 P 375d0f2ec11d4e25bd9d093bb6a5572d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } attrs { replace: true } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } }
I20240503 02:00:37.982053 21416 ts_tablet_manager.cc:1398] T ec69ad86be3d4793b10aca98dc4c1736 P 375d0f2ec11d4e25bd9d093bb6a5572d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:37.982733 17508 tablet_copy_service.cc:342] P 16d437fe629643948c95d628926d5ffa: Request end of tablet copy session 375d0f2ec11d4e25bd9d093bb6a5572d-ec69ad86be3d4793b10aca98dc4c1736 received from {username='slave'} at 127.0.99.136:36855
I20240503 02:00:37.982864 17508 tablet_copy_service.cc:433] P 16d437fe629643948c95d628926d5ffa: ending tablet copy session 375d0f2ec11d4e25bd9d093bb6a5572d-ec69ad86be3d4793b10aca98dc4c1736 on tablet ec69ad86be3d4793b10aca98dc4c1736 with peer 375d0f2ec11d4e25bd9d093bb6a5572d
I20240503 02:00:38.185056 19369 raft_consensus.cc:1203] T d52afe1281064d6480427554ca4bb2f6 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 02:00:38.247493 19369 raft_consensus.cc:1203] T 5d2104698d5f452c8350cce10afd3898 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 02:00:38.303153 19369 raft_consensus.cc:1203] T 5833b4f9a5434734817f154d660aa9f7 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 02:00:38.323385 19108 raft_consensus.cc:1203] T fbec07350ad04c49afd42e5d88dda6ad P 838ef968bb45484bb27f5af6f3989bae [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 02:00:38.355640 19108 raft_consensus.cc:1203] T 78d106d705df4089a4dfae4d5dc533cc P 838ef968bb45484bb27f5af6f3989bae [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 02:00:38.366777 19108 raft_consensus.cc:1203] T 4782de9826d6460e9e993e01327034e4 P 838ef968bb45484bb27f5af6f3989bae [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 02:00:38.369415 19108 raft_consensus.cc:1203] T d81dcfefb4fd401b9cfd2ca36034a010 P 838ef968bb45484bb27f5af6f3989bae [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 02:00:38.411787 19108 raft_consensus.cc:1203] T ab536115ac7a4ba193f718d421f4000d P 838ef968bb45484bb27f5af6f3989bae [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 02:00:38.413856 19369 raft_consensus.cc:1203] T ec69ad86be3d4793b10aca98dc4c1736 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 02:00:38.421425 19369 raft_consensus.cc:1203] T 63962b169caf4f14888871e246c00075 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 02:00:38.617601 21478 raft_consensus.cc:1050] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa: attempting to promote NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d to VOTER
I20240503 02:00:38.618152 21478 consensus_queue.cc:237] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } }
I20240503 02:00:38.620832 17222 raft_consensus.cc:1261] T d52afe1281064d6480427554ca4bb2f6 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:38.620832 17750 raft_consensus.cc:1261] T d52afe1281064d6480427554ca4bb2f6 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:38.620986 19239 raft_consensus.cc:1261] T d52afe1281064d6480427554ca4bb2f6 P 9564311af29a4493831a8db658a085c4 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:38.621333 19369 raft_consensus.cc:1261] T d52afe1281064d6480427554ca4bb2f6 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 LEARNER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:38.621342 21589 consensus_queue.cc:1035] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:38.621552 21130 consensus_queue.cc:1035] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:38.621608 17619 raft_consensus.cc:1261] T d52afe1281064d6480427554ca4bb2f6 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:38.622284 16958 raft_consensus.cc:1261] T d52afe1281064d6480427554ca4bb2f6 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:38.622328 21589 consensus_queue.cc:1035] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:38.622418 17090 raft_consensus.cc:1261] T d52afe1281064d6480427554ca4bb2f6 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:38.622664 21478 consensus_queue.cc:1035] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:38.622849 21589 consensus_queue.cc:1035] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:38.623899 21589 consensus_queue.cc:1035] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:38.624332 21478 consensus_queue.cc:1035] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:38.626010 21121 raft_consensus.cc:2932] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:38.627247 16958 raft_consensus.cc:2932] T d52afe1281064d6480427554ca4bb2f6 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:38.627487 19369 raft_consensus.cc:2932] T d52afe1281064d6480427554ca4bb2f6 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:38.628999 19239 raft_consensus.cc:2932] T d52afe1281064d6480427554ca4bb2f6 P 9564311af29a4493831a8db658a085c4 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:38.630542 16812 catalog_manager.cc:5484] T d52afe1281064d6480427554ca4bb2f6 P 375d0f2ec11d4e25bd9d093bb6a5572d reported cstate change: config changed from index 8 to 9, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "16d437fe629643948c95d628926d5ffa" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:38.630645 17750 raft_consensus.cc:2932] T d52afe1281064d6480427554ca4bb2f6 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:38.631155 17222 raft_consensus.cc:2932] T d52afe1281064d6480427554ca4bb2f6 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:38.631433 17619 raft_consensus.cc:2932] T d52afe1281064d6480427554ca4bb2f6 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:38.636610 17090 raft_consensus.cc:2932] T d52afe1281064d6480427554ca4bb2f6 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:38.640287 17487 consensus_queue.cc:237] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } }
I20240503 02:00:38.642097 19369 raft_consensus.cc:1261] T d52afe1281064d6480427554ca4bb2f6 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:00:38.642139 17750 raft_consensus.cc:1261] T d52afe1281064d6480427554ca4bb2f6 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:00:38.642596 21589 consensus_queue.cc:1035] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:00:38.643044 19239 raft_consensus.cc:1261] T d52afe1281064d6480427554ca4bb2f6 P 9564311af29a4493831a8db658a085c4 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:00:38.643026 21478 consensus_queue.cc:1035] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:00:38.643087 16958 raft_consensus.cc:1261] T d52afe1281064d6480427554ca4bb2f6 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:00:38.643698 21478 consensus_queue.cc:1035] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:00:38.643887 17619 raft_consensus.cc:1261] T d52afe1281064d6480427554ca4bb2f6 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:00:38.644002 17090 raft_consensus.cc:1261] T d52afe1281064d6480427554ca4bb2f6 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:00:38.644063 21478 consensus_queue.cc:1035] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:00:38.644397 21478 consensus_queue.cc:1035] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:00:38.645273 21586 consensus_queue.cc:1035] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:00:38.647074 21586 raft_consensus.cc:2932] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 5b8291b26a7f448b8636187d1caf86d0 (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:38.647758 17090 raft_consensus.cc:2932] T d52afe1281064d6480427554ca4bb2f6 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 5b8291b26a7f448b8636187d1caf86d0 (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:38.647873 19239 raft_consensus.cc:2932] T d52afe1281064d6480427554ca4bb2f6 P 9564311af29a4493831a8db658a085c4 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 5b8291b26a7f448b8636187d1caf86d0 (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:38.647878 16958 raft_consensus.cc:2932] T d52afe1281064d6480427554ca4bb2f6 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 5b8291b26a7f448b8636187d1caf86d0 (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:38.649291 16802 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d52afe1281064d6480427554ca4bb2f6 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:00:38.649773 19369 raft_consensus.cc:2932] T d52afe1281064d6480427554ca4bb2f6 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 5b8291b26a7f448b8636187d1caf86d0 (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:38.650683 16811 catalog_manager.cc:5484] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa reported cstate change: config changed from index 9 to 10, VOTER 5b8291b26a7f448b8636187d1caf86d0 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "16d437fe629643948c95d628926d5ffa" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:00:38.652230 17619 raft_consensus.cc:2932] T d52afe1281064d6480427554ca4bb2f6 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 5b8291b26a7f448b8636187d1caf86d0 (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:38.652904 17750 raft_consensus.cc:2932] T d52afe1281064d6480427554ca4bb2f6 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 5b8291b26a7f448b8636187d1caf86d0 (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:38.655673 17202 tablet_service.cc:1514] Processing DeleteTablet for tablet d52afe1281064d6480427554ca4bb2f6 with delete_type TABLET_DATA_TOMBSTONED (TS 5b8291b26a7f448b8636187d1caf86d0 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:35990
I20240503 02:00:38.659752 21602 tablet_replica.cc:330] T d52afe1281064d6480427554ca4bb2f6 P 5b8291b26a7f448b8636187d1caf86d0: stopping tablet replica
I20240503 02:00:38.660131 21602 raft_consensus.cc:2229] T d52afe1281064d6480427554ca4bb2f6 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:38.660425 21602 raft_consensus.cc:2258] T d52afe1281064d6480427554ca4bb2f6 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:38.661820 21602 ts_tablet_manager.cc:1875] T d52afe1281064d6480427554ca4bb2f6 P 5b8291b26a7f448b8636187d1caf86d0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:38.664276 21602 ts_tablet_manager.cc:1888] T d52afe1281064d6480427554ca4bb2f6 P 5b8291b26a7f448b8636187d1caf86d0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 02:00:38.664371 21602 log.cc:1196] T d52afe1281064d6480427554ca4bb2f6 P 5b8291b26a7f448b8636187d1caf86d0: Deleting WAL directory at /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/d52afe1281064d6480427554ca4bb2f6
I20240503 02:00:38.665009 16803 catalog_manager.cc:4830] TS 5b8291b26a7f448b8636187d1caf86d0 (127.0.99.131:44479): tablet d52afe1281064d6480427554ca4bb2f6 (table rebalance_test_table_3 [id=78777800a49e46109c1b3bb45ab99e28]) successfully deleted
I20240503 02:00:38.747040 21484 raft_consensus.cc:1050] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6: attempting to promote NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d to VOTER
I20240503 02:00:38.747632 21484 consensus_queue.cc:237] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } attrs { replace: true } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } }
I20240503 02:00:38.749583 19239 raft_consensus.cc:1261] T 5d2104698d5f452c8350cce10afd3898 P 9564311af29a4493831a8db658a085c4 [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:38.749907 17090 raft_consensus.cc:1261] T 5d2104698d5f452c8350cce10afd3898 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:38.749897 17222 raft_consensus.cc:1261] T 5d2104698d5f452c8350cce10afd3898 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:38.750267 21144 consensus_queue.cc:1035] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:38.750528 21485 consensus_queue.cc:1035] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:38.750833 21484 consensus_queue.cc:1035] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:38.751047 19369 raft_consensus.cc:1261] T 5d2104698d5f452c8350cce10afd3898 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 LEARNER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:38.751823 16958 raft_consensus.cc:1261] T 5d2104698d5f452c8350cce10afd3898 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:38.752353 21607 consensus_queue.cc:1035] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:38.752486 17619 raft_consensus.cc:1261] T 5d2104698d5f452c8350cce10afd3898 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:38.753651 21607 consensus_queue.cc:1035] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:38.755342 17353 raft_consensus.cc:1261] T 5d2104698d5f452c8350cce10afd3898 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:38.756286 21606 consensus_queue.cc:1035] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:38.756665 21606 consensus_queue.cc:1035] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:38.759725 21607 raft_consensus.cc:2932] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } attrs { replace: true } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:38.759752 17090 raft_consensus.cc:2932] T 5d2104698d5f452c8350cce10afd3898 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } attrs { replace: true } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:38.759821 19369 raft_consensus.cc:2932] T 5d2104698d5f452c8350cce10afd3898 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } attrs { replace: true } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:38.761684 17353 raft_consensus.cc:2932] T 5d2104698d5f452c8350cce10afd3898 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } attrs { replace: true } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:38.762948 16958 raft_consensus.cc:2932] T 5d2104698d5f452c8350cce10afd3898 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } attrs { replace: true } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:38.763504 17619 raft_consensus.cc:2932] T 5d2104698d5f452c8350cce10afd3898 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } attrs { replace: true } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:38.764312 17222 raft_consensus.cc:2932] T 5d2104698d5f452c8350cce10afd3898 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } attrs { replace: true } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:38.764724 19239 raft_consensus.cc:2932] T 5d2104698d5f452c8350cce10afd3898 P 9564311af29a4493831a8db658a085c4 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } attrs { replace: true } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:38.765709 16812 catalog_manager.cc:5484] T 5d2104698d5f452c8350cce10afd3898 P 89dd09f8aaee4b249ae3ce924a422a67 reported cstate change: config changed from index 8 to 9, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } attrs { replace: true } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:38.768445 21531 raft_consensus.cc:1050] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0: attempting to promote NON_VOTER 838ef968bb45484bb27f5af6f3989bae to VOTER
I20240503 02:00:38.769070 21531 consensus_queue.cc:237] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } attrs { replace: true } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } }
I20240503 02:00:38.772174 17487 raft_consensus.cc:1261] T 4782de9826d6460e9e993e01327034e4 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Refusing update from remote peer 5b8291b26a7f448b8636187d1caf86d0: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:38.772452 17619 raft_consensus.cc:1261] T 4782de9826d6460e9e993e01327034e4 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer 5b8291b26a7f448b8636187d1caf86d0: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:38.772809 16958 raft_consensus.cc:1261] T 4782de9826d6460e9e993e01327034e4 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 5b8291b26a7f448b8636187d1caf86d0: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:38.772819 19369 raft_consensus.cc:1261] T 4782de9826d6460e9e993e01327034e4 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 FOLLOWER]: Refusing update from remote peer 5b8291b26a7f448b8636187d1caf86d0: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:38.772938 21202 consensus_queue.cc:1035] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:38.773208 21422 consensus_queue.cc:1035] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:38.773372 21202 consensus_queue.cc:1035] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:38.774374 19108 raft_consensus.cc:1261] T 4782de9826d6460e9e993e01327034e4 P 838ef968bb45484bb27f5af6f3989bae [term 1 LEARNER]: Refusing update from remote peer 5b8291b26a7f448b8636187d1caf86d0: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:38.774669 21202 consensus_queue.cc:1035] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:38.775678 21422 consensus_queue.cc:1035] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:38.776041 17750 raft_consensus.cc:1261] T 4782de9826d6460e9e993e01327034e4 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer 5b8291b26a7f448b8636187d1caf86d0: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:38.777065 21422 consensus_queue.cc:1035] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:38.778334 17750 consensus_queue.cc:237] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } }
I20240503 02:00:38.778949 17090 raft_consensus.cc:1261] T 4782de9826d6460e9e993e01327034e4 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer 5b8291b26a7f448b8636187d1caf86d0: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:38.780316 19239 raft_consensus.cc:1261] T 5d2104698d5f452c8350cce10afd3898 P 9564311af29a4493831a8db658a085c4 [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:00:38.780369 16958 raft_consensus.cc:1261] T 5d2104698d5f452c8350cce10afd3898 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:00:38.780443 17353 raft_consensus.cc:1261] T 5d2104698d5f452c8350cce10afd3898 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:00:38.780530 19369 raft_consensus.cc:1261] T 5d2104698d5f452c8350cce10afd3898 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:00:38.780536 21422 consensus_queue.cc:1035] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:38.780977 21484 consensus_queue.cc:1035] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:00:38.781040 17223 raft_consensus.cc:1261] T 5d2104698d5f452c8350cce10afd3898 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:00:38.781133 21606 consensus_queue.cc:1035] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:00:38.781150 21422 raft_consensus.cc:2932] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } attrs { replace: true } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:38.781410 21605 consensus_queue.cc:1035] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:00:38.783028 17620 raft_consensus.cc:1261] T 5d2104698d5f452c8350cce10afd3898 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:00:38.783973 21605 consensus_queue.cc:1035] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:00:38.784441 21484 consensus_queue.cc:1035] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:00:38.784835 16958 raft_consensus.cc:2932] T 4782de9826d6460e9e993e01327034e4 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } attrs { replace: true } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:38.785699 21144 consensus_queue.cc:1035] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:00:38.785775 17619 raft_consensus.cc:2932] T 4782de9826d6460e9e993e01327034e4 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } attrs { replace: true } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:38.786037 21144 raft_consensus.cc:2932] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 89dd09f8aaee4b249ae3ce924a422a67 (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:38.786684 19108 raft_consensus.cc:2932] T 4782de9826d6460e9e993e01327034e4 P 838ef968bb45484bb27f5af6f3989bae [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } attrs { replace: true } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:38.786939 17487 raft_consensus.cc:2932] T 4782de9826d6460e9e993e01327034e4 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } attrs { replace: true } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:38.788379 19369 raft_consensus.cc:2932] T 4782de9826d6460e9e993e01327034e4 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } attrs { replace: true } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:38.788995 16802 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5d2104698d5f452c8350cce10afd3898 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:00:38.789196 19370 raft_consensus.cc:2932] T 5d2104698d5f452c8350cce10afd3898 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 89dd09f8aaee4b249ae3ce924a422a67 (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:38.790987 17353 raft_consensus.cc:2932] T 5d2104698d5f452c8350cce10afd3898 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 89dd09f8aaee4b249ae3ce924a422a67 (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:38.791574 16818 catalog_manager.cc:5484] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 reported cstate change: config changed from index 8 to 9, 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5b8291b26a7f448b8636187d1caf86d0" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:00:38.792547 16811 catalog_manager.cc:5484] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6 reported cstate change: config changed from index 9 to 10, VOTER 89dd09f8aaee4b249ae3ce924a422a67 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:00:38.793733 17751 raft_consensus.cc:2932] T 4782de9826d6460e9e993e01327034e4 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } attrs { replace: true } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:38.795200 17223 raft_consensus.cc:2932] T 5d2104698d5f452c8350cce10afd3898 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 89dd09f8aaee4b249ae3ce924a422a67 (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:38.796612 17070 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d2104698d5f452c8350cce10afd3898 with delete_type TABLET_DATA_TOMBSTONED (TS 89dd09f8aaee4b249ae3ce924a422a67 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:56194
I20240503 02:00:38.797008 21185 raft_consensus.cc:1050] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67: attempting to promote NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d to VOTER
I20240503 02:00:38.797693 21185 consensus_queue.cc:237] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } }
I20240503 02:00:38.798967 17620 raft_consensus.cc:2932] T 5d2104698d5f452c8350cce10afd3898 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 89dd09f8aaee4b249ae3ce924a422a67 (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:38.799437 21631 tablet_replica.cc:330] T 5d2104698d5f452c8350cce10afd3898 P 89dd09f8aaee4b249ae3ce924a422a67: stopping tablet replica
I20240503 02:00:38.799829 21631 raft_consensus.cc:2229] T 5d2104698d5f452c8350cce10afd3898 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:38.800148 21631 raft_consensus.cc:2258] T 5d2104698d5f452c8350cce10afd3898 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:38.800712 19107 raft_consensus.cc:1261] T 5833b4f9a5434734817f154d660aa9f7 P 838ef968bb45484bb27f5af6f3989bae [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:38.800793 17222 raft_consensus.cc:1261] T 5833b4f9a5434734817f154d660aa9f7 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:38.801249 21636 consensus_queue.cc:1035] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:38.801373 19369 raft_consensus.cc:1261] T 5833b4f9a5434734817f154d660aa9f7 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 LEARNER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:38.801568 21631 ts_tablet_manager.cc:1875] T 5d2104698d5f452c8350cce10afd3898 P 89dd09f8aaee4b249ae3ce924a422a67: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:38.801622 17487 raft_consensus.cc:1261] T 5833b4f9a5434734817f154d660aa9f7 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:38.802014 16958 raft_consensus.cc:1261] T 5833b4f9a5434734817f154d660aa9f7 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:38.802060 20866 consensus_queue.cc:1035] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:38.802378 20866 consensus_queue.cc:1035] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:38.802742 20866 consensus_queue.cc:1035] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:38.803510 17751 raft_consensus.cc:1261] T 5833b4f9a5434734817f154d660aa9f7 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:38.804734 21631 ts_tablet_manager.cc:1888] T 5d2104698d5f452c8350cce10afd3898 P 89dd09f8aaee4b249ae3ce924a422a67: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 02:00:38.804884 21631 log.cc:1196] T 5d2104698d5f452c8350cce10afd3898 P 89dd09f8aaee4b249ae3ce924a422a67: Deleting WAL directory at /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/5d2104698d5f452c8350cce10afd3898
I20240503 02:00:38.789757 19239 raft_consensus.cc:2932] T 5d2104698d5f452c8350cce10afd3898 P 9564311af29a4493831a8db658a085c4 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 89dd09f8aaee4b249ae3ce924a422a67 (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:38.805033 17353 raft_consensus.cc:1261] T 5833b4f9a5434734817f154d660aa9f7 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:38.805495 20866 consensus_queue.cc:1035] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:38.805745 21185 consensus_queue.cc:1035] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:38.808560 21478 raft_consensus.cc:1050] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa: attempting to promote NON_VOTER 838ef968bb45484bb27f5af6f3989bae to VOTER
I20240503 02:00:38.809010 17224 consensus_queue.cc:237] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } }
I20240503 02:00:38.810612 21478 consensus_queue.cc:237] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } }
I20240503 02:00:38.811594 17486 raft_consensus.cc:1261] T 4782de9826d6460e9e993e01327034e4 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Refusing update from remote peer 5b8291b26a7f448b8636187d1caf86d0: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:00:38.811395 16959 raft_consensus.cc:2932] T 5d2104698d5f452c8350cce10afd3898 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 89dd09f8aaee4b249ae3ce924a422a67 (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:38.811652 21185 consensus_queue.cc:1035] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:38.812109 21422 consensus_queue.cc:1035] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:00:38.814630 19369 raft_consensus.cc:1261] T 4782de9826d6460e9e993e01327034e4 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 FOLLOWER]: Refusing update from remote peer 5b8291b26a7f448b8636187d1caf86d0: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:00:38.814625 17356 raft_consensus.cc:1261] T fbec07350ad04c49afd42e5d88dda6ad P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:38.815135 16959 raft_consensus.cc:1261] T 4782de9826d6460e9e993e01327034e4 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 5b8291b26a7f448b8636187d1caf86d0: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:00:38.813908 16805 catalog_manager.cc:4830] TS 89dd09f8aaee4b249ae3ce924a422a67 (127.0.99.130:38889): tablet 5d2104698d5f452c8350cce10afd3898 (table rebalance_test_table_2 [id=9f4cafd5cdb041c1bbe1592f04370c0b]) successfully deleted
I20240503 02:00:38.815124 21646 consensus_queue.cc:1035] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:38.815531 17222 raft_consensus.cc:1261] T fbec07350ad04c49afd42e5d88dda6ad P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:38.815536 21531 consensus_queue.cc:1035] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:00:38.815912 19239 raft_consensus.cc:1261] T fbec07350ad04c49afd42e5d88dda6ad P 9564311af29a4493831a8db658a085c4 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:38.816084 19107 raft_consensus.cc:1261] T fbec07350ad04c49afd42e5d88dda6ad P 838ef968bb45484bb27f5af6f3989bae [term 1 LEARNER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:38.816282 17091 raft_consensus.cc:1261] T fbec07350ad04c49afd42e5d88dda6ad P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:38.816365 21478 consensus_queue.cc:1035] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:38.816519 17620 raft_consensus.cc:1261] T 4782de9826d6460e9e993e01327034e4 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer 5b8291b26a7f448b8636187d1caf86d0: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:00:38.816607 21588 consensus_queue.cc:1035] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:38.817106 16957 raft_consensus.cc:1261] T fbec07350ad04c49afd42e5d88dda6ad P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:38.817142 19106 raft_consensus.cc:1261] T 4782de9826d6460e9e993e01327034e4 P 838ef968bb45484bb27f5af6f3989bae [term 1 FOLLOWER]: Refusing update from remote peer 5b8291b26a7f448b8636187d1caf86d0: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:00:38.817765 21531 consensus_queue.cc:1035] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:00:38.818496 21531 consensus_queue.cc:1035] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:00:38.818822 21531 consensus_queue.cc:1035] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:00:38.819082 21588 consensus_queue.cc:1035] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:38.819420 21588 consensus_queue.cc:1035] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:38.819568 17749 raft_consensus.cc:1261] T 4782de9826d6460e9e993e01327034e4 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer 5b8291b26a7f448b8636187d1caf86d0: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:00:38.819743 21588 consensus_queue.cc:1035] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:38.822461 21619 consensus_queue.cc:1035] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:00:38.823463 21619 raft_consensus.cc:2932] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 89dd09f8aaee4b249ae3ce924a422a67 (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:38.823832 17750 raft_consensus.cc:1261] T fbec07350ad04c49afd42e5d88dda6ad P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:38.824122 21636 raft_consensus.cc:2932] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:38.824390 21478 consensus_queue.cc:1035] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:38.824623 19370 raft_consensus.cc:2932] T 5833b4f9a5434734817f154d660aa9f7 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:38.824666 17353 raft_consensus.cc:2932] T 5833b4f9a5434734817f154d660aa9f7 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:38.826613 17487 raft_consensus.cc:2932] T 5833b4f9a5434734817f154d660aa9f7 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:38.828886 19106 raft_consensus.cc:2932] T 4782de9826d6460e9e993e01327034e4 P 838ef968bb45484bb27f5af6f3989bae [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 89dd09f8aaee4b249ae3ce924a422a67 (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:38.830237 16812 catalog_manager.cc:5484] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 reported cstate change: config changed from index 8 to 9, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "89dd09f8aaee4b249ae3ce924a422a67" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:00:38.832437 17749 raft_consensus.cc:2932] T 4782de9826d6460e9e993e01327034e4 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 89dd09f8aaee4b249ae3ce924a422a67 (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:38.832870 17224 raft_consensus.cc:2932] T 5833b4f9a5434734817f154d660aa9f7 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:38.833205 17620 raft_consensus.cc:2932] T 4782de9826d6460e9e993e01327034e4 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 89dd09f8aaee4b249ae3ce924a422a67 (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:38.835088 16811 catalog_manager.cc:5484] T 4782de9826d6460e9e993e01327034e4 P 838ef968bb45484bb27f5af6f3989bae reported cstate change: config changed from index 9 to 10, VOTER 89dd09f8aaee4b249ae3ce924a422a67 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "5b8291b26a7f448b8636187d1caf86d0" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:38.835064 19106 raft_consensus.cc:2932] T 5833b4f9a5434734817f154d660aa9f7 P 838ef968bb45484bb27f5af6f3989bae [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:38.836136 21484 raft_consensus.cc:1050] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6: attempting to promote NON_VOTER 838ef968bb45484bb27f5af6f3989bae to VOTER
I20240503 02:00:38.836778 21484 consensus_queue.cc:237] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } }
I20240503 02:00:38.838307 19370 raft_consensus.cc:2932] T 4782de9826d6460e9e993e01327034e4 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 89dd09f8aaee4b249ae3ce924a422a67 (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:38.839167 17487 raft_consensus.cc:1261] T 78d106d705df4089a4dfae4d5dc533cc P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:38.839186 16957 raft_consensus.cc:1261] T 78d106d705df4089a4dfae4d5dc533cc P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:38.839289 17092 raft_consensus.cc:1261] T 78d106d705df4089a4dfae4d5dc533cc P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:38.840101 21658 consensus_queue.cc:1035] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:38.841890 16803 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4782de9826d6460e9e993e01327034e4 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:00:38.842578 21478 raft_consensus.cc:1050] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa: attempting to promote NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d to VOTER
I20240503 02:00:38.842628 21658 consensus_queue.cc:1035] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:38.843066 21478 consensus_queue.cc:237] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } attrs { replace: true } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } }
I20240503 02:00:38.844962 17070 tablet_service.cc:1514] Processing DeleteTablet for tablet 4782de9826d6460e9e993e01327034e4 with delete_type TABLET_DATA_TOMBSTONED (TS 89dd09f8aaee4b249ae3ce924a422a67 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:56194
I20240503 02:00:38.845099 17092 raft_consensus.cc:1261] T ec69ad86be3d4793b10aca98dc4c1736 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:38.845439 21631 tablet_replica.cc:330] T 4782de9826d6460e9e993e01327034e4 P 89dd09f8aaee4b249ae3ce924a422a67: stopping tablet replica
I20240503 02:00:38.845396 21606 consensus_queue.cc:1035] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:38.845634 21631 raft_consensus.cc:2229] T 4782de9826d6460e9e993e01327034e4 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:38.845690 21586 consensus_queue.cc:1035] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:38.845913 21631 pending_rounds.cc:62] T 4782de9826d6460e9e993e01327034e4 P 89dd09f8aaee4b249ae3ce924a422a67: Trying to abort 1 pending ops.
I20240503 02:00:38.845999 21631 pending_rounds.cc:68] T 4782de9826d6460e9e993e01327034e4 P 89dd09f8aaee4b249ae3ce924a422a67: Aborting op as it isn't in flight: id { term: 1 index: 9 } timestamp: 7023417912396107776 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "4782de9826d6460e9e993e01327034e4" old_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } attrs { replace: true } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } } new_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } attrs { replace: true } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } } }
I20240503 02:00:38.846235 21631 raft_consensus.cc:2894] T 4782de9826d6460e9e993e01327034e4 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Aborting config change with OpId 1.9: Aborted: Op aborted
I20240503 02:00:38.846333 21631 raft_consensus.cc:2866] T 4782de9826d6460e9e993e01327034e4 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240503 02:00:38.846396 21631 raft_consensus.cc:2258] T 4782de9826d6460e9e993e01327034e4 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:38.846817 17486 raft_consensus.cc:2932] T 4782de9826d6460e9e993e01327034e4 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 89dd09f8aaee4b249ae3ce924a422a67 (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:38.847620 19106 raft_consensus.cc:1261] T ec69ad86be3d4793b10aca98dc4c1736 P 838ef968bb45484bb27f5af6f3989bae [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:38.847843 17224 raft_consensus.cc:1261] T ec69ad86be3d4793b10aca98dc4c1736 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:38.848019 21586 consensus_queue.cc:1035] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:38.848335 21130 consensus_queue.cc:1035] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:38.848758 17749 raft_consensus.cc:1261] T ec69ad86be3d4793b10aca98dc4c1736 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:38.849087 19238 raft_consensus.cc:1261] T 78d106d705df4089a4dfae4d5dc533cc P 9564311af29a4493831a8db658a085c4 [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:38.849402 16960 raft_consensus.cc:2932] T 4782de9826d6460e9e993e01327034e4 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 89dd09f8aaee4b249ae3ce924a422a67 (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:38.849658 21606 consensus_queue.cc:1035] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240503 02:00:38.849764 21644 consensus_queue.cc:1159] T 4782de9826d6460e9e993e01327034e4 P 89dd09f8aaee4b249ae3ce924a422a67 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "89dd09f8aaee4b249ae3ce924a422a67" status { last_received { term: 1 index: 9 } last_committed_idx: 8 last_received_current_leader { term: 1 index: 9 } }
I20240503 02:00:38.850746 21631 ts_tablet_manager.cc:1875] T 4782de9826d6460e9e993e01327034e4 P 89dd09f8aaee4b249ae3ce924a422a67: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:38.851888 17618 raft_consensus.cc:1261] T 78d106d705df4089a4dfae4d5dc533cc P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:38.852238 17619 raft_consensus.cc:1261] T ec69ad86be3d4793b10aca98dc4c1736 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:38.852417 17223 raft_consensus.cc:1261] T 78d106d705df4089a4dfae4d5dc533cc P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:38.853358 21631 ts_tablet_manager.cc:1888] T 4782de9826d6460e9e993e01327034e4 P 89dd09f8aaee4b249ae3ce924a422a67: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 02:00:38.853428 21631 log.cc:1196] T 4782de9826d6460e9e993e01327034e4 P 89dd09f8aaee4b249ae3ce924a422a67: Deleting WAL directory at /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/4782de9826d6460e9e993e01327034e4
I20240503 02:00:38.854023 16805 catalog_manager.cc:4830] TS 89dd09f8aaee4b249ae3ce924a422a67 (127.0.99.130:38889): tablet 4782de9826d6460e9e993e01327034e4 (table rebalance_test_table_6 [id=fc19f05fb43b48068855575b19b79556]) successfully deleted
I20240503 02:00:38.854801 21606 consensus_queue.cc:1035] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:38.856590 21606 consensus_queue.cc:1035] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:38.856666 19370 raft_consensus.cc:1261] T ec69ad86be3d4793b10aca98dc4c1736 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 LEARNER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:38.857275 21646 consensus_queue.cc:1035] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:38.862421 21586 consensus_queue.cc:1035] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:38.862893 21586 consensus_queue.cc:1035] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:38.863179 17751 raft_consensus.cc:2932] T 5833b4f9a5434734817f154d660aa9f7 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:38.863762 16960 raft_consensus.cc:1261] T ec69ad86be3d4793b10aca98dc4c1736 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:38.863735 21586 raft_consensus.cc:2932] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:38.866137 21130 consensus_queue.cc:1035] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:38.866333 17089 consensus_queue.cc:237] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } }
I20240503 02:00:38.868700 17353 raft_consensus.cc:2932] T fbec07350ad04c49afd42e5d88dda6ad P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:38.868952 17223 raft_consensus.cc:1261] T 5833b4f9a5434734817f154d660aa9f7 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:00:38.869580 21638 consensus_queue.cc:1035] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:00:38.871117 19369 raft_consensus.cc:1261] T 5833b4f9a5434734817f154d660aa9f7 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:00:38.871647 17751 raft_consensus.cc:1261] T 5833b4f9a5434734817f154d660aa9f7 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:00:38.871632 19105 raft_consensus.cc:1261] T 5833b4f9a5434734817f154d660aa9f7 P 838ef968bb45484bb27f5af6f3989bae [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:00:38.872103 21638 consensus_queue.cc:1035] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:00:38.874141 21185 consensus_queue.cc:1035] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:00:38.874293 17486 raft_consensus.cc:1261] T 5833b4f9a5434734817f154d660aa9f7 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:00:38.874285 16959 raft_consensus.cc:2932] T fbec07350ad04c49afd42e5d88dda6ad P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:38.874573 21185 consensus_queue.cc:1035] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:00:38.874873 19239 raft_consensus.cc:2932] T fbec07350ad04c49afd42e5d88dda6ad P 9564311af29a4493831a8db658a085c4 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:38.875046 16811 catalog_manager.cc:5484] T fbec07350ad04c49afd42e5d88dda6ad P d43ed6b8663947d99a72cd922cc323aa reported cstate change: config changed from index 8 to 9, 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "16d437fe629643948c95d628926d5ffa" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:38.877050 19107 raft_consensus.cc:2932] T fbec07350ad04c49afd42e5d88dda6ad P 838ef968bb45484bb27f5af6f3989bae [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:38.877524 20866 consensus_queue.cc:1035] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:00:38.878660 20866 raft_consensus.cc:2932] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 6398c8993c3c45dcaab25b42142d5671 (127.0.99.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:38.879235 17223 raft_consensus.cc:2932] T fbec07350ad04c49afd42e5d88dda6ad P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:38.881027 19106 raft_consensus.cc:1261] T 78d106d705df4089a4dfae4d5dc533cc P 838ef968bb45484bb27f5af6f3989bae [term 1 LEARNER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:38.881857 21658 consensus_queue.cc:1035] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:38.882892 16805 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5833b4f9a5434734817f154d660aa9f7 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:00:38.882943 19369 raft_consensus.cc:2932] T 5833b4f9a5434734817f154d660aa9f7 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 6398c8993c3c45dcaab25b42142d5671 (127.0.99.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:38.883164 17224 raft_consensus.cc:2932] T 5833b4f9a5434734817f154d660aa9f7 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 6398c8993c3c45dcaab25b42142d5671 (127.0.99.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:38.883348 21646 raft_consensus.cc:2932] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } attrs { replace: true } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:38.884833 16809 catalog_manager.cc:5484] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 reported cstate change: config changed from index 9 to 10, VOTER 6398c8993c3c45dcaab25b42142d5671 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "89dd09f8aaee4b249ae3ce924a422a67" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:00:38.885874 17749 raft_consensus.cc:2932] T 5833b4f9a5434734817f154d660aa9f7 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 6398c8993c3c45dcaab25b42142d5671 (127.0.99.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:38.886281 17089 raft_consensus.cc:2932] T ec69ad86be3d4793b10aca98dc4c1736 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } attrs { replace: true } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:38.887678 19107 raft_consensus.cc:2932] T 5833b4f9a5434734817f154d660aa9f7 P 838ef968bb45484bb27f5af6f3989bae [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 6398c8993c3c45dcaab25b42142d5671 (127.0.99.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:38.888499 16958 raft_consensus.cc:2932] T ec69ad86be3d4793b10aca98dc4c1736 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } attrs { replace: true } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:38.889681 17353 raft_consensus.cc:1261] T 5833b4f9a5434734817f154d660aa9f7 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:00:38.890226 21635 consensus_queue.cc:1035] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:00:38.891000 17749 raft_consensus.cc:2932] T ec69ad86be3d4793b10aca98dc4c1736 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } attrs { replace: true } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:38.891502 17620 raft_consensus.cc:2932] T ec69ad86be3d4793b10aca98dc4c1736 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } attrs { replace: true } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:38.892927 17224 raft_consensus.cc:2932] T ec69ad86be3d4793b10aca98dc4c1736 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } attrs { replace: true } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:38.893675 16938 tablet_service.cc:1514] Processing DeleteTablet for tablet 5833b4f9a5434734817f154d660aa9f7 with delete_type TABLET_DATA_TOMBSTONED (TS 6398c8993c3c45dcaab25b42142d5671 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:47986
I20240503 02:00:38.895606 16812 catalog_manager.cc:5484] T ec69ad86be3d4793b10aca98dc4c1736 P c843193b801b4f1a88f630ed3ed0f375 reported cstate change: config changed from index 8 to 9, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "16d437fe629643948c95d628926d5ffa" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } attrs { replace: true } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:38.896152 17353 raft_consensus.cc:2932] T 5833b4f9a5434734817f154d660aa9f7 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 6398c8993c3c45dcaab25b42142d5671 (127.0.99.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:38.897810 21484 raft_consensus.cc:2932] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:38.899008 21534 raft_consensus.cc:1050] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa: attempting to promote NON_VOTER 838ef968bb45484bb27f5af6f3989bae to VOTER
I20240503 02:00:38.899691 21534 consensus_queue.cc:237] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } }
I20240503 02:00:38.899807 17619 raft_consensus.cc:2932] T 78d106d705df4089a4dfae4d5dc533cc P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:38.899893 17222 raft_consensus.cc:2932] T 78d106d705df4089a4dfae4d5dc533cc P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:38.901738 17089 raft_consensus.cc:2932] T fbec07350ad04c49afd42e5d88dda6ad P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:38.902633 21533 raft_consensus.cc:1050] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa: attempting to promote NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d to VOTER
I20240503 02:00:38.903266 21533 consensus_queue.cc:237] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } }
I20240503 02:00:38.903187 19239 raft_consensus.cc:2932] T 78d106d705df4089a4dfae4d5dc533cc P 9564311af29a4493831a8db658a085c4 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:38.903496 17486 raft_consensus.cc:2932] T 5833b4f9a5434734817f154d660aa9f7 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 6398c8993c3c45dcaab25b42142d5671 (127.0.99.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:38.906219 19369 raft_consensus.cc:1261] T 63962b169caf4f14888871e246c00075 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 LEARNER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:38.906157 16958 raft_consensus.cc:2932] T 78d106d705df4089a4dfae4d5dc533cc P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:38.906963 21220 consensus_queue.cc:1035] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:38.908432 16812 catalog_manager.cc:5484] T 78d106d705df4089a4dfae4d5dc533cc P c843193b801b4f1a88f630ed3ed0f375 reported cstate change: config changed from index 8 to 9, 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:38.909101 17486 raft_consensus.cc:2932] T 78d106d705df4089a4dfae4d5dc533cc P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:38.909531 19239 raft_consensus.cc:1261] T 63962b169caf4f14888871e246c00075 P 9564311af29a4493831a8db658a085c4 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:38.909505 19108 raft_consensus.cc:2932] T ec69ad86be3d4793b10aca98dc4c1736 P 838ef968bb45484bb27f5af6f3989bae [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } attrs { replace: true } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:38.910027 21220 consensus_queue.cc:1035] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:38.911705 17222 raft_consensus.cc:1261] T 63962b169caf4f14888871e246c00075 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:38.913478 17749 raft_consensus.cc:1261] T ab536115ac7a4ba193f718d421f4000d P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:38.913774 17619 raft_consensus.cc:1261] T ab536115ac7a4ba193f718d421f4000d P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:38.913945 19238 raft_consensus.cc:1261] T ab536115ac7a4ba193f718d421f4000d P 9564311af29a4493831a8db658a085c4 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:38.914988 16958 raft_consensus.cc:1261] T ab536115ac7a4ba193f718d421f4000d P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:38.915360 17486 consensus_queue.cc:237] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } }
I20240503 02:00:38.915710 21534 consensus_queue.cc:1035] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:38.916126 21220 consensus_queue.cc:1035] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:38.916512 21220 consensus_queue.cc:1035] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:38.916947 21220 consensus_queue.cc:1035] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:38.917685 19238 raft_consensus.cc:1261] T fbec07350ad04c49afd42e5d88dda6ad P 9564311af29a4493831a8db658a085c4 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:00:38.917929 19108 raft_consensus.cc:1261] T fbec07350ad04c49afd42e5d88dda6ad P 838ef968bb45484bb27f5af6f3989bae [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:00:38.918813 17751 raft_consensus.cc:1261] T fbec07350ad04c49afd42e5d88dda6ad P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:00:38.918850 21587 consensus_queue.cc:1035] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:00:38.919476 21130 consensus_queue.cc:1035] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:38.920078 17089 raft_consensus.cc:1261] T ab536115ac7a4ba193f718d421f4000d P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:38.920258 17751 raft_consensus.cc:2932] T fbec07350ad04c49afd42e5d88dda6ad P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:38.922537 19108 raft_consensus.cc:1261] T ab536115ac7a4ba193f718d421f4000d P 838ef968bb45484bb27f5af6f3989bae [term 1 LEARNER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:38.923007 21220 consensus_queue.cc:1035] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:38.923069 17356 raft_consensus.cc:1261] T fbec07350ad04c49afd42e5d88dda6ad P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:00:38.923380 21220 consensus_queue.cc:1035] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:38.923578 21587 consensus_queue.cc:1035] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:00:38.924003 21220 consensus_queue.cc:1035] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:38.924726 21587 consensus_queue.cc:1035] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:00:38.926625 17091 raft_consensus.cc:1261] T fbec07350ad04c49afd42e5d88dda6ad P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:00:38.927021 21587 consensus_queue.cc:1035] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:00:38.927098 16959 raft_consensus.cc:1261] T fbec07350ad04c49afd42e5d88dda6ad P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:00:38.927309 21532 raft_consensus.cc:1050] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375: attempting to promote NON_VOTER 838ef968bb45484bb27f5af6f3989bae to VOTER
I20240503 02:00:38.928157 16959 raft_consensus.cc:1261] T 63962b169caf4f14888871e246c00075 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:38.928069 21532 consensus_queue.cc:237] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } }
I20240503 02:00:38.928285 21130 consensus_queue.cc:1035] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:00:38.928879 21533 consensus_queue.cc:1035] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:38.929117 17091 raft_consensus.cc:2932] T fbec07350ad04c49afd42e5d88dda6ad P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 5b8291b26a7f448b8636187d1caf86d0 (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:38.930429 17223 raft_consensus.cc:1261] T ab536115ac7a4ba193f718d421f4000d P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:38.930574 16959 raft_consensus.cc:2932] T fbec07350ad04c49afd42e5d88dda6ad P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 5b8291b26a7f448b8636187d1caf86d0 (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:38.931013 21533 consensus_queue.cc:1035] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:38.931978 17750 raft_consensus.cc:1261] T 63962b169caf4f14888871e246c00075 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:38.932955 19108 raft_consensus.cc:2932] T fbec07350ad04c49afd42e5d88dda6ad P 838ef968bb45484bb27f5af6f3989bae [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 5b8291b26a7f448b8636187d1caf86d0 (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:38.933310 17091 raft_consensus.cc:1261] T d81dcfefb4fd401b9cfd2ca36034a010 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer c843193b801b4f1a88f630ed3ed0f375: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:38.933764 17487 raft_consensus.cc:1261] T d81dcfefb4fd401b9cfd2ca36034a010 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Refusing update from remote peer c843193b801b4f1a88f630ed3ed0f375: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:38.933753 21238 consensus_queue.cc:1035] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:38.934486 21238 consensus_queue.cc:1035] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:38.934760 16957 raft_consensus.cc:1261] T d81dcfefb4fd401b9cfd2ca36034a010 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer c843193b801b4f1a88f630ed3ed0f375: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:38.935112 17486 raft_consensus.cc:1261] T 63962b169caf4f14888871e246c00075 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:38.934984 17356 raft_consensus.cc:2932] T fbec07350ad04c49afd42e5d88dda6ad P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 5b8291b26a7f448b8636187d1caf86d0 (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:38.935463 19368 raft_consensus.cc:1261] T d81dcfefb4fd401b9cfd2ca36034a010 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 FOLLOWER]: Refusing update from remote peer c843193b801b4f1a88f630ed3ed0f375: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:38.937126 21532 consensus_queue.cc:1035] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 [LEADER]: Connected to new peer: Peer: permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:38.937268 19108 raft_consensus.cc:1261] T d81dcfefb4fd401b9cfd2ca36034a010 P 838ef968bb45484bb27f5af6f3989bae [term 1 LEARNER]: Refusing update from remote peer c843193b801b4f1a88f630ed3ed0f375: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:38.937461 21532 consensus_queue.cc:1035] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:38.937793 21532 consensus_queue.cc:1035] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 [LEADER]: Connected to new peer: Peer: permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:38.938097 17090 raft_consensus.cc:1261] T 63962b169caf4f14888871e246c00075 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:38.938786 21220 consensus_queue.cc:1035] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:38.939237 21220 consensus_queue.cc:1035] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:38.939743 21220 consensus_queue.cc:1035] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:38.940963 21586 raft_consensus.cc:2932] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 5b8291b26a7f448b8636187d1caf86d0 (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:38.942221 17223 raft_consensus.cc:1261] T d81dcfefb4fd401b9cfd2ca36034a010 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer c843193b801b4f1a88f630ed3ed0f375: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:38.942737 21238 consensus_queue.cc:1035] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:38.943856 16802 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet fbec07350ad04c49afd42e5d88dda6ad with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:00:38.944140 16817 catalog_manager.cc:5484] T fbec07350ad04c49afd42e5d88dda6ad P 6398c8993c3c45dcaab25b42142d5671 reported cstate change: config changed from index 9 to 10, VOTER 5b8291b26a7f448b8636187d1caf86d0 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "16d437fe629643948c95d628926d5ffa" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:38.945150 17752 raft_consensus.cc:2932] T fbec07350ad04c49afd42e5d88dda6ad P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 5b8291b26a7f448b8636187d1caf86d0 (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:38.945153 19237 raft_consensus.cc:2932] T fbec07350ad04c49afd42e5d88dda6ad P 9564311af29a4493831a8db658a085c4 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 5b8291b26a7f448b8636187d1caf86d0 (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:38.948218 19370 raft_consensus.cc:2932] T ec69ad86be3d4793b10aca98dc4c1736 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } attrs { replace: true } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:38.948163 17092 raft_consensus.cc:2932] T 78d106d705df4089a4dfae4d5dc533cc P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:38.951201 19238 raft_consensus.cc:2932] T ab536115ac7a4ba193f718d421f4000d P 9564311af29a4493831a8db658a085c4 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:38.953572 17750 raft_consensus.cc:1261] T d81dcfefb4fd401b9cfd2ca36034a010 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer c843193b801b4f1a88f630ed3ed0f375: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:38.953727 19106 raft_consensus.cc:2932] T 78d106d705df4089a4dfae4d5dc533cc P 838ef968bb45484bb27f5af6f3989bae [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:38.953691 21534 raft_consensus.cc:2932] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:38.955303 21679 tablet_replica.cc:330] T 5833b4f9a5434734817f154d660aa9f7 P 6398c8993c3c45dcaab25b42142d5671: stopping tablet replica
I20240503 02:00:38.955662 21679 raft_consensus.cc:2229] T 5833b4f9a5434734817f154d660aa9f7 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:38.955924 21679 pending_rounds.cc:62] T 5833b4f9a5434734817f154d660aa9f7 P 6398c8993c3c45dcaab25b42142d5671: Trying to abort 1 pending ops.
I20240503 02:00:38.955991 21679 pending_rounds.cc:68] T 5833b4f9a5434734817f154d660aa9f7 P 6398c8993c3c45dcaab25b42142d5671: Aborting op as it isn't in flight: id { term: 1 index: 9 } timestamp: 7023417912513425408 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "5833b4f9a5434734817f154d660aa9f7" old_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } } new_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } } }
I20240503 02:00:38.956277 21679 raft_consensus.cc:2894] T 5833b4f9a5434734817f154d660aa9f7 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Aborting config change with OpId 1.9: Aborted: Op aborted
I20240503 02:00:38.956394 21679 raft_consensus.cc:2866] T 5833b4f9a5434734817f154d660aa9f7 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240503 02:00:38.956482 21679 raft_consensus.cc:2258] T 5833b4f9a5434734817f154d660aa9f7 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:38.956815 19369 raft_consensus.cc:2932] T 63962b169caf4f14888871e246c00075 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:38.957214 16958 raft_consensus.cc:2932] T ab536115ac7a4ba193f718d421f4000d P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:38.957898 21679 ts_tablet_manager.cc:1875] T 5833b4f9a5434734817f154d660aa9f7 P 6398c8993c3c45dcaab25b42142d5671: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:38.958967 17202 tablet_service.cc:1514] Processing DeleteTablet for tablet fbec07350ad04c49afd42e5d88dda6ad with delete_type TABLET_DATA_TOMBSTONED (TS 5b8291b26a7f448b8636187d1caf86d0 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:35990
I20240503 02:00:38.959475 21602 tablet_replica.cc:330] T fbec07350ad04c49afd42e5d88dda6ad P 5b8291b26a7f448b8636187d1caf86d0: stopping tablet replica
I20240503 02:00:38.959290 17092 raft_consensus.cc:2932] T ab536115ac7a4ba193f718d421f4000d P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:38.959698 21602 raft_consensus.cc:2229] T fbec07350ad04c49afd42e5d88dda6ad P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:38.956368 21220 raft_consensus.cc:2932] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:38.960040 21602 raft_consensus.cc:2258] T fbec07350ad04c49afd42e5d88dda6ad P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:38.961076 21602 ts_tablet_manager.cc:1875] T fbec07350ad04c49afd42e5d88dda6ad P 5b8291b26a7f448b8636187d1caf86d0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:38.962272 21700 consensus_queue.cc:1035] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:38.962623 16958 raft_consensus.cc:2932] T 63962b169caf4f14888871e246c00075 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:38.963603 21602 ts_tablet_manager.cc:1888] T fbec07350ad04c49afd42e5d88dda6ad P 5b8291b26a7f448b8636187d1caf86d0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 02:00:38.963343 19238 raft_consensus.cc:2932] T 63962b169caf4f14888871e246c00075 P 9564311af29a4493831a8db658a085c4 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:38.963676 21602 log.cc:1196] T fbec07350ad04c49afd42e5d88dda6ad P 5b8291b26a7f448b8636187d1caf86d0: Deleting WAL directory at /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/fbec07350ad04c49afd42e5d88dda6ad
I20240503 02:00:38.964824 17222 raft_consensus.cc:2932] T 63962b169caf4f14888871e246c00075 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:38.965598 21679 ts_tablet_manager.cc:1888] T 5833b4f9a5434734817f154d660aa9f7 P 6398c8993c3c45dcaab25b42142d5671: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 02:00:38.965672 21679 log.cc:1196] T 5833b4f9a5434734817f154d660aa9f7 P 6398c8993c3c45dcaab25b42142d5671: Deleting WAL directory at /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/5833b4f9a5434734817f154d660aa9f7
I20240503 02:00:38.967514 16803 catalog_manager.cc:4830] TS 5b8291b26a7f448b8636187d1caf86d0 (127.0.99.131:44479): tablet fbec07350ad04c49afd42e5d88dda6ad (table rebalance_test_table_0 [id=ee1e0b20af0b4ecea04ae147701a0482]) successfully deleted
I20240503 02:00:38.967531 19106 raft_consensus.cc:2932] T ab536115ac7a4ba193f718d421f4000d P 838ef968bb45484bb27f5af6f3989bae [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:38.969501 17619 raft_consensus.cc:2932] T ab536115ac7a4ba193f718d421f4000d P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:38.972759 17749 raft_consensus.cc:2932] T ab536115ac7a4ba193f718d421f4000d P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:38.972862 17750 consensus_queue.cc:237] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } }
I20240503 02:00:38.973140 16804 catalog_manager.cc:4830] TS 6398c8993c3c45dcaab25b42142d5671 (127.0.99.129:36983): tablet 5833b4f9a5434734817f154d660aa9f7 (table rebalance_test_table_1 [id=38eeb06923e14f2ead405ffb039fd231]) successfully deleted
I20240503 02:00:38.976262 17091 raft_consensus.cc:1261] T 78d106d705df4089a4dfae4d5dc533cc P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:00:38.976756 21144 consensus_queue.cc:1035] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:00:38.977527 16809 catalog_manager.cc:5484] T 63962b169caf4f14888871e246c00075 P 5b8291b26a7f448b8636187d1caf86d0 reported cstate change: config changed from index 8 to 9, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d43ed6b8663947d99a72cd922cc323aa" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:38.977710 19106 raft_consensus.cc:1261] T 78d106d705df4089a4dfae4d5dc533cc P 838ef968bb45484bb27f5af6f3989bae [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:00:38.978358 21658 consensus_queue.cc:1035] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:00:38.979532 16811 catalog_manager.cc:5484] T ab536115ac7a4ba193f718d421f4000d P 838ef968bb45484bb27f5af6f3989bae reported cstate change: config changed from index 8 to 9, 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d43ed6b8663947d99a72cd922cc323aa" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:38.981709 17619 raft_consensus.cc:1261] T 78d106d705df4089a4dfae4d5dc533cc P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:00:38.981725 17488 raft_consensus.cc:1261] T 78d106d705df4089a4dfae4d5dc533cc P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:00:38.982118 21700 raft_consensus.cc:2932] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:38.982702 17222 raft_consensus.cc:1261] T 78d106d705df4089a4dfae4d5dc533cc P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:00:38.983893 19238 raft_consensus.cc:1261] T 78d106d705df4089a4dfae4d5dc533cc P 9564311af29a4493831a8db658a085c4 [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:00:38.983842 17222 raft_consensus.cc:2932] T d81dcfefb4fd401b9cfd2ca36034a010 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:38.986075 21144 consensus_queue.cc:1035] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:00:38.985932 17752 raft_consensus.cc:2932] T 63962b169caf4f14888871e246c00075 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:38.986464 21144 consensus_queue.cc:1035] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:00:38.986521 17488 consensus_queue.cc:237] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } }
I20240503 02:00:38.988358 21484 consensus_queue.cc:1035] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:00:38.989017 17619 raft_consensus.cc:1261] T ec69ad86be3d4793b10aca98dc4c1736 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:00:38.989153 16958 raft_consensus.cc:1261] T ec69ad86be3d4793b10aca98dc4c1736 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:00:38.989553 21588 consensus_queue.cc:1035] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:00:38.989660 19368 raft_consensus.cc:2932] T d81dcfefb4fd401b9cfd2ca36034a010 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:38.989759 21658 raft_consensus.cc:2932] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 6398c8993c3c45dcaab25b42142d5671 (127.0.99.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:38.990584 21588 consensus_queue.cc:1035] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:00:38.992882 19369 raft_consensus.cc:1261] T ec69ad86be3d4793b10aca98dc4c1736 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:00:38.992663 17091 raft_consensus.cc:2932] T 78d106d705df4089a4dfae4d5dc533cc P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 6398c8993c3c45dcaab25b42142d5671 (127.0.99.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:38.992848 17485 raft_consensus.cc:2932] T 78d106d705df4089a4dfae4d5dc533cc P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 6398c8993c3c45dcaab25b42142d5671 (127.0.99.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:38.993124 16802 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 78d106d705df4089a4dfae4d5dc533cc with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:00:38.994366 21589 consensus_queue.cc:1035] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:00:38.994472 17092 raft_consensus.cc:2932] T 63962b169caf4f14888871e246c00075 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:38.994850 17620 raft_consensus.cc:2932] T 78d106d705df4089a4dfae4d5dc533cc P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 6398c8993c3c45dcaab25b42142d5671 (127.0.99.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:38.996420 21589 raft_consensus.cc:2932] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 89dd09f8aaee4b249ae3ce924a422a67 (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:38.998960 16802 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ec69ad86be3d4793b10aca98dc4c1736 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:00:38.999480 21144 consensus_queue.cc:1035] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:00:38.999570 17220 raft_consensus.cc:1261] T ec69ad86be3d4793b10aca98dc4c1736 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:00:38.999666 16958 raft_consensus.cc:2932] T ec69ad86be3d4793b10aca98dc4c1736 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 89dd09f8aaee4b249ae3ce924a422a67 (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:38.999861 17620 raft_consensus.cc:2932] T ec69ad86be3d4793b10aca98dc4c1736 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 89dd09f8aaee4b249ae3ce924a422a67 (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:39.000049 21586 consensus_queue.cc:1035] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:00:39.001870 17752 raft_consensus.cc:1261] T ec69ad86be3d4793b10aca98dc4c1736 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:00:39.002187 17220 raft_consensus.cc:2932] T ec69ad86be3d4793b10aca98dc4c1736 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 89dd09f8aaee4b249ae3ce924a422a67 (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:39.002465 17751 raft_consensus.cc:2932] T d81dcfefb4fd401b9cfd2ca36034a010 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:39.002807 21588 consensus_queue.cc:1035] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:00:39.004087 17752 raft_consensus.cc:2932] T ec69ad86be3d4793b10aca98dc4c1736 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 89dd09f8aaee4b249ae3ce924a422a67 (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:39.005185 17089 raft_consensus.cc:2932] T d81dcfefb4fd401b9cfd2ca36034a010 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:39.006825 16818 catalog_manager.cc:5484] T d81dcfefb4fd401b9cfd2ca36034a010 P 5b8291b26a7f448b8636187d1caf86d0 reported cstate change: config changed from index 8 to 9, 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c843193b801b4f1a88f630ed3ed0f375" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:39.009209 17487 raft_consensus.cc:2932] T 63962b169caf4f14888871e246c00075 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:39.012001 16817 catalog_manager.cc:5484] T ec69ad86be3d4793b10aca98dc4c1736 P 6398c8993c3c45dcaab25b42142d5671 reported cstate change: config changed from index 9 to 10, VOTER 89dd09f8aaee4b249ae3ce924a422a67 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "16d437fe629643948c95d628926d5ffa" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:39.013073 16958 raft_consensus.cc:2932] T d81dcfefb4fd401b9cfd2ca36034a010 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:39.014050 19368 raft_consensus.cc:2932] T ec69ad86be3d4793b10aca98dc4c1736 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 89dd09f8aaee4b249ae3ce924a422a67 (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:39.014608 19106 raft_consensus.cc:2932] T 78d106d705df4089a4dfae4d5dc533cc P 838ef968bb45484bb27f5af6f3989bae [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 6398c8993c3c45dcaab25b42142d5671 (127.0.99.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:39.016076 17353 consensus_queue.cc:237] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } }
I20240503 02:00:39.019130 17620 raft_consensus.cc:1261] T ab536115ac7a4ba193f718d421f4000d P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:00:39.019623 19237 raft_consensus.cc:1261] T ab536115ac7a4ba193f718d421f4000d P 9564311af29a4493831a8db658a085c4 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:00:39.019428 19238 raft_consensus.cc:2932] T 78d106d705df4089a4dfae4d5dc533cc P 9564311af29a4493831a8db658a085c4 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 6398c8993c3c45dcaab25b42142d5671 (127.0.99.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:39.019649 21681 consensus_queue.cc:1035] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:00:39.019874 16958 raft_consensus.cc:1261] T ab536115ac7a4ba193f718d421f4000d P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:00:39.020068 21696 consensus_queue.cc:1035] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:00:39.020236 21687 consensus_queue.cc:1035] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:00:39.021972 17751 raft_consensus.cc:1261] T ab536115ac7a4ba193f718d421f4000d P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:00:39.022179 19108 raft_consensus.cc:1261] T ec69ad86be3d4793b10aca98dc4c1736 P 838ef968bb45484bb27f5af6f3989bae [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:00:39.022434 16812 catalog_manager.cc:5484] T 78d106d705df4089a4dfae4d5dc533cc P c843193b801b4f1a88f630ed3ed0f375 reported cstate change: config changed from index 9 to 10, VOTER 6398c8993c3c45dcaab25b42142d5671 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:39.022840 21588 consensus_queue.cc:1035] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:00:39.024076 19105 raft_consensus.cc:2932] T ec69ad86be3d4793b10aca98dc4c1736 P 838ef968bb45484bb27f5af6f3989bae [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 89dd09f8aaee4b249ae3ce924a422a67 (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:39.024375 21681 consensus_queue.cc:1035] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:00:39.025521 19108 raft_consensus.cc:1261] T ab536115ac7a4ba193f718d421f4000d P 838ef968bb45484bb27f5af6f3989bae [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:00:39.025455 21681 raft_consensus.cc:2932] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 5b8291b26a7f448b8636187d1caf86d0 (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:39.025970 17089 raft_consensus.cc:1261] T ab536115ac7a4ba193f718d421f4000d P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:00:39.028612 21685 consensus_queue.cc:1035] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:00:39.028942 21685 consensus_queue.cc:1035] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:00:39.029412 16803 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ab536115ac7a4ba193f718d421f4000d with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:00:39.029776 17070 tablet_service.cc:1514] Processing DeleteTablet for tablet ec69ad86be3d4793b10aca98dc4c1736 with delete_type TABLET_DATA_TOMBSTONED (TS 89dd09f8aaee4b249ae3ce924a422a67 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:56194
I20240503 02:00:39.030045 21631 tablet_replica.cc:330] T ec69ad86be3d4793b10aca98dc4c1736 P 89dd09f8aaee4b249ae3ce924a422a67: stopping tablet replica
I20240503 02:00:39.030205 21631 raft_consensus.cc:2229] T ec69ad86be3d4793b10aca98dc4c1736 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:39.030037 19237 raft_consensus.cc:2932] T ab536115ac7a4ba193f718d421f4000d P 9564311af29a4493831a8db658a085c4 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 5b8291b26a7f448b8636187d1caf86d0 (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:39.030516 21631 raft_consensus.cc:2258] T ec69ad86be3d4793b10aca98dc4c1736 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:39.030742 16818 catalog_manager.cc:5484] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa reported cstate change: config changed from index 9 to 10, VOTER 5b8291b26a7f448b8636187d1caf86d0 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "d43ed6b8663947d99a72cd922cc323aa" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:39.031383 21631 ts_tablet_manager.cc:1875] T ec69ad86be3d4793b10aca98dc4c1736 P 89dd09f8aaee4b249ae3ce924a422a67: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:39.032444 19107 raft_consensus.cc:2932] T ab536115ac7a4ba193f718d421f4000d P 838ef968bb45484bb27f5af6f3989bae [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 5b8291b26a7f448b8636187d1caf86d0 (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:39.032956 17620 raft_consensus.cc:2932] T ab536115ac7a4ba193f718d421f4000d P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 5b8291b26a7f448b8636187d1caf86d0 (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:39.033094 19108 raft_consensus.cc:2932] T d81dcfefb4fd401b9cfd2ca36034a010 P 838ef968bb45484bb27f5af6f3989bae [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:39.033975 21631 ts_tablet_manager.cc:1888] T ec69ad86be3d4793b10aca98dc4c1736 P 89dd09f8aaee4b249ae3ce924a422a67: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 02:00:39.034045 21631 log.cc:1196] T ec69ad86be3d4793b10aca98dc4c1736 P 89dd09f8aaee4b249ae3ce924a422a67: Deleting WAL directory at /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/ec69ad86be3d4793b10aca98dc4c1736
I20240503 02:00:39.034344 17089 raft_consensus.cc:2932] T ab536115ac7a4ba193f718d421f4000d P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 5b8291b26a7f448b8636187d1caf86d0 (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:39.035333 16805 catalog_manager.cc:4830] TS 89dd09f8aaee4b249ae3ce924a422a67 (127.0.99.130:38889): tablet ec69ad86be3d4793b10aca98dc4c1736 (table rebalance_test_table_5 [id=90cb01160ab843bbae459ab57f0425ed]) successfully deleted
I20240503 02:00:39.036123 16958 raft_consensus.cc:2932] T ab536115ac7a4ba193f718d421f4000d P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 5b8291b26a7f448b8636187d1caf86d0 (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:39.022361 17486 raft_consensus.cc:2932] T d81dcfefb4fd401b9cfd2ca36034a010 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:39.038182 17353 consensus_queue.cc:237] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } }
I20240503 02:00:39.040740 16938 tablet_service.cc:1514] Processing DeleteTablet for tablet 78d106d705df4089a4dfae4d5dc533cc with delete_type TABLET_DATA_TOMBSTONED (TS 6398c8993c3c45dcaab25b42142d5671 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:47986
I20240503 02:00:39.041042 21679 tablet_replica.cc:330] T 78d106d705df4089a4dfae4d5dc533cc P 6398c8993c3c45dcaab25b42142d5671: stopping tablet replica
I20240503 02:00:39.041219 21679 raft_consensus.cc:2229] T 78d106d705df4089a4dfae4d5dc533cc P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:39.039927 17222 raft_consensus.cc:2932] T 78d106d705df4089a4dfae4d5dc533cc P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 6398c8993c3c45dcaab25b42142d5671 (127.0.99.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:39.042546 21679 raft_consensus.cc:2258] T 78d106d705df4089a4dfae4d5dc533cc P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:39.044114 21679 ts_tablet_manager.cc:1875] T 78d106d705df4089a4dfae4d5dc533cc P 6398c8993c3c45dcaab25b42142d5671: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:39.046480 21679 ts_tablet_manager.cc:1888] T 78d106d705df4089a4dfae4d5dc533cc P 6398c8993c3c45dcaab25b42142d5671: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 02:00:39.046619 21679 log.cc:1196] T 78d106d705df4089a4dfae4d5dc533cc P 6398c8993c3c45dcaab25b42142d5671: Deleting WAL directory at /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/78d106d705df4089a4dfae4d5dc533cc
I20240503 02:00:39.047256 16804 catalog_manager.cc:4830] TS 6398c8993c3c45dcaab25b42142d5671 (127.0.99.129:36983): tablet 78d106d705df4089a4dfae4d5dc533cc (table rebalance_test_table_5 [id=90cb01160ab843bbae459ab57f0425ed]) successfully deleted
I20240503 02:00:39.049943 17202 tablet_service.cc:1514] Processing DeleteTablet for tablet ab536115ac7a4ba193f718d421f4000d with delete_type TABLET_DATA_TOMBSTONED (TS 5b8291b26a7f448b8636187d1caf86d0 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:35990
I20240503 02:00:39.050212 21602 tablet_replica.cc:330] T ab536115ac7a4ba193f718d421f4000d P 5b8291b26a7f448b8636187d1caf86d0: stopping tablet replica
I20240503 02:00:39.050360 21602 raft_consensus.cc:2229] T ab536115ac7a4ba193f718d421f4000d P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:39.050591 21602 pending_rounds.cc:62] T ab536115ac7a4ba193f718d421f4000d P 5b8291b26a7f448b8636187d1caf86d0: Trying to abort 1 pending ops.
I20240503 02:00:39.050668 21602 pending_rounds.cc:68] T ab536115ac7a4ba193f718d421f4000d P 5b8291b26a7f448b8636187d1caf86d0: Aborting op as it isn't in flight: id { term: 1 index: 9 } timestamp: 7023417912930922496 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "ab536115ac7a4ba193f718d421f4000d" old_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } } new_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } } }
I20240503 02:00:39.050889 21602 raft_consensus.cc:2894] T ab536115ac7a4ba193f718d421f4000d P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Aborting config change with OpId 1.9: Aborted: Op aborted
I20240503 02:00:39.050959 21602 raft_consensus.cc:2866] T ab536115ac7a4ba193f718d421f4000d P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240503 02:00:39.051007 21602 raft_consensus.cc:2258] T ab536115ac7a4ba193f718d421f4000d P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:39.051594 16958 raft_consensus.cc:1261] T 63962b169caf4f14888871e246c00075 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:00:39.051759 21602 ts_tablet_manager.cc:1875] T ab536115ac7a4ba193f718d421f4000d P 5b8291b26a7f448b8636187d1caf86d0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:39.051826 19237 raft_consensus.cc:1261] T 63962b169caf4f14888871e246c00075 P 9564311af29a4493831a8db658a085c4 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:00:39.052047 21681 consensus_queue.cc:1035] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:00:39.052270 21687 consensus_queue.cc:1035] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:00:39.053133 19368 raft_consensus.cc:1261] T 63962b169caf4f14888871e246c00075 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:00:39.053575 21681 consensus_queue.cc:1035] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:00:39.054245 21602 ts_tablet_manager.cc:1888] T ab536115ac7a4ba193f718d421f4000d P 5b8291b26a7f448b8636187d1caf86d0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 02:00:39.054322 21602 log.cc:1196] T ab536115ac7a4ba193f718d421f4000d P 5b8291b26a7f448b8636187d1caf86d0: Deleting WAL directory at /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/ab536115ac7a4ba193f718d421f4000d
I20240503 02:00:39.054894 16803 catalog_manager.cc:4830] TS 5b8291b26a7f448b8636187d1caf86d0 (127.0.99.131:44479): tablet ab536115ac7a4ba193f718d421f4000d (table rebalance_test_table_6 [id=fc19f05fb43b48068855575b19b79556]) successfully deleted
I20240503 02:00:39.055238 17751 raft_consensus.cc:2932] T ab536115ac7a4ba193f718d421f4000d P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 5b8291b26a7f448b8636187d1caf86d0 (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:39.057822 17620 consensus_queue.cc:237] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } }
I20240503 02:00:39.058698 21534 raft_consensus.cc:2932] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 5b8291b26a7f448b8636187d1caf86d0 (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:39.060343 19108 raft_consensus.cc:1261] T d81dcfefb4fd401b9cfd2ca36034a010 P 838ef968bb45484bb27f5af6f3989bae [term 1 FOLLOWER]: Refusing update from remote peer c843193b801b4f1a88f630ed3ed0f375: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:00:39.060755 16803 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 63962b169caf4f14888871e246c00075 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:00:39.060802 17486 raft_consensus.cc:1261] T 63962b169caf4f14888871e246c00075 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:00:39.060807 21700 consensus_queue.cc:1035] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 [LEADER]: Connected to new peer: Peer: permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:00:39.061198 21533 consensus_queue.cc:1035] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:00:39.062382 17486 raft_consensus.cc:2932] T 63962b169caf4f14888871e246c00075 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 5b8291b26a7f448b8636187d1caf86d0 (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:39.062450 19368 raft_consensus.cc:2932] T 63962b169caf4f14888871e246c00075 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 5b8291b26a7f448b8636187d1caf86d0 (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:39.063673 17752 raft_consensus.cc:1261] T d81dcfefb4fd401b9cfd2ca36034a010 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer c843193b801b4f1a88f630ed3ed0f375: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:00:39.064106 21700 consensus_queue.cc:1035] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:00:39.064895 19369 raft_consensus.cc:1261] T d81dcfefb4fd401b9cfd2ca36034a010 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 FOLLOWER]: Refusing update from remote peer c843193b801b4f1a88f630ed3ed0f375: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:00:39.065371 21700 consensus_queue.cc:1035] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 [LEADER]: Connected to new peer: Peer: permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:00:39.066844 17486 raft_consensus.cc:1261] T d81dcfefb4fd401b9cfd2ca36034a010 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Refusing update from remote peer c843193b801b4f1a88f630ed3ed0f375: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:00:39.066869 19237 raft_consensus.cc:2932] T 63962b169caf4f14888871e246c00075 P 9564311af29a4493831a8db658a085c4 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 5b8291b26a7f448b8636187d1caf86d0 (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:39.067222 17092 raft_consensus.cc:1261] T d81dcfefb4fd401b9cfd2ca36034a010 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer c843193b801b4f1a88f630ed3ed0f375: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:00:39.067221 17222 raft_consensus.cc:1261] T d81dcfefb4fd401b9cfd2ca36034a010 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer c843193b801b4f1a88f630ed3ed0f375: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:00:39.067441 21699 consensus_queue.cc:1035] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:00:39.067698 16814 catalog_manager.cc:5484] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa reported cstate change: config changed from index 9 to 10, VOTER 5b8291b26a7f448b8636187d1caf86d0 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "d43ed6b8663947d99a72cd922cc323aa" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:00:39.067692 16958 raft_consensus.cc:2932] T 63962b169caf4f14888871e246c00075 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 5b8291b26a7f448b8636187d1caf86d0 (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:39.068295 21699 consensus_queue.cc:1035] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:00:39.070094 21704 consensus_queue.cc:1035] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:00:39.070282 21700 raft_consensus.cc:2932] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 6398c8993c3c45dcaab25b42142d5671 (127.0.99.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:39.070768 17751 raft_consensus.cc:1261] T 63962b169caf4f14888871e246c00075 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:00:39.070933 17089 raft_consensus.cc:1261] T 63962b169caf4f14888871e246c00075 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:00:39.071508 17092 raft_consensus.cc:2932] T d81dcfefb4fd401b9cfd2ca36034a010 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 6398c8993c3c45dcaab25b42142d5671 (127.0.99.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:39.071807 21534 consensus_queue.cc:1035] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:00:39.073668 21534 consensus_queue.cc:1035] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:00:39.074162 17752 raft_consensus.cc:2932] T 63962b169caf4f14888871e246c00075 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 5b8291b26a7f448b8636187d1caf86d0 (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:39.074734 17089 raft_consensus.cc:2932] T 63962b169caf4f14888871e246c00075 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 5b8291b26a7f448b8636187d1caf86d0 (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:39.077134 16802 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d81dcfefb4fd401b9cfd2ca36034a010 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:00:39.078181 17486 raft_consensus.cc:2932] T d81dcfefb4fd401b9cfd2ca36034a010 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 6398c8993c3c45dcaab25b42142d5671 (127.0.99.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:39.080132 17222 raft_consensus.cc:2932] T d81dcfefb4fd401b9cfd2ca36034a010 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 6398c8993c3c45dcaab25b42142d5671 (127.0.99.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:39.080914 17202 tablet_service.cc:1514] Processing DeleteTablet for tablet 63962b169caf4f14888871e246c00075 with delete_type TABLET_DATA_TOMBSTONED (TS 5b8291b26a7f448b8636187d1caf86d0 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:35990
I20240503 02:00:39.081537 17752 raft_consensus.cc:2932] T d81dcfefb4fd401b9cfd2ca36034a010 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 6398c8993c3c45dcaab25b42142d5671 (127.0.99.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:39.083012 19108 raft_consensus.cc:2932] T d81dcfefb4fd401b9cfd2ca36034a010 P 838ef968bb45484bb27f5af6f3989bae [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 6398c8993c3c45dcaab25b42142d5671 (127.0.99.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:39.083940 16810 catalog_manager.cc:5484] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 reported cstate change: config changed from index 9 to 10, VOTER 6398c8993c3c45dcaab25b42142d5671 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "c843193b801b4f1a88f630ed3ed0f375" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:00:39.087723 21602 tablet_replica.cc:330] T 63962b169caf4f14888871e246c00075 P 5b8291b26a7f448b8636187d1caf86d0: stopping tablet replica
I20240503 02:00:39.087965 21602 raft_consensus.cc:2229] T 63962b169caf4f14888871e246c00075 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:39.088351 21602 raft_consensus.cc:2258] T 63962b169caf4f14888871e246c00075 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:39.090095 16938 tablet_service.cc:1514] Processing DeleteTablet for tablet d81dcfefb4fd401b9cfd2ca36034a010 with delete_type TABLET_DATA_TOMBSTONED (TS 6398c8993c3c45dcaab25b42142d5671 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:47986
I20240503 02:00:39.090168 21602 ts_tablet_manager.cc:1875] T 63962b169caf4f14888871e246c00075 P 5b8291b26a7f448b8636187d1caf86d0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:39.091159 21679 tablet_replica.cc:330] T d81dcfefb4fd401b9cfd2ca36034a010 P 6398c8993c3c45dcaab25b42142d5671: stopping tablet replica
I20240503 02:00:39.091359 21679 raft_consensus.cc:2229] T d81dcfefb4fd401b9cfd2ca36034a010 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:39.091706 21679 raft_consensus.cc:2258] T d81dcfefb4fd401b9cfd2ca36034a010 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:39.092698 21679 ts_tablet_manager.cc:1875] T d81dcfefb4fd401b9cfd2ca36034a010 P 6398c8993c3c45dcaab25b42142d5671: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:39.093664 19369 raft_consensus.cc:2932] T d81dcfefb4fd401b9cfd2ca36034a010 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 6398c8993c3c45dcaab25b42142d5671 (127.0.99.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:39.094797 21602 ts_tablet_manager.cc:1888] T 63962b169caf4f14888871e246c00075 P 5b8291b26a7f448b8636187d1caf86d0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 02:00:39.094944 21602 log.cc:1196] T 63962b169caf4f14888871e246c00075 P 5b8291b26a7f448b8636187d1caf86d0: Deleting WAL directory at /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/63962b169caf4f14888871e246c00075
I20240503 02:00:39.097314 21679 ts_tablet_manager.cc:1888] T d81dcfefb4fd401b9cfd2ca36034a010 P 6398c8993c3c45dcaab25b42142d5671: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 02:00:39.097548 21679 log.cc:1196] T d81dcfefb4fd401b9cfd2ca36034a010 P 6398c8993c3c45dcaab25b42142d5671: Deleting WAL directory at /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/d81dcfefb4fd401b9cfd2ca36034a010
I20240503 02:00:39.098158 16803 catalog_manager.cc:4830] TS 5b8291b26a7f448b8636187d1caf86d0 (127.0.99.131:44479): tablet 63962b169caf4f14888871e246c00075 (table rebalance_test_table_3 [id=78777800a49e46109c1b3bb45ab99e28]) successfully deleted
I20240503 02:00:39.098698 16804 catalog_manager.cc:4830] TS 6398c8993c3c45dcaab25b42142d5671 (127.0.99.129:36983): tablet d81dcfefb4fd401b9cfd2ca36034a010 (table rebalance_test_table_4 [id=421dd60555d242a78638e28180581a9c]) successfully deleted
I20240503 02:00:39.146342 17092 consensus_queue.cc:237] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } }
I20240503 02:00:39.148453 17752 raft_consensus.cc:1261] T 78f81fe4c6234ff0b5fcabd974779a66 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:39.148510 16958 raft_consensus.cc:1261] T 78f81fe4c6234ff0b5fcabd974779a66 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:39.148906 17486 raft_consensus.cc:1261] T 78f81fe4c6234ff0b5fcabd974779a66 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:39.149020 21635 consensus_queue.cc:1035] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:39.149103 17620 raft_consensus.cc:1261] T 78f81fe4c6234ff0b5fcabd974779a66 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:39.149544 19108 raft_consensus.cc:1261] T 78f81fe4c6234ff0b5fcabd974779a66 P 838ef968bb45484bb27f5af6f3989bae [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:39.149534 17222 raft_consensus.cc:1261] T 78f81fe4c6234ff0b5fcabd974779a66 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:39.149822 21185 consensus_queue.cc:1035] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240503 02:00:39.150082 17026 consensus_peers.cc:485] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 78f81fe4c6234ff0b5fcabd974779a66. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:39.150218 21185 consensus_queue.cc:1035] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:39.150354 21634 consensus_queue.cc:1035] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:39.150605 21636 consensus_queue.cc:1035] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:39.150836 21634 consensus_queue.cc:1035] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:39.153883 21637 raft_consensus.cc:2932] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:39.154139 17486 raft_consensus.cc:2932] T 78f81fe4c6234ff0b5fcabd974779a66 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:39.154139 17620 raft_consensus.cc:2932] T 78f81fe4c6234ff0b5fcabd974779a66 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:39.154446 16958 raft_consensus.cc:2932] T 78f81fe4c6234ff0b5fcabd974779a66 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:39.157263 17752 raft_consensus.cc:2932] T 78f81fe4c6234ff0b5fcabd974779a66 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:39.157500 19108 raft_consensus.cc:2932] T 78f81fe4c6234ff0b5fcabd974779a66 P 838ef968bb45484bb27f5af6f3989bae [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:39.158988 17220 raft_consensus.cc:2932] T 78f81fe4c6234ff0b5fcabd974779a66 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:39.160238 16812 catalog_manager.cc:5484] T 78f81fe4c6234ff0b5fcabd974779a66 P 838ef968bb45484bb27f5af6f3989bae reported cstate change: config changed from index 7 to 8, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "89dd09f8aaee4b249ae3ce924a422a67" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:39.174041 17486 consensus_queue.cc:237] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } attrs { replace: true } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } }
I20240503 02:00:39.176496 17092 raft_consensus.cc:1261] T acfd773c1f384c219d776878407d1144 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:39.176621 19369 raft_consensus.cc:1261] T acfd773c1f384c219d776878407d1144 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:39.176990 16958 raft_consensus.cc:1261] T acfd773c1f384c219d776878407d1144 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:39.177039 21589 consensus_queue.cc:1035] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:39.177201 17220 raft_consensus.cc:1261] T acfd773c1f384c219d776878407d1144 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240503 02:00:39.177619 17422 consensus_peers.cc:485] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: acfd773c1f384c219d776878407d1144. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:39.177871 21586 consensus_queue.cc:1035] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:39.178050 21588 consensus_queue.cc:1035] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:39.178244 21478 consensus_queue.cc:1035] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:39.179225 17620 raft_consensus.cc:1261] T acfd773c1f384c219d776878407d1144 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:39.179734 21586 consensus_queue.cc:1035] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:39.180228 17752 raft_consensus.cc:1261] T acfd773c1f384c219d776878407d1144 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:39.180871 21586 consensus_queue.cc:1035] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:39.181424 21589 raft_consensus.cc:2932] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } attrs { replace: true } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } }
I20240503 02:00:39.183470 17220 raft_consensus.cc:2932] T acfd773c1f384c219d776878407d1144 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } attrs { replace: true } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } }
I20240503 02:00:39.184320 17092 raft_consensus.cc:2932] T acfd773c1f384c219d776878407d1144 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } attrs { replace: true } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } }
I20240503 02:00:39.184705 16958 raft_consensus.cc:2932] T acfd773c1f384c219d776878407d1144 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } attrs { replace: true } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } }
I20240503 02:00:39.186089 17620 raft_consensus.cc:2932] T acfd773c1f384c219d776878407d1144 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } attrs { replace: true } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } }
I20240503 02:00:39.187583 16814 catalog_manager.cc:5484] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa reported cstate change: config changed from index 7 to 8, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New cstate: current_term: 1 leader_uuid: "16d437fe629643948c95d628926d5ffa" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:39.190232 19369 raft_consensus.cc:2932] T acfd773c1f384c219d776878407d1144 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } attrs { replace: true } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } }
I20240503 02:00:39.192965 17752 raft_consensus.cc:2932] T acfd773c1f384c219d776878407d1144 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } attrs { replace: true } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } }
I20240503 02:00:39.565606 21830 ts_tablet_manager.cc:902] T acfd773c1f384c219d776878407d1144 P 838ef968bb45484bb27f5af6f3989bae: Initiating tablet copy from peer 16d437fe629643948c95d628926d5ffa (127.0.99.133:46581)
I20240503 02:00:39.566203 21830 tablet_copy_client.cc:308] T acfd773c1f384c219d776878407d1144 P 838ef968bb45484bb27f5af6f3989bae: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.133:46581
I20240503 02:00:39.567059 17508 tablet_copy_service.cc:140] P 16d437fe629643948c95d628926d5ffa: Received BeginTabletCopySession request for tablet acfd773c1f384c219d776878407d1144 from peer 838ef968bb45484bb27f5af6f3989bae ({username='slave'} at 127.0.99.137:37139)
I20240503 02:00:39.567196 17508 tablet_copy_service.cc:161] P 16d437fe629643948c95d628926d5ffa: Beginning new tablet copy session on tablet acfd773c1f384c219d776878407d1144 from peer 838ef968bb45484bb27f5af6f3989bae at {username='slave'} at 127.0.99.137:37139: session id = 838ef968bb45484bb27f5af6f3989bae-acfd773c1f384c219d776878407d1144
I20240503 02:00:39.568081 17508 tablet_copy_source_session.cc:205] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:39.569216 21830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acfd773c1f384c219d776878407d1144. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:39.571429 21830 tablet_copy_client.cc:783] T acfd773c1f384c219d776878407d1144 P 838ef968bb45484bb27f5af6f3989bae: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:39.571589 21830 tablet_copy_client.cc:647] T acfd773c1f384c219d776878407d1144 P 838ef968bb45484bb27f5af6f3989bae: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:39.572870 21830 tablet_copy_client.cc:523] T acfd773c1f384c219d776878407d1144 P 838ef968bb45484bb27f5af6f3989bae: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:39.573910 21830 tablet_bootstrap.cc:492] T acfd773c1f384c219d776878407d1144 P 838ef968bb45484bb27f5af6f3989bae: Bootstrap starting.
I20240503 02:00:39.594823 21830 tablet_bootstrap.cc:492] T acfd773c1f384c219d776878407d1144 P 838ef968bb45484bb27f5af6f3989bae: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:39.595130 21830 tablet_bootstrap.cc:492] T acfd773c1f384c219d776878407d1144 P 838ef968bb45484bb27f5af6f3989bae: Bootstrap complete.
I20240503 02:00:39.595371 21830 ts_tablet_manager.cc:1367] T acfd773c1f384c219d776878407d1144 P 838ef968bb45484bb27f5af6f3989bae: Time spent bootstrapping tablet: real 0.021s user 0.022s sys 0.000s
I20240503 02:00:39.595760 21830 raft_consensus.cc:348] T acfd773c1f384c219d776878407d1144 P 838ef968bb45484bb27f5af6f3989bae [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } attrs { replace: true } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } }
I20240503 02:00:39.595883 21830 raft_consensus.cc:729] T acfd773c1f384c219d776878407d1144 P 838ef968bb45484bb27f5af6f3989bae [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 838ef968bb45484bb27f5af6f3989bae, State: Initialized, Role: LEARNER
I20240503 02:00:39.595957 21830 consensus_queue.cc:260] T acfd773c1f384c219d776878407d1144 P 838ef968bb45484bb27f5af6f3989bae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } attrs { replace: true } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } }
I20240503 02:00:39.596232 21830 ts_tablet_manager.cc:1398] T acfd773c1f384c219d776878407d1144 P 838ef968bb45484bb27f5af6f3989bae: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:39.596884 17508 tablet_copy_service.cc:342] P 16d437fe629643948c95d628926d5ffa: Request end of tablet copy session 838ef968bb45484bb27f5af6f3989bae-acfd773c1f384c219d776878407d1144 received from {username='slave'} at 127.0.99.137:37139
I20240503 02:00:39.596978 17508 tablet_copy_service.cc:433] P 16d437fe629643948c95d628926d5ffa: ending tablet copy session 838ef968bb45484bb27f5af6f3989bae-acfd773c1f384c219d776878407d1144 on tablet acfd773c1f384c219d776878407d1144 with peer 838ef968bb45484bb27f5af6f3989bae
I20240503 02:00:39.717141 21847 ts_tablet_manager.cc:902] T 78f81fe4c6234ff0b5fcabd974779a66 P 375d0f2ec11d4e25bd9d093bb6a5572d: Initiating tablet copy from peer 89dd09f8aaee4b249ae3ce924a422a67 (127.0.99.130:38889)
I20240503 02:00:39.717864 21847 tablet_copy_client.cc:308] T 78f81fe4c6234ff0b5fcabd974779a66 P 375d0f2ec11d4e25bd9d093bb6a5572d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:38889
I20240503 02:00:39.718616 17111 tablet_copy_service.cc:140] P 89dd09f8aaee4b249ae3ce924a422a67: Received BeginTabletCopySession request for tablet 78f81fe4c6234ff0b5fcabd974779a66 from peer 375d0f2ec11d4e25bd9d093bb6a5572d ({username='slave'} at 127.0.99.136:56011)
I20240503 02:00:39.718796 17111 tablet_copy_service.cc:161] P 89dd09f8aaee4b249ae3ce924a422a67: Beginning new tablet copy session on tablet 78f81fe4c6234ff0b5fcabd974779a66 from peer 375d0f2ec11d4e25bd9d093bb6a5572d at {username='slave'} at 127.0.99.136:56011: session id = 375d0f2ec11d4e25bd9d093bb6a5572d-78f81fe4c6234ff0b5fcabd974779a66
I20240503 02:00:39.720291 17111 tablet_copy_source_session.cc:205] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:39.721664 21847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78f81fe4c6234ff0b5fcabd974779a66. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:39.724099 21847 tablet_copy_client.cc:783] T 78f81fe4c6234ff0b5fcabd974779a66 P 375d0f2ec11d4e25bd9d093bb6a5572d: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:39.724270 21847 tablet_copy_client.cc:647] T 78f81fe4c6234ff0b5fcabd974779a66 P 375d0f2ec11d4e25bd9d093bb6a5572d: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:39.725952 21847 tablet_copy_client.cc:523] T 78f81fe4c6234ff0b5fcabd974779a66 P 375d0f2ec11d4e25bd9d093bb6a5572d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:39.727399 21847 tablet_bootstrap.cc:492] T 78f81fe4c6234ff0b5fcabd974779a66 P 375d0f2ec11d4e25bd9d093bb6a5572d: Bootstrap starting.
I20240503 02:00:39.757089 21847 tablet_bootstrap.cc:492] T 78f81fe4c6234ff0b5fcabd974779a66 P 375d0f2ec11d4e25bd9d093bb6a5572d: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:39.757411 21847 tablet_bootstrap.cc:492] T 78f81fe4c6234ff0b5fcabd974779a66 P 375d0f2ec11d4e25bd9d093bb6a5572d: Bootstrap complete.
I20240503 02:00:39.757624 21847 ts_tablet_manager.cc:1367] T 78f81fe4c6234ff0b5fcabd974779a66 P 375d0f2ec11d4e25bd9d093bb6a5572d: Time spent bootstrapping tablet: real 0.030s user 0.026s sys 0.005s
I20240503 02:00:39.758047 21847 raft_consensus.cc:348] T 78f81fe4c6234ff0b5fcabd974779a66 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } }
I20240503 02:00:39.758165 21847 raft_consensus.cc:729] T 78f81fe4c6234ff0b5fcabd974779a66 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 375d0f2ec11d4e25bd9d093bb6a5572d, State: Initialized, Role: LEARNER
I20240503 02:00:39.758272 21847 consensus_queue.cc:260] T 78f81fe4c6234ff0b5fcabd974779a66 P 375d0f2ec11d4e25bd9d093bb6a5572d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } }
I20240503 02:00:39.758669 21847 ts_tablet_manager.cc:1398] T 78f81fe4c6234ff0b5fcabd974779a66 P 375d0f2ec11d4e25bd9d093bb6a5572d: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 02:00:39.759491 17111 tablet_copy_service.cc:342] P 89dd09f8aaee4b249ae3ce924a422a67: Request end of tablet copy session 375d0f2ec11d4e25bd9d093bb6a5572d-78f81fe4c6234ff0b5fcabd974779a66 received from {username='slave'} at 127.0.99.136:56011
I20240503 02:00:39.759610 17111 tablet_copy_service.cc:433] P 89dd09f8aaee4b249ae3ce924a422a67: ending tablet copy session 375d0f2ec11d4e25bd9d093bb6a5572d-78f81fe4c6234ff0b5fcabd974779a66 on tablet 78f81fe4c6234ff0b5fcabd974779a66 with peer 375d0f2ec11d4e25bd9d093bb6a5572d
I20240503 02:00:39.950326 19108 raft_consensus.cc:1203] T acfd773c1f384c219d776878407d1144 P 838ef968bb45484bb27f5af6f3989bae [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 02:00:40.094544 19369 raft_consensus.cc:1203] T 78f81fe4c6234ff0b5fcabd974779a66 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 02:00:40.481276 21588 raft_consensus.cc:1050] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa: attempting to promote NON_VOTER 838ef968bb45484bb27f5af6f3989bae to VOTER
I20240503 02:00:40.482044 21588 consensus_queue.cc:237] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } attrs { replace: true } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } }
I20240503 02:00:40.483937 19108 raft_consensus.cc:1261] T acfd773c1f384c219d776878407d1144 P 838ef968bb45484bb27f5af6f3989bae [term 1 LEARNER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:40.484305 19369 raft_consensus.cc:1261] T acfd773c1f384c219d776878407d1144 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:40.484465 21587 consensus_queue.cc:1035] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:40.485489 17752 raft_consensus.cc:1261] T acfd773c1f384c219d776878407d1144 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:40.485800 17220 raft_consensus.cc:1261] T acfd773c1f384c219d776878407d1144 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:40.485862 21888 consensus_queue.cc:1035] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:40.486217 21888 consensus_queue.cc:1035] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:40.486346 16958 raft_consensus.cc:1261] T acfd773c1f384c219d776878407d1144 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:40.486574 21888 consensus_queue.cc:1035] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:40.486761 21587 consensus_queue.cc:1035] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:40.486932 17089 raft_consensus.cc:1261] T acfd773c1f384c219d776878407d1144 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:40.488055 21888 consensus_queue.cc:1035] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:40.488212 17620 raft_consensus.cc:1261] T acfd773c1f384c219d776878407d1144 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:40.489218 21887 consensus_queue.cc:1035] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:40.490279 21588 raft_consensus.cc:2932] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } attrs { replace: true } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:40.490679 17220 raft_consensus.cc:2932] T acfd773c1f384c219d776878407d1144 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } attrs { replace: true } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:40.490844 17752 raft_consensus.cc:2932] T acfd773c1f384c219d776878407d1144 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } attrs { replace: true } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:40.493299 19369 raft_consensus.cc:2932] T acfd773c1f384c219d776878407d1144 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } attrs { replace: true } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:40.493475 16958 raft_consensus.cc:2932] T acfd773c1f384c219d776878407d1144 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } attrs { replace: true } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:40.494096 16810 catalog_manager.cc:5484] T acfd773c1f384c219d776878407d1144 P 5b8291b26a7f448b8636187d1caf86d0 reported cstate change: config changed from index 8 to 9, 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "16d437fe629643948c95d628926d5ffa" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } attrs { replace: true } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:40.494520 17089 raft_consensus.cc:2932] T acfd773c1f384c219d776878407d1144 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } attrs { replace: true } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:40.495844 19108 raft_consensus.cc:2932] T acfd773c1f384c219d776878407d1144 P 838ef968bb45484bb27f5af6f3989bae [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } attrs { replace: true } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:40.499895 17620 raft_consensus.cc:2932] T acfd773c1f384c219d776878407d1144 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } attrs { replace: true } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:40.508008 17486 consensus_queue.cc:237] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } }
I20240503 02:00:40.510294 19108 raft_consensus.cc:1261] T acfd773c1f384c219d776878407d1144 P 838ef968bb45484bb27f5af6f3989bae [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:00:40.510339 17752 raft_consensus.cc:1261] T acfd773c1f384c219d776878407d1144 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:00:40.510419 16958 raft_consensus.cc:1261] T acfd773c1f384c219d776878407d1144 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:00:40.510689 17620 raft_consensus.cc:1261] T acfd773c1f384c219d776878407d1144 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:00:40.510883 19369 raft_consensus.cc:1261] T acfd773c1f384c219d776878407d1144 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:00:40.510879 21886 consensus_queue.cc:1035] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:00:40.511190 21888 consensus_queue.cc:1035] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:00:40.511418 21887 consensus_queue.cc:1035] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:00:40.511620 21886 consensus_queue.cc:1035] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:00:40.511916 21588 consensus_queue.cc:1035] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:00:40.514923 17220 raft_consensus.cc:1261] T acfd773c1f384c219d776878407d1144 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:00:40.515436 21887 raft_consensus.cc:2932] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 89dd09f8aaee4b249ae3ce924a422a67 (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:40.515746 21588 consensus_queue.cc:1035] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:00:40.516019 19108 raft_consensus.cc:2932] T acfd773c1f384c219d776878407d1144 P 838ef968bb45484bb27f5af6f3989bae [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 89dd09f8aaee4b249ae3ce924a422a67 (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:40.516460 16958 raft_consensus.cc:2932] T acfd773c1f384c219d776878407d1144 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 89dd09f8aaee4b249ae3ce924a422a67 (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:40.517542 17220 raft_consensus.cc:2932] T acfd773c1f384c219d776878407d1144 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 89dd09f8aaee4b249ae3ce924a422a67 (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:40.518015 19369 raft_consensus.cc:2932] T acfd773c1f384c219d776878407d1144 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 89dd09f8aaee4b249ae3ce924a422a67 (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:40.518239 16802 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet acfd773c1f384c219d776878407d1144 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:00:40.519771 16812 catalog_manager.cc:5484] T acfd773c1f384c219d776878407d1144 P 838ef968bb45484bb27f5af6f3989bae reported cstate change: config changed from index 9 to 10, VOTER 89dd09f8aaee4b249ae3ce924a422a67 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "16d437fe629643948c95d628926d5ffa" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:40.520502 17752 raft_consensus.cc:2932] T acfd773c1f384c219d776878407d1144 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 89dd09f8aaee4b249ae3ce924a422a67 (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:40.520511 17620 raft_consensus.cc:2932] T acfd773c1f384c219d776878407d1144 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 89dd09f8aaee4b249ae3ce924a422a67 (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } }
I20240503 02:00:40.526793 17070 tablet_service.cc:1514] Processing DeleteTablet for tablet acfd773c1f384c219d776878407d1144 with delete_type TABLET_DATA_TOMBSTONED (TS 89dd09f8aaee4b249ae3ce924a422a67 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:56194
I20240503 02:00:40.529470 21908 tablet_replica.cc:330] T acfd773c1f384c219d776878407d1144 P 89dd09f8aaee4b249ae3ce924a422a67: stopping tablet replica
I20240503 02:00:40.529676 21908 raft_consensus.cc:2229] T acfd773c1f384c219d776878407d1144 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:40.530007 21908 raft_consensus.cc:2258] T acfd773c1f384c219d776878407d1144 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:40.530766 21908 ts_tablet_manager.cc:1875] T acfd773c1f384c219d776878407d1144 P 89dd09f8aaee4b249ae3ce924a422a67: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:40.533950 21908 ts_tablet_manager.cc:1888] T acfd773c1f384c219d776878407d1144 P 89dd09f8aaee4b249ae3ce924a422a67: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 02:00:40.534235 21908 log.cc:1196] T acfd773c1f384c219d776878407d1144 P 89dd09f8aaee4b249ae3ce924a422a67: Deleting WAL directory at /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/acfd773c1f384c219d776878407d1144
I20240503 02:00:40.535017 16805 catalog_manager.cc:4830] TS 89dd09f8aaee4b249ae3ce924a422a67 (127.0.99.130:38889): tablet acfd773c1f384c219d776878407d1144 (table rebalance_test_table_0 [id=ee1e0b20af0b4ecea04ae147701a0482]) successfully deleted
I20240503 02:00:40.711453 21638 raft_consensus.cc:1050] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67: attempting to promote NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d to VOTER
I20240503 02:00:40.712006 21638 consensus_queue.cc:237] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } }
I20240503 02:00:40.713814 19369 raft_consensus.cc:1261] T 78f81fe4c6234ff0b5fcabd974779a66 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 LEARNER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:40.713999 19108 raft_consensus.cc:1261] T 78f81fe4c6234ff0b5fcabd974779a66 P 838ef968bb45484bb27f5af6f3989bae [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:40.714390 17486 raft_consensus.cc:1261] T 78f81fe4c6234ff0b5fcabd974779a66 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:40.714424 17620 raft_consensus.cc:1261] T 78f81fe4c6234ff0b5fcabd974779a66 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:40.714838 17752 raft_consensus.cc:1261] T 78f81fe4c6234ff0b5fcabd974779a66 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:40.714921 21923 consensus_queue.cc:1035] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:40.715075 16958 raft_consensus.cc:1261] T 78f81fe4c6234ff0b5fcabd974779a66 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:40.715097 17220 raft_consensus.cc:1261] T 78f81fe4c6234ff0b5fcabd974779a66 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:40.715343 21854 consensus_queue.cc:1035] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:40.715662 21638 consensus_queue.cc:1035] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:40.715857 21921 consensus_queue.cc:1035] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:40.716230 21923 consensus_queue.cc:1035] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:40.716637 21924 consensus_queue.cc:1035] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:40.717119 21925 consensus_queue.cc:1035] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:40.721500 21925 raft_consensus.cc:2932] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:40.723654 16958 raft_consensus.cc:2932] T 78f81fe4c6234ff0b5fcabd974779a66 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:40.724077 19108 raft_consensus.cc:2932] T 78f81fe4c6234ff0b5fcabd974779a66 P 838ef968bb45484bb27f5af6f3989bae [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:40.724817 16816 catalog_manager.cc:5484] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 reported cstate change: config changed from index 8 to 9, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "89dd09f8aaee4b249ae3ce924a422a67" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:40.726850 17486 raft_consensus.cc:2932] T 78f81fe4c6234ff0b5fcabd974779a66 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:40.728363 17752 raft_consensus.cc:2932] T 78f81fe4c6234ff0b5fcabd974779a66 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:40.728416 17620 raft_consensus.cc:2932] T 78f81fe4c6234ff0b5fcabd974779a66 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:40.729982 17220 raft_consensus.cc:2932] T 78f81fe4c6234ff0b5fcabd974779a66 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:40.731624 19369 raft_consensus.cc:2932] T 78f81fe4c6234ff0b5fcabd974779a66 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:40.739605 17089 consensus_queue.cc:237] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } }
I20240503 02:00:40.741568 19108 raft_consensus.cc:1261] T 78f81fe4c6234ff0b5fcabd974779a66 P 838ef968bb45484bb27f5af6f3989bae [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:00:40.741730 17752 raft_consensus.cc:1261] T 78f81fe4c6234ff0b5fcabd974779a66 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:00:40.741902 19369 raft_consensus.cc:1261] T 78f81fe4c6234ff0b5fcabd974779a66 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:00:40.742100 16958 raft_consensus.cc:1261] T 78f81fe4c6234ff0b5fcabd974779a66 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:00:40.742298 21922 consensus_queue.cc:1035] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:00:40.742524 21924 consensus_queue.cc:1035] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:00:40.742969 17486 raft_consensus.cc:1261] T 78f81fe4c6234ff0b5fcabd974779a66 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:00:40.743031 17620 raft_consensus.cc:1261] T 78f81fe4c6234ff0b5fcabd974779a66 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:00:40.743310 21922 consensus_queue.cc:1035] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:00:40.743464 21924 consensus_queue.cc:1035] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:00:40.744273 21921 consensus_queue.cc:1035] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:00:40.745062 21854 consensus_queue.cc:1035] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:00:40.746356 21922 raft_consensus.cc:2932] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 5b8291b26a7f448b8636187d1caf86d0 (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:40.746718 17620 raft_consensus.cc:2932] T 78f81fe4c6234ff0b5fcabd974779a66 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 5b8291b26a7f448b8636187d1caf86d0 (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:40.749287 16805 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 78f81fe4c6234ff0b5fcabd974779a66 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:00:40.750128 19108 raft_consensus.cc:2932] T 78f81fe4c6234ff0b5fcabd974779a66 P 838ef968bb45484bb27f5af6f3989bae [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 5b8291b26a7f448b8636187d1caf86d0 (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:40.749442 17752 raft_consensus.cc:2932] T 78f81fe4c6234ff0b5fcabd974779a66 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 5b8291b26a7f448b8636187d1caf86d0 (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:40.752440 16816 catalog_manager.cc:5484] T 78f81fe4c6234ff0b5fcabd974779a66 P c843193b801b4f1a88f630ed3ed0f375 reported cstate change: config changed from index 9 to 10, VOTER 5b8291b26a7f448b8636187d1caf86d0 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "89dd09f8aaee4b249ae3ce924a422a67" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:40.752413 16958 raft_consensus.cc:2932] T 78f81fe4c6234ff0b5fcabd974779a66 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 5b8291b26a7f448b8636187d1caf86d0 (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:40.757560 17486 raft_consensus.cc:2932] T 78f81fe4c6234ff0b5fcabd974779a66 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 5b8291b26a7f448b8636187d1caf86d0 (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:40.759150 17202 tablet_service.cc:1514] Processing DeleteTablet for tablet 78f81fe4c6234ff0b5fcabd974779a66 with delete_type TABLET_DATA_TOMBSTONED (TS 5b8291b26a7f448b8636187d1caf86d0 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:35990
I20240503 02:00:40.759757 21938 tablet_replica.cc:330] T 78f81fe4c6234ff0b5fcabd974779a66 P 5b8291b26a7f448b8636187d1caf86d0: stopping tablet replica
I20240503 02:00:40.760012 21938 raft_consensus.cc:2229] T 78f81fe4c6234ff0b5fcabd974779a66 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:40.760418 21938 raft_consensus.cc:2258] T 78f81fe4c6234ff0b5fcabd974779a66 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:40.762234 21938 ts_tablet_manager.cc:1875] T 78f81fe4c6234ff0b5fcabd974779a66 P 5b8291b26a7f448b8636187d1caf86d0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:40.763733 19369 raft_consensus.cc:2932] T 78f81fe4c6234ff0b5fcabd974779a66 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 5b8291b26a7f448b8636187d1caf86d0 (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:40.764905 21938 ts_tablet_manager.cc:1888] T 78f81fe4c6234ff0b5fcabd974779a66 P 5b8291b26a7f448b8636187d1caf86d0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 02:00:40.765020 21938 log.cc:1196] T 78f81fe4c6234ff0b5fcabd974779a66 P 5b8291b26a7f448b8636187d1caf86d0: Deleting WAL directory at /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/78f81fe4c6234ff0b5fcabd974779a66
I20240503 02:00:40.765843 16803 catalog_manager.cc:4830] TS 5b8291b26a7f448b8636187d1caf86d0 (127.0.99.131:44479): tablet 78f81fe4c6234ff0b5fcabd974779a66 (table rebalance_test_table_2 [id=9f4cafd5cdb041c1bbe1592f04370c0b]) successfully deleted
I20240503 02:00:40.846837 17202 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:40.846940 17467 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:00:40.847319 19088 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:40.848491 17070 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:00:40.860152 16938 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:40.868821 17335 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:00:40.872764 17598 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:40.882786 19350 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:40.891439 17730 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:00:40.912485 19219 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:40.951982 16957 consensus_queue.cc:237] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } }
I20240503 02:00:40.954087 19237 raft_consensus.cc:1261] T 885fe0c3a5e04f408e5b02c3c133248a P 9564311af29a4493831a8db658a085c4 [term 1 FOLLOWER]: Refusing update from remote peer 6398c8993c3c45dcaab25b42142d5671: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:40.954501 17752 raft_consensus.cc:1261] T 885fe0c3a5e04f408e5b02c3c133248a P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer 6398c8993c3c45dcaab25b42142d5671: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:40.954613 17222 raft_consensus.cc:1261] T 885fe0c3a5e04f408e5b02c3c133248a P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer 6398c8993c3c45dcaab25b42142d5671: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:40.954625 17620 raft_consensus.cc:1261] T 885fe0c3a5e04f408e5b02c3c133248a P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer 6398c8993c3c45dcaab25b42142d5671: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:40.954977 17486 raft_consensus.cc:1261] T 885fe0c3a5e04f408e5b02c3c133248a P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Refusing update from remote peer 6398c8993c3c45dcaab25b42142d5671: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:40.955024 21992 consensus_queue.cc:1035] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:40.955165 17089 raft_consensus.cc:1261] T 885fe0c3a5e04f408e5b02c3c133248a P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer 6398c8993c3c45dcaab25b42142d5671: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:40.955353 21993 consensus_queue.cc:1035] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671 [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:40.955579 21654 consensus_queue.cc:1035] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:40.955739 21991 consensus_queue.cc:1035] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:40.955920 21995 consensus_queue.cc:1035] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:40.956210 21654 consensus_queue.cc:1035] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:40.959137 21992 raft_consensus.cc:2932] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:40.959530 19237 raft_consensus.cc:2932] T 885fe0c3a5e04f408e5b02c3c133248a P 9564311af29a4493831a8db658a085c4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:40.961378 17620 raft_consensus.cc:2932] T 885fe0c3a5e04f408e5b02c3c133248a P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:40.962440 17089 raft_consensus.cc:2932] T 885fe0c3a5e04f408e5b02c3c133248a P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:40.962668 17222 raft_consensus.cc:2932] T 885fe0c3a5e04f408e5b02c3c133248a P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:40.962709 16813 catalog_manager.cc:5484] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671 reported cstate change: config changed from index 7 to 8, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "6398c8993c3c45dcaab25b42142d5671" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 02:00:40.966727 16894 consensus_peers.cc:485] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 885fe0c3a5e04f408e5b02c3c133248a. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:40.967012 17752 raft_consensus.cc:2932] T 885fe0c3a5e04f408e5b02c3c133248a P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:40.968555 17486 raft_consensus.cc:2932] T 885fe0c3a5e04f408e5b02c3c133248a P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:41.373261 22014 ts_tablet_manager.cc:902] T 885fe0c3a5e04f408e5b02c3c133248a P 375d0f2ec11d4e25bd9d093bb6a5572d: Initiating tablet copy from peer 6398c8993c3c45dcaab25b42142d5671 (127.0.99.129:36983)
I20240503 02:00:41.373766 22014 tablet_copy_client.cc:308] T 885fe0c3a5e04f408e5b02c3c133248a P 375d0f2ec11d4e25bd9d093bb6a5572d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:36983
I20240503 02:00:41.377400 16980 tablet_copy_service.cc:140] P 6398c8993c3c45dcaab25b42142d5671: Received BeginTabletCopySession request for tablet 885fe0c3a5e04f408e5b02c3c133248a from peer 375d0f2ec11d4e25bd9d093bb6a5572d ({username='slave'} at 127.0.99.136:55495)
I20240503 02:00:41.377496 16980 tablet_copy_service.cc:161] P 6398c8993c3c45dcaab25b42142d5671: Beginning new tablet copy session on tablet 885fe0c3a5e04f408e5b02c3c133248a from peer 375d0f2ec11d4e25bd9d093bb6a5572d at {username='slave'} at 127.0.99.136:55495: session id = 375d0f2ec11d4e25bd9d093bb6a5572d-885fe0c3a5e04f408e5b02c3c133248a
I20240503 02:00:41.378346 16980 tablet_copy_source_session.cc:205] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:41.379254 22014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 885fe0c3a5e04f408e5b02c3c133248a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:41.381008 22014 tablet_copy_client.cc:783] T 885fe0c3a5e04f408e5b02c3c133248a P 375d0f2ec11d4e25bd9d093bb6a5572d: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:41.381203 22014 tablet_copy_client.cc:647] T 885fe0c3a5e04f408e5b02c3c133248a P 375d0f2ec11d4e25bd9d093bb6a5572d: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:41.382392 22014 tablet_copy_client.cc:523] T 885fe0c3a5e04f408e5b02c3c133248a P 375d0f2ec11d4e25bd9d093bb6a5572d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:41.383522 22014 tablet_bootstrap.cc:492] T 885fe0c3a5e04f408e5b02c3c133248a P 375d0f2ec11d4e25bd9d093bb6a5572d: Bootstrap starting.
I20240503 02:00:41.405748 22014 tablet_bootstrap.cc:492] T 885fe0c3a5e04f408e5b02c3c133248a P 375d0f2ec11d4e25bd9d093bb6a5572d: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:41.405988 22014 tablet_bootstrap.cc:492] T 885fe0c3a5e04f408e5b02c3c133248a P 375d0f2ec11d4e25bd9d093bb6a5572d: Bootstrap complete.
I20240503 02:00:41.406121 22014 ts_tablet_manager.cc:1367] T 885fe0c3a5e04f408e5b02c3c133248a P 375d0f2ec11d4e25bd9d093bb6a5572d: Time spent bootstrapping tablet: real 0.023s user 0.020s sys 0.001s
I20240503 02:00:41.406483 22014 raft_consensus.cc:348] T 885fe0c3a5e04f408e5b02c3c133248a P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } }
I20240503 02:00:41.406620 22014 raft_consensus.cc:729] T 885fe0c3a5e04f408e5b02c3c133248a P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 375d0f2ec11d4e25bd9d093bb6a5572d, State: Initialized, Role: LEARNER
I20240503 02:00:41.406713 22014 consensus_queue.cc:260] T 885fe0c3a5e04f408e5b02c3c133248a P 375d0f2ec11d4e25bd9d093bb6a5572d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } }
I20240503 02:00:41.407552 22014 ts_tablet_manager.cc:1398] T 885fe0c3a5e04f408e5b02c3c133248a P 375d0f2ec11d4e25bd9d093bb6a5572d: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 02:00:41.408006 16980 tablet_copy_service.cc:342] P 6398c8993c3c45dcaab25b42142d5671: Request end of tablet copy session 375d0f2ec11d4e25bd9d093bb6a5572d-885fe0c3a5e04f408e5b02c3c133248a received from {username='slave'} at 127.0.99.136:55495
I20240503 02:00:41.408087 16980 tablet_copy_service.cc:433] P 6398c8993c3c45dcaab25b42142d5671: ending tablet copy session 375d0f2ec11d4e25bd9d093bb6a5572d-885fe0c3a5e04f408e5b02c3c133248a on tablet 885fe0c3a5e04f408e5b02c3c133248a with peer 375d0f2ec11d4e25bd9d093bb6a5572d
I20240503 02:00:41.823482 19369 raft_consensus.cc:1203] T 885fe0c3a5e04f408e5b02c3c133248a P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 02:00:42.370270 22049 raft_consensus.cc:1050] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671: attempting to promote NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d to VOTER
I20240503 02:00:42.370836 22049 consensus_queue.cc:237] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } }
I20240503 02:00:42.373129 19237 raft_consensus.cc:1261] T 885fe0c3a5e04f408e5b02c3c133248a P 9564311af29a4493831a8db658a085c4 [term 1 FOLLOWER]: Refusing update from remote peer 6398c8993c3c45dcaab25b42142d5671: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:42.373262 17486 raft_consensus.cc:1261] T 885fe0c3a5e04f408e5b02c3c133248a P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Refusing update from remote peer 6398c8993c3c45dcaab25b42142d5671: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:42.373301 17620 raft_consensus.cc:1261] T 885fe0c3a5e04f408e5b02c3c133248a P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer 6398c8993c3c45dcaab25b42142d5671: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:42.373507 19369 raft_consensus.cc:1261] T 885fe0c3a5e04f408e5b02c3c133248a P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 LEARNER]: Refusing update from remote peer 6398c8993c3c45dcaab25b42142d5671: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:42.373648 22055 consensus_queue.cc:1035] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:42.374071 22055 consensus_queue.cc:1035] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671 [LEADER]: Connected to new peer: Peer: permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:42.374202 17222 raft_consensus.cc:1261] T 885fe0c3a5e04f408e5b02c3c133248a P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer 6398c8993c3c45dcaab25b42142d5671: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:42.374635 22055 consensus_queue.cc:1035] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:42.374788 17089 raft_consensus.cc:1261] T 885fe0c3a5e04f408e5b02c3c133248a P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer 6398c8993c3c45dcaab25b42142d5671: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:42.374846 22053 consensus_queue.cc:1035] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:42.375192 22052 consensus_queue.cc:1035] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:42.375654 22052 consensus_queue.cc:1035] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:42.378443 17752 raft_consensus.cc:1261] T 885fe0c3a5e04f408e5b02c3c133248a P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer 6398c8993c3c45dcaab25b42142d5671: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:42.378240 22053 raft_consensus.cc:2932] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:42.378959 22055 consensus_queue.cc:1035] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671 [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:42.378788 17620 raft_consensus.cc:2932] T 885fe0c3a5e04f408e5b02c3c133248a P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:42.380079 17752 raft_consensus.cc:2932] T 885fe0c3a5e04f408e5b02c3c133248a P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:42.382220 19237 raft_consensus.cc:2932] T 885fe0c3a5e04f408e5b02c3c133248a P 9564311af29a4493831a8db658a085c4 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:42.382398 16818 catalog_manager.cc:5484] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671 reported cstate change: config changed from index 8 to 9, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6398c8993c3c45dcaab25b42142d5671" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:42.382474 17486 raft_consensus.cc:2932] T 885fe0c3a5e04f408e5b02c3c133248a P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:42.384275 17222 raft_consensus.cc:2932] T 885fe0c3a5e04f408e5b02c3c133248a P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:42.386092 17092 raft_consensus.cc:2932] T 885fe0c3a5e04f408e5b02c3c133248a P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:42.386279 19369 raft_consensus.cc:2932] T 885fe0c3a5e04f408e5b02c3c133248a P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:42.428519 16958 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "885fe0c3a5e04f408e5b02c3c133248a"
dest_uuid: "6398c8993c3c45dcaab25b42142d5671"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:37800
I20240503 02:00:42.428732 16958 raft_consensus.cc:595] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671 [term 1 LEADER]: Received request to transfer leadership
I20240503 02:00:42.641557 16958 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "885fe0c3a5e04f408e5b02c3c133248a"
dest_uuid: "6398c8993c3c45dcaab25b42142d5671"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:37818
I20240503 02:00:42.641717 16958 raft_consensus.cc:595] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671 [term 1 LEADER]: Received request to transfer leadership
I20240503 02:00:42.763200 22055 raft_consensus.cc:979] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671: : Instructing follower 9564311af29a4493831a8db658a085c4 to start an election
I20240503 02:00:42.763370 22049 raft_consensus.cc:1067] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671 [term 1 LEADER]: Signalling peer 9564311af29a4493831a8db658a085c4 to start an election
I20240503 02:00:42.764142 19237 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "885fe0c3a5e04f408e5b02c3c133248a"
dest_uuid: "9564311af29a4493831a8db658a085c4"
from {username='slave'} at 127.0.99.129:33959
I20240503 02:00:42.764369 19237 raft_consensus.cc:482] T 885fe0c3a5e04f408e5b02c3c133248a P 9564311af29a4493831a8db658a085c4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:42.764492 19237 raft_consensus.cc:3037] T 885fe0c3a5e04f408e5b02c3c133248a P 9564311af29a4493831a8db658a085c4 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:42.765569 19237 raft_consensus.cc:504] T 885fe0c3a5e04f408e5b02c3c133248a P 9564311af29a4493831a8db658a085c4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } }
I20240503 02:00:42.768000 17486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "885fe0c3a5e04f408e5b02c3c133248a" candidate_uuid: "9564311af29a4493831a8db658a085c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "16d437fe629643948c95d628926d5ffa"
I20240503 02:00:42.768190 17486 raft_consensus.cc:3037] T 885fe0c3a5e04f408e5b02c3c133248a P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:42.768911 19237 leader_election.cc:288] T 885fe0c3a5e04f408e5b02c3c133248a P 9564311af29a4493831a8db658a085c4 [CANDIDATE]: Term 2 election: Requested vote from peers 89dd09f8aaee4b249ae3ce924a422a67 (127.0.99.130:38889), 6398c8993c3c45dcaab25b42142d5671 (127.0.99.129:36983), 5b8291b26a7f448b8636187d1caf86d0 (127.0.99.131:44479), 16d437fe629643948c95d628926d5ffa (127.0.99.133:46581), c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134:44343), 002cc3411acb4fdfa0b67c6f57f687a6 (127.0.99.135:38989), 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167)
I20240503 02:00:42.769570 17752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "885fe0c3a5e04f408e5b02c3c133248a" candidate_uuid: "9564311af29a4493831a8db658a085c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "002cc3411acb4fdfa0b67c6f57f687a6"
I20240503 02:00:42.769704 17486 raft_consensus.cc:2454] T 885fe0c3a5e04f408e5b02c3c133248a P 16d437fe629643948c95d628926d5ffa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9564311af29a4493831a8db658a085c4 in term 2.
I20240503 02:00:42.769729 17752 raft_consensus.cc:3037] T 885fe0c3a5e04f408e5b02c3c133248a P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:42.770335 16958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "885fe0c3a5e04f408e5b02c3c133248a" candidate_uuid: "9564311af29a4493831a8db658a085c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "6398c8993c3c45dcaab25b42142d5671"
I20240503 02:00:42.770476 16958 raft_consensus.cc:3032] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:00:42.770527 16958 raft_consensus.cc:729] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6398c8993c3c45dcaab25b42142d5671, State: Running, Role: LEADER
I20240503 02:00:42.770745 16958 consensus_queue.cc:260] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } }
I20240503 02:00:42.771005 17752 raft_consensus.cc:2454] T 885fe0c3a5e04f408e5b02c3c133248a P 002cc3411acb4fdfa0b67c6f57f687a6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9564311af29a4493831a8db658a085c4 in term 2.
I20240503 02:00:42.771174 16958 raft_consensus.cc:3037] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:42.772013 17092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "885fe0c3a5e04f408e5b02c3c133248a" candidate_uuid: "9564311af29a4493831a8db658a085c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "89dd09f8aaee4b249ae3ce924a422a67"
I20240503 02:00:42.772140 17092 raft_consensus.cc:3037] T 885fe0c3a5e04f408e5b02c3c133248a P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:42.772343 16958 raft_consensus.cc:2454] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9564311af29a4493831a8db658a085c4 in term 2.
I20240503 02:00:42.773273 17092 raft_consensus.cc:2454] T 885fe0c3a5e04f408e5b02c3c133248a P 89dd09f8aaee4b249ae3ce924a422a67 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9564311af29a4493831a8db658a085c4 in term 2.
I20240503 02:00:42.773643 19175 leader_election.cc:302] T 885fe0c3a5e04f408e5b02c3c133248a P 9564311af29a4493831a8db658a085c4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 5 responses out of 8 voters: 5 yes votes; 0 no votes. yes voters: 002cc3411acb4fdfa0b67c6f57f687a6, 16d437fe629643948c95d628926d5ffa, 6398c8993c3c45dcaab25b42142d5671, 89dd09f8aaee4b249ae3ce924a422a67, 9564311af29a4493831a8db658a085c4; no voters:
I20240503 02:00:42.773929 22064 raft_consensus.cc:2781] T 885fe0c3a5e04f408e5b02c3c133248a P 9564311af29a4493831a8db658a085c4 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:00:42.774133 22064 raft_consensus.cc:686] T 885fe0c3a5e04f408e5b02c3c133248a P 9564311af29a4493831a8db658a085c4 [term 2 LEADER]: Becoming Leader. State: Replica: 9564311af29a4493831a8db658a085c4, State: Running, Role: LEADER
I20240503 02:00:42.774482 22064 consensus_queue.cc:237] T 885fe0c3a5e04f408e5b02c3c133248a P 9564311af29a4493831a8db658a085c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 2, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } }
I20240503 02:00:42.777789 17620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "885fe0c3a5e04f408e5b02c3c133248a" candidate_uuid: "9564311af29a4493831a8db658a085c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "c843193b801b4f1a88f630ed3ed0f375"
I20240503 02:00:42.777927 17620 raft_consensus.cc:3037] T 885fe0c3a5e04f408e5b02c3c133248a P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:42.777851 16815 catalog_manager.cc:5484] T 885fe0c3a5e04f408e5b02c3c133248a P 9564311af29a4493831a8db658a085c4 reported cstate change: term changed from 1 to 2, leader changed from 6398c8993c3c45dcaab25b42142d5671 (127.0.99.129) to 9564311af29a4493831a8db658a085c4 (127.0.99.138). New cstate: current_term: 2 leader_uuid: "9564311af29a4493831a8db658a085c4" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:42.778801 17620 raft_consensus.cc:2454] T 885fe0c3a5e04f408e5b02c3c133248a P c843193b801b4f1a88f630ed3ed0f375 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9564311af29a4493831a8db658a085c4 in term 2.
I20240503 02:00:42.779086 19368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "885fe0c3a5e04f408e5b02c3c133248a" candidate_uuid: "9564311af29a4493831a8db658a085c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d"
I20240503 02:00:42.779304 19368 raft_consensus.cc:3037] T 885fe0c3a5e04f408e5b02c3c133248a P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:42.780409 19368 raft_consensus.cc:2454] T 885fe0c3a5e04f408e5b02c3c133248a P 375d0f2ec11d4e25bd9d093bb6a5572d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9564311af29a4493831a8db658a085c4 in term 2.
I20240503 02:00:42.784356 17222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "885fe0c3a5e04f408e5b02c3c133248a" candidate_uuid: "9564311af29a4493831a8db658a085c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "5b8291b26a7f448b8636187d1caf86d0"
I20240503 02:00:42.784490 17222 raft_consensus.cc:3037] T 885fe0c3a5e04f408e5b02c3c133248a P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:42.785405 17222 raft_consensus.cc:2454] T 885fe0c3a5e04f408e5b02c3c133248a P 5b8291b26a7f448b8636187d1caf86d0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9564311af29a4493831a8db658a085c4 in term 2.
I20240503 02:00:43.159198 16958 raft_consensus.cc:1261] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671 [term 2 FOLLOWER]: Refusing update from remote peer 9564311af29a4493831a8db658a085c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 02:00:43.159747 22064 consensus_queue.cc:1035] T 885fe0c3a5e04f408e5b02c3c133248a P 9564311af29a4493831a8db658a085c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:00:43.200507 17220 raft_consensus.cc:1261] T 885fe0c3a5e04f408e5b02c3c133248a P 5b8291b26a7f448b8636187d1caf86d0 [term 2 FOLLOWER]: Refusing update from remote peer 9564311af29a4493831a8db658a085c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 02:00:43.201082 22064 consensus_queue.cc:1035] T 885fe0c3a5e04f408e5b02c3c133248a P 9564311af29a4493831a8db658a085c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:00:43.279361 17486 raft_consensus.cc:1261] T 885fe0c3a5e04f408e5b02c3c133248a P 16d437fe629643948c95d628926d5ffa [term 2 FOLLOWER]: Refusing update from remote peer 9564311af29a4493831a8db658a085c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 02:00:43.279803 22064 consensus_queue.cc:1035] T 885fe0c3a5e04f408e5b02c3c133248a P 9564311af29a4493831a8db658a085c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:00:43.296160 17752 raft_consensus.cc:1261] T 885fe0c3a5e04f408e5b02c3c133248a P 002cc3411acb4fdfa0b67c6f57f687a6 [term 2 FOLLOWER]: Refusing update from remote peer 9564311af29a4493831a8db658a085c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 02:00:43.296627 22064 consensus_queue.cc:1035] T 885fe0c3a5e04f408e5b02c3c133248a P 9564311af29a4493831a8db658a085c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:00:43.301010 19369 raft_consensus.cc:1261] T 885fe0c3a5e04f408e5b02c3c133248a P 375d0f2ec11d4e25bd9d093bb6a5572d [term 2 FOLLOWER]: Refusing update from remote peer 9564311af29a4493831a8db658a085c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 02:00:43.303023 17092 raft_consensus.cc:1261] T 885fe0c3a5e04f408e5b02c3c133248a P 89dd09f8aaee4b249ae3ce924a422a67 [term 2 FOLLOWER]: Refusing update from remote peer 9564311af29a4493831a8db658a085c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 02:00:43.303504 22120 consensus_queue.cc:1035] T 885fe0c3a5e04f408e5b02c3c133248a P 9564311af29a4493831a8db658a085c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:00:43.303978 17620 raft_consensus.cc:1261] T 885fe0c3a5e04f408e5b02c3c133248a P c843193b801b4f1a88f630ed3ed0f375 [term 2 FOLLOWER]: Refusing update from remote peer 9564311af29a4493831a8db658a085c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 02:00:43.304030 22122 consensus_queue.cc:1035] T 885fe0c3a5e04f408e5b02c3c133248a P 9564311af29a4493831a8db658a085c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:00:43.304536 22120 consensus_queue.cc:1035] T 885fe0c3a5e04f408e5b02c3c133248a P 9564311af29a4493831a8db658a085c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:00:43.903062 19237 consensus_queue.cc:237] T 885fe0c3a5e04f408e5b02c3c133248a P 9564311af29a4493831a8db658a085c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 9, Current term: 2, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } }
I20240503 02:00:43.904946 17752 raft_consensus.cc:1261] T 885fe0c3a5e04f408e5b02c3c133248a P 002cc3411acb4fdfa0b67c6f57f687a6 [term 2 FOLLOWER]: Refusing update from remote peer 9564311af29a4493831a8db658a085c4: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240503 02:00:43.904989 19369 raft_consensus.cc:1261] T 885fe0c3a5e04f408e5b02c3c133248a P 375d0f2ec11d4e25bd9d093bb6a5572d [term 2 FOLLOWER]: Refusing update from remote peer 9564311af29a4493831a8db658a085c4: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240503 02:00:43.905777 22118 consensus_queue.cc:1035] T 885fe0c3a5e04f408e5b02c3c133248a P 9564311af29a4493831a8db658a085c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:00:43.905932 17092 raft_consensus.cc:1261] T 885fe0c3a5e04f408e5b02c3c133248a P 89dd09f8aaee4b249ae3ce924a422a67 [term 2 FOLLOWER]: Refusing update from remote peer 9564311af29a4493831a8db658a085c4: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240503 02:00:43.905974 17620 raft_consensus.cc:1261] T 885fe0c3a5e04f408e5b02c3c133248a P c843193b801b4f1a88f630ed3ed0f375 [term 2 FOLLOWER]: Refusing update from remote peer 9564311af29a4493831a8db658a085c4: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240503 02:00:43.906194 22118 consensus_queue.cc:1035] T 885fe0c3a5e04f408e5b02c3c133248a P 9564311af29a4493831a8db658a085c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:00:43.906421 17220 raft_consensus.cc:1261] T 885fe0c3a5e04f408e5b02c3c133248a P 5b8291b26a7f448b8636187d1caf86d0 [term 2 FOLLOWER]: Refusing update from remote peer 9564311af29a4493831a8db658a085c4: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240503 02:00:43.906512 22118 consensus_queue.cc:1035] T 885fe0c3a5e04f408e5b02c3c133248a P 9564311af29a4493831a8db658a085c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:00:43.906991 17486 raft_consensus.cc:1261] T 885fe0c3a5e04f408e5b02c3c133248a P 16d437fe629643948c95d628926d5ffa [term 2 FOLLOWER]: Refusing update from remote peer 9564311af29a4493831a8db658a085c4: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240503 02:00:43.907446 22118 consensus_queue.cc:1035] T 885fe0c3a5e04f408e5b02c3c133248a P 9564311af29a4493831a8db658a085c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:00:43.908412 22120 consensus_queue.cc:1035] T 885fe0c3a5e04f408e5b02c3c133248a P 9564311af29a4493831a8db658a085c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:00:43.908560 22122 consensus_queue.cc:1035] T 885fe0c3a5e04f408e5b02c3c133248a P 9564311af29a4493831a8db658a085c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:00:43.910152 22118 raft_consensus.cc:2932] T 885fe0c3a5e04f408e5b02c3c133248a P 9564311af29a4493831a8db658a085c4 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER 6398c8993c3c45dcaab25b42142d5671 (127.0.99.129) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:43.911787 16805 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 885fe0c3a5e04f408e5b02c3c133248a with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:00:43.912407 19369 raft_consensus.cc:2932] T 885fe0c3a5e04f408e5b02c3c133248a P 375d0f2ec11d4e25bd9d093bb6a5572d [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER 6398c8993c3c45dcaab25b42142d5671 (127.0.99.129) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:43.913024 17752 raft_consensus.cc:2932] T 885fe0c3a5e04f408e5b02c3c133248a P 002cc3411acb4fdfa0b67c6f57f687a6 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER 6398c8993c3c45dcaab25b42142d5671 (127.0.99.129) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:43.913403 17220 raft_consensus.cc:2932] T 885fe0c3a5e04f408e5b02c3c133248a P 5b8291b26a7f448b8636187d1caf86d0 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER 6398c8993c3c45dcaab25b42142d5671 (127.0.99.129) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:43.913457 17486 raft_consensus.cc:2932] T 885fe0c3a5e04f408e5b02c3c133248a P 16d437fe629643948c95d628926d5ffa [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER 6398c8993c3c45dcaab25b42142d5671 (127.0.99.129) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:43.915987 17620 raft_consensus.cc:2932] T 885fe0c3a5e04f408e5b02c3c133248a P c843193b801b4f1a88f630ed3ed0f375 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER 6398c8993c3c45dcaab25b42142d5671 (127.0.99.129) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:43.917941 16812 catalog_manager.cc:5484] T 885fe0c3a5e04f408e5b02c3c133248a P 16d437fe629643948c95d628926d5ffa reported cstate change: config changed from index 9 to 11, VOTER 6398c8993c3c45dcaab25b42142d5671 (127.0.99.129) evicted. New cstate: current_term: 2 leader_uuid: "9564311af29a4493831a8db658a085c4" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:43.918721 17092 raft_consensus.cc:2932] T 885fe0c3a5e04f408e5b02c3c133248a P 89dd09f8aaee4b249ae3ce924a422a67 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER 6398c8993c3c45dcaab25b42142d5671 (127.0.99.129) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } }
I20240503 02:00:43.923017 16938 tablet_service.cc:1514] Processing DeleteTablet for tablet 885fe0c3a5e04f408e5b02c3c133248a with delete_type TABLET_DATA_TOMBSTONED (TS 6398c8993c3c45dcaab25b42142d5671 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:47986
I20240503 02:00:43.923460 22162 tablet_replica.cc:330] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671: stopping tablet replica
I20240503 02:00:43.923643 22162 raft_consensus.cc:2229] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:43.923825 22162 raft_consensus.cc:2258] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:43.925282 22162 ts_tablet_manager.cc:1875] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:43.927693 22162 ts_tablet_manager.cc:1888] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.10
I20240503 02:00:43.927772 22162 log.cc:1196] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671: Deleting WAL directory at /tmp/dist-test-task42rE7p/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714701523524339-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/885fe0c3a5e04f408e5b02c3c133248a
I20240503 02:00:43.928371 16804 catalog_manager.cc:4830] TS 6398c8993c3c45dcaab25b42142d5671 (127.0.99.129:36983): tablet 885fe0c3a5e04f408e5b02c3c133248a (table rebalance_test_table_1 [id=38eeb06923e14f2ead405ffb039fd231]) successfully deleted
I20240503 02:00:44.175316 17467 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:00:44.175518 19350 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:44.181973 19088 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:44.185225 16938 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:44.185546 17070 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:00:44.192327 17335 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:00:44.201223 17730 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:00:44.207129 19219 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:44.212838 17598 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:44.223611 17202 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:44.351285 17730 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:00:44.353392 17202 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:44.358561 17467 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:00:44.362635 17070 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:00:44.362884 19219 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:44.384675 16938 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:44.396189 19350 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:44.403182 17335 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:00:44.417749 17598 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:44.418843 19088 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:44.494251 19237 consensus_queue.cc:237] T 885fe0c3a5e04f408e5b02c3c133248a P 9564311af29a4493831a8db658a085c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 9, Current term: 2, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } attrs { replace: true } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { promote: true } }
I20240503 02:00:44.494699 17752 consensus_queue.cc:237] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false replace: true } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } }
I20240503 02:00:44.495990 19369 raft_consensus.cc:1261] T 885fe0c3a5e04f408e5b02c3c133248a P 375d0f2ec11d4e25bd9d093bb6a5572d [term 2 FOLLOWER]: Refusing update from remote peer 9564311af29a4493831a8db658a085c4: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240503 02:00:44.496569 22154 consensus_queue.cc:1035] T 885fe0c3a5e04f408e5b02c3c133248a P 9564311af29a4493831a8db658a085c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:00:44.498195 17089 raft_consensus.cc:1261] T fdf0a4b9e17642849a4094d2f0b9a255 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:44.498814 19369 raft_consensus.cc:1261] T fdf0a4b9e17642849a4094d2f0b9a255 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:44.498806 17486 consensus_queue.cc:237] T 42a900125241432a8bb31dad0b940968 P 16d437fe629643948c95d628926d5ffa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } attrs { replace: true } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { promote: true } }
I20240503 02:00:44.499274 22141 consensus_queue.cc:1035] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:44.499639 22141 consensus_queue.cc:1035] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240503 02:00:44.499749 19174 consensus_peers.cc:485] T 885fe0c3a5e04f408e5b02c3c133248a P 9564311af29a4493831a8db658a085c4 -> Peer 6398c8993c3c45dcaab25b42142d5671 (127.0.99.129:36983): Couldn't send request to peer 6398c8993c3c45dcaab25b42142d5671. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:44.500047 16958 raft_consensus.cc:1261] T fdf0a4b9e17642849a4094d2f0b9a255 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:44.501538 17092 raft_consensus.cc:1261] T 42a900125241432a8bb31dad0b940968 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:44.501935 17220 raft_consensus.cc:1261] T fdf0a4b9e17642849a4094d2f0b9a255 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:44.502043 22295 consensus_queue.cc:1035] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:44.502143 17222 raft_consensus.cc:1261] T 885fe0c3a5e04f408e5b02c3c133248a P 5b8291b26a7f448b8636187d1caf86d0 [term 2 FOLLOWER]: Refusing update from remote peer 9564311af29a4493831a8db658a085c4: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
W20240503 02:00:44.502393 17686 consensus_peers.cc:485] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fdf0a4b9e17642849a4094d2f0b9a255. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:44.502504 17752 raft_consensus.cc:1261] T 42a900125241432a8bb31dad0b940968 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:44.502928 19368 raft_consensus.cc:1261] T 42a900125241432a8bb31dad0b940968 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:44.503403 22295 consensus_queue.cc:1035] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:44.503485 17092 raft_consensus.cc:1261] T 885fe0c3a5e04f408e5b02c3c133248a P 89dd09f8aaee4b249ae3ce924a422a67 [term 2 FOLLOWER]: Refusing update from remote peer 9564311af29a4493831a8db658a085c4: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240503 02:00:44.503541 17222 raft_consensus.cc:1261] T 42a900125241432a8bb31dad0b940968 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:44.503857 17620 raft_consensus.cc:1261] T 42a900125241432a8bb31dad0b940968 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:44.503697 22154 consensus_queue.cc:1035] T 885fe0c3a5e04f408e5b02c3c133248a P 9564311af29a4493831a8db658a085c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:00:44.503857 17619 raft_consensus.cc:1261] T 885fe0c3a5e04f408e5b02c3c133248a P c843193b801b4f1a88f630ed3ed0f375 [term 2 FOLLOWER]: Refusing update from remote peer 9564311af29a4493831a8db658a085c4: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240503 02:00:44.504117 22301 consensus_queue.cc:1035] T 42a900125241432a8bb31dad0b940968 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:44.504154 22289 consensus_queue.cc:1035] T 885fe0c3a5e04f408e5b02c3c133248a P 9564311af29a4493831a8db658a085c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:00:44.504674 22272 consensus_queue.cc:1035] T 42a900125241432a8bb31dad0b940968 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:44.504884 22154 consensus_queue.cc:1035] T 885fe0c3a5e04f408e5b02c3c133248a P 9564311af29a4493831a8db658a085c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
W20240503 02:00:44.505336 17422 consensus_peers.cc:485] T 42a900125241432a8bb31dad0b940968 P 16d437fe629643948c95d628926d5ffa -> Peer d43ed6b8663947d99a72cd922cc323aa (127.0.99.132:40023): Couldn't send request to peer d43ed6b8663947d99a72cd922cc323aa. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:44.507081 17752 raft_consensus.cc:1261] T 885fe0c3a5e04f408e5b02c3c133248a P 002cc3411acb4fdfa0b67c6f57f687a6 [term 2 FOLLOWER]: Refusing update from remote peer 9564311af29a4493831a8db658a085c4: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240503 02:00:44.507257 17487 raft_consensus.cc:1261] T fdf0a4b9e17642849a4094d2f0b9a255 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:44.507709 17618 raft_consensus.cc:1261] T fdf0a4b9e17642849a4094d2f0b9a255 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:44.507738 16957 raft_consensus.cc:1261] T 42a900125241432a8bb31dad0b940968 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:44.508450 22288 consensus_queue.cc:1035] T 885fe0c3a5e04f408e5b02c3c133248a P 9564311af29a4493831a8db658a085c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:00:44.508350 22154 raft_consensus.cc:2932] T 885fe0c3a5e04f408e5b02c3c133248a P 9564311af29a4493831a8db658a085c4 [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER 6398c8993c3c45dcaab25b42142d5671 (127.0.99.129) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } attrs { replace: true } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { promote: true } } }
I20240503 02:00:44.509044 17618 raft_consensus.cc:2932] T 885fe0c3a5e04f408e5b02c3c133248a P c843193b801b4f1a88f630ed3ed0f375 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER 6398c8993c3c45dcaab25b42142d5671 (127.0.99.129) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } attrs { replace: true } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { promote: true } } }
I20240503 02:00:44.509243 22294 consensus_queue.cc:1035] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:44.509585 22294 consensus_queue.cc:1035] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:44.505517 22301 consensus_queue.cc:1035] T 42a900125241432a8bb31dad0b940968 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.005s
I20240503 02:00:44.510520 22301 consensus_queue.cc:1035] T 42a900125241432a8bb31dad0b940968 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:44.510969 22272 consensus_queue.cc:1035] T 42a900125241432a8bb31dad0b940968 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:44.511406 17091 raft_consensus.cc:2932] T 885fe0c3a5e04f408e5b02c3c133248a P 89dd09f8aaee4b249ae3ce924a422a67 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER 6398c8993c3c45dcaab25b42142d5671 (127.0.99.129) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } attrs { replace: true } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { promote: true } } }
I20240503 02:00:44.511977 19369 raft_consensus.cc:2932] T 885fe0c3a5e04f408e5b02c3c133248a P 375d0f2ec11d4e25bd9d093bb6a5572d [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER 6398c8993c3c45dcaab25b42142d5671 (127.0.99.129) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } attrs { replace: true } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { promote: true } } }
I20240503 02:00:44.513541 22141 raft_consensus.cc:2932] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false replace: true } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } }
I20240503 02:00:44.514006 16818 catalog_manager.cc:5484] T 885fe0c3a5e04f408e5b02c3c133248a P c843193b801b4f1a88f630ed3ed0f375 reported cstate change: config changed from index 11 to 12, NON_VOTER 6398c8993c3c45dcaab25b42142d5671 (127.0.99.129) added. New cstate: current_term: 2 leader_uuid: "9564311af29a4493831a8db658a085c4" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } attrs { replace: true } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { promote: true } } }
I20240503 02:00:44.514802 17220 raft_consensus.cc:2932] T 885fe0c3a5e04f408e5b02c3c133248a P 5b8291b26a7f448b8636187d1caf86d0 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER 6398c8993c3c45dcaab25b42142d5671 (127.0.99.129) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } attrs { replace: true } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { promote: true } } }
I20240503 02:00:44.515403 17751 raft_consensus.cc:2932] T 885fe0c3a5e04f408e5b02c3c133248a P 002cc3411acb4fdfa0b67c6f57f687a6 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER 6398c8993c3c45dcaab25b42142d5671 (127.0.99.129) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } attrs { replace: true } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { promote: true } } }
I20240503 02:00:44.516347 16957 raft_consensus.cc:2932] T fdf0a4b9e17642849a4094d2f0b9a255 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false replace: true } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } }
I20240503 02:00:44.518975 17486 raft_consensus.cc:1261] T 885fe0c3a5e04f408e5b02c3c133248a P 16d437fe629643948c95d628926d5ffa [term 2 FOLLOWER]: Refusing update from remote peer 9564311af29a4493831a8db658a085c4: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240503 02:00:44.521227 17220 raft_consensus.cc:2932] T fdf0a4b9e17642849a4094d2f0b9a255 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false replace: true } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } }
I20240503 02:00:44.521801 17089 raft_consensus.cc:2932] T fdf0a4b9e17642849a4094d2f0b9a255 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false replace: true } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } }
I20240503 02:00:44.523558 19369 raft_consensus.cc:2932] T fdf0a4b9e17642849a4094d2f0b9a255 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false replace: true } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } }
I20240503 02:00:44.523910 16816 catalog_manager.cc:5484] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 reported cstate change: config changed from index 7 to 8, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New cstate: current_term: 1 leader_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:44.524555 22272 raft_consensus.cc:2932] T 42a900125241432a8bb31dad0b940968 P 16d437fe629643948c95d628926d5ffa [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } attrs { replace: true } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { promote: true } } }
I20240503 02:00:44.525959 22288 consensus_queue.cc:1035] T 885fe0c3a5e04f408e5b02c3c133248a P 9564311af29a4493831a8db658a085c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:00:44.527215 17486 raft_consensus.cc:2932] T 885fe0c3a5e04f408e5b02c3c133248a P 16d437fe629643948c95d628926d5ffa [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER 6398c8993c3c45dcaab25b42142d5671 (127.0.99.129) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } attrs { replace: true } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { promote: true } } }
I20240503 02:00:44.528237 16958 raft_consensus.cc:2932] T 42a900125241432a8bb31dad0b940968 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } attrs { replace: true } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { promote: true } } }
I20240503 02:00:44.528304 17089 raft_consensus.cc:2932] T 42a900125241432a8bb31dad0b940968 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } attrs { replace: true } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { promote: true } } }
I20240503 02:00:44.529515 22300 consensus_queue.cc:1035] T 42a900125241432a8bb31dad0b940968 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:44.530597 17619 raft_consensus.cc:2932] T 42a900125241432a8bb31dad0b940968 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } attrs { replace: true } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { promote: true } } }
I20240503 02:00:44.530812 17752 raft_consensus.cc:2932] T 42a900125241432a8bb31dad0b940968 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } attrs { replace: true } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { promote: true } } }
I20240503 02:00:44.533749 19369 raft_consensus.cc:2932] T 42a900125241432a8bb31dad0b940968 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } attrs { replace: true } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { promote: true } } }
I20240503 02:00:44.535677 16814 catalog_manager.cc:5484] T 42a900125241432a8bb31dad0b940968 P 002cc3411acb4fdfa0b67c6f57f687a6 reported cstate change: config changed from index 7 to 8, NON_VOTER d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "16d437fe629643948c95d628926d5ffa" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } attrs { replace: true } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { promote: true } } }
I20240503 02:00:44.538563 17222 raft_consensus.cc:2932] T 42a900125241432a8bb31dad0b940968 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } attrs { replace: true } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { promote: true } } }
I20240503 02:00:44.547405 17089 consensus_queue.cc:237] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false replace: true } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } }
I20240503 02:00:44.552428 17356 raft_consensus.cc:1261] T 30abfada8ab44449b9a4237c57093b73 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:44.552732 17486 raft_consensus.cc:1261] T 30abfada8ab44449b9a4237c57093b73 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:44.552845 17752 raft_consensus.cc:1261] T 30abfada8ab44449b9a4237c57093b73 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:44.553040 16958 raft_consensus.cc:1261] T 30abfada8ab44449b9a4237c57093b73 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:44.553179 21921 consensus_queue.cc:1035] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:44.553567 21921 consensus_queue.cc:1035] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:44.553900 22271 consensus_queue.cc:1035] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240503 02:00:44.554129 17026 consensus_peers.cc:485] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 30abfada8ab44449b9a4237c57093b73. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:44.554229 21921 consensus_queue.cc:1035] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:44.554315 17222 raft_consensus.cc:1261] T 30abfada8ab44449b9a4237c57093b73 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:44.554795 19369 raft_consensus.cc:1261] T 30abfada8ab44449b9a4237c57093b73 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:44.555295 22343 consensus_queue.cc:1035] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:44.555485 22345 consensus_queue.cc:1035] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:44.559664 17487 raft_consensus.cc:2932] T fdf0a4b9e17642849a4094d2f0b9a255 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false replace: true } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } }
I20240503 02:00:44.561393 17620 raft_consensus.cc:2932] T fdf0a4b9e17642849a4094d2f0b9a255 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false replace: true } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } }
I20240503 02:00:44.562973 22344 raft_consensus.cc:2932] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false replace: true } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } }
I20240503 02:00:44.567054 19369 raft_consensus.cc:2932] T 30abfada8ab44449b9a4237c57093b73 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false replace: true } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } }
I20240503 02:00:44.569092 17222 raft_consensus.cc:2932] T 30abfada8ab44449b9a4237c57093b73 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false replace: true } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } }
I20240503 02:00:44.569799 17353 consensus_queue.cc:237] T a39f284167e942188c181147e8a9e74e P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } attrs { replace: true } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { promote: true } }
I20240503 02:00:44.569895 16813 catalog_manager.cc:5484] T 30abfada8ab44449b9a4237c57093b73 P 375d0f2ec11d4e25bd9d093bb6a5572d reported cstate change: config changed from index 7 to 8, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New cstate: current_term: 1 leader_uuid: "89dd09f8aaee4b249ae3ce924a422a67" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false replace: true } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } }
I20240503 02:00:44.572441 16958 raft_consensus.cc:2932] T 30abfada8ab44449b9a4237c57093b73 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false replace: true } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } }
I20240503 02:00:44.572692 17089 raft_consensus.cc:1261] T a39f284167e942188c181147e8a9e74e P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:44.573542 22147 consensus_queue.cc:1035] T a39f284167e942188c181147e8a9e74e P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:44.573894 16957 raft_consensus.cc:1261] T a39f284167e942188c181147e8a9e74e P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:44.574151 17752 raft_consensus.cc:1261] T a39f284167e942188c181147e8a9e74e P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:44.575320 17620 raft_consensus.cc:1261] T a39f284167e942188c181147e8a9e74e P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:44.575423 17486 raft_consensus.cc:2932] T 30abfada8ab44449b9a4237c57093b73 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false replace: true } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } }
I20240503 02:00:44.575930 19108 raft_consensus.cc:1261] T a39f284167e942188c181147e8a9e74e P 838ef968bb45484bb27f5af6f3989bae [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:44.576295 17222 raft_consensus.cc:1261] T a39f284167e942188c181147e8a9e74e P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:44.576370 22240 consensus_queue.cc:1035] T a39f284167e942188c181147e8a9e74e P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:44.576772 22240 consensus_queue.cc:1035] T a39f284167e942188c181147e8a9e74e P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:44.577121 22240 consensus_queue.cc:1035] T a39f284167e942188c181147e8a9e74e P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:44.578382 22358 consensus_queue.cc:1035] T a39f284167e942188c181147e8a9e74e P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:44.578747 22358 consensus_queue.cc:1035] T a39f284167e942188c181147e8a9e74e P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240503 02:00:44.579249 17289 consensus_peers.cc:485] T a39f284167e942188c181147e8a9e74e P d43ed6b8663947d99a72cd922cc323aa -> Peer 16d437fe629643948c95d628926d5ffa (127.0.99.133:46581): Couldn't send request to peer 16d437fe629643948c95d628926d5ffa. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:44.583832 17751 raft_consensus.cc:2932] T 30abfada8ab44449b9a4237c57093b73 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false replace: true } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } }
I20240503 02:00:44.584046 17353 consensus_queue.cc:237] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { promote: true } }
I20240503 02:00:44.584368 22240 raft_consensus.cc:2932] T a39f284167e942188c181147e8a9e74e P d43ed6b8663947d99a72cd922cc323aa [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 16d437fe629643948c95d628926d5ffa (127.0.99.133) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } attrs { replace: true } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { promote: true } } }
I20240503 02:00:44.585289 17620 raft_consensus.cc:2932] T a39f284167e942188c181147e8a9e74e P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 16d437fe629643948c95d628926d5ffa (127.0.99.133) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } attrs { replace: true } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { promote: true } } }
I20240503 02:00:44.586483 19237 raft_consensus.cc:1261] T 63962b169caf4f14888871e246c00075 P 9564311af29a4493831a8db658a085c4 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:00:44.587034 22147 consensus_queue.cc:1035] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:00:44.588496 19108 raft_consensus.cc:2932] T a39f284167e942188c181147e8a9e74e P 838ef968bb45484bb27f5af6f3989bae [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 16d437fe629643948c95d628926d5ffa (127.0.99.133) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } attrs { replace: true } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { promote: true } } }
I20240503 02:00:44.589195 19369 raft_consensus.cc:1261] T 63962b169caf4f14888871e246c00075 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:00:44.589102 16957 raft_consensus.cc:2932] T a39f284167e942188c181147e8a9e74e P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 16d437fe629643948c95d628926d5ffa (127.0.99.133) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } attrs { replace: true } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { promote: true } } }
I20240503 02:00:44.590685 16813 catalog_manager.cc:5484] T a39f284167e942188c181147e8a9e74e P c843193b801b4f1a88f630ed3ed0f375 reported cstate change: config changed from index 7 to 8, NON_VOTER 16d437fe629643948c95d628926d5ffa (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "d43ed6b8663947d99a72cd922cc323aa" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } attrs { replace: true } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { promote: true } } }
I20240503 02:00:44.591845 17751 raft_consensus.cc:2932] T a39f284167e942188c181147e8a9e74e P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 16d437fe629643948c95d628926d5ffa (127.0.99.133) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } attrs { replace: true } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { promote: true } } }
I20240503 02:00:44.592095 17487 raft_consensus.cc:1261] T 63962b169caf4f14888871e246c00075 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:00:44.592501 22240 consensus_queue.cc:1035] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:00:44.593382 17752 raft_consensus.cc:1261] T 63962b169caf4f14888871e246c00075 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:00:44.593866 22359 consensus_queue.cc:1035] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240503 02:00:44.594761 17290 consensus_peers.cc:485] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa -> Peer 5b8291b26a7f448b8636187d1caf86d0 (127.0.99.131:44479): Couldn't send request to peer 5b8291b26a7f448b8636187d1caf86d0. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:44.595026 22359 consensus_queue.cc:1035] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:00:44.595703 16957 raft_consensus.cc:1261] T 63962b169caf4f14888871e246c00075 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:00:44.596280 17092 raft_consensus.cc:2932] T a39f284167e942188c181147e8a9e74e P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 16d437fe629643948c95d628926d5ffa (127.0.99.133) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } attrs { replace: true } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { promote: true } } }
I20240503 02:00:44.596726 22240 consensus_queue.cc:1035] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:00:44.596305 17089 raft_consensus.cc:1261] T 63962b169caf4f14888871e246c00075 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:00:44.598119 22356 consensus_queue.cc:1035] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:00:44.598570 17353 raft_consensus.cc:2932] T 30abfada8ab44449b9a4237c57093b73 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 838ef968bb45484bb27f5af6f3989bae (127.0.99.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false replace: true } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } }
I20240503 02:00:44.606851 22359 raft_consensus.cc:2932] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 5b8291b26a7f448b8636187d1caf86d0 (127.0.99.131) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { promote: true } } }
I20240503 02:00:44.606915 17222 raft_consensus.cc:2932] T a39f284167e942188c181147e8a9e74e P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 16d437fe629643948c95d628926d5ffa (127.0.99.133) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } attrs { replace: true } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { promote: true } } }
I20240503 02:00:44.608479 17092 consensus_queue.cc:237] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } attrs { replace: true } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false replace: true } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { promote: true } }
I20240503 02:00:44.610020 19237 raft_consensus.cc:2932] T 63962b169caf4f14888871e246c00075 P 9564311af29a4493831a8db658a085c4 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 5b8291b26a7f448b8636187d1caf86d0 (127.0.99.131) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { promote: true } } }
I20240503 02:00:44.611618 17752 raft_consensus.cc:1261] T 30abfada8ab44449b9a4237c57093b73 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:44.611793 16958 raft_consensus.cc:2932] T 63962b169caf4f14888871e246c00075 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 5b8291b26a7f448b8636187d1caf86d0 (127.0.99.131) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { promote: true } } }
W20240503 02:00:44.612111 17025 consensus_peers.cc:485] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134:44343): Couldn't send request to peer c843193b801b4f1a88f630ed3ed0f375. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:44.612324 16957 raft_consensus.cc:1261] T 30abfada8ab44449b9a4237c57093b73 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:44.612301 21921 consensus_queue.cc:1035] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240503 02:00:44.612699 17026 consensus_peers.cc:485] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 30abfada8ab44449b9a4237c57093b73. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:44.613989 17486 consensus_queue.cc:237] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false replace: true } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } }
I20240503 02:00:44.615345 16816 catalog_manager.cc:5484] T 63962b169caf4f14888871e246c00075 P 9564311af29a4493831a8db658a085c4 reported cstate change: config changed from index 10 to 11, NON_VOTER 5b8291b26a7f448b8636187d1caf86d0 (127.0.99.131) added. New cstate: current_term: 1 leader_uuid: "d43ed6b8663947d99a72cd922cc323aa" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { promote: true } } }
I20240503 02:00:44.616844 17092 raft_consensus.cc:1261] T 3f67591d6a3744ab90a3e4d2375a59db P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:44.617172 19369 raft_consensus.cc:1261] T 30abfada8ab44449b9a4237c57093b73 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:44.617558 19108 raft_consensus.cc:1261] T 3f67591d6a3744ab90a3e4d2375a59db P 838ef968bb45484bb27f5af6f3989bae [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:44.617980 22344 consensus_queue.cc:1035] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:44.618288 22344 consensus_queue.cc:1035] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:44.618785 22272 consensus_queue.cc:1035] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:44.619467 22272 consensus_queue.cc:1035] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:44.619930 19368 raft_consensus.cc:2932] T 63962b169caf4f14888871e246c00075 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 5b8291b26a7f448b8636187d1caf86d0 (127.0.99.131) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { promote: true } } }
I20240503 02:00:44.621345 17220 raft_consensus.cc:1261] T 3f67591d6a3744ab90a3e4d2375a59db P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:44.621572 17356 raft_consensus.cc:1261] T 3f67591d6a3744ab90a3e4d2375a59db P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240503 02:00:44.621948 17422 consensus_peers.cc:485] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3f67591d6a3744ab90a3e4d2375a59db. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:44.622383 17751 raft_consensus.cc:1261] T 3f67591d6a3744ab90a3e4d2375a59db P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:44.624068 17486 raft_consensus.cc:2932] T 63962b169caf4f14888871e246c00075 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 5b8291b26a7f448b8636187d1caf86d0 (127.0.99.131) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { promote: true } } }
I20240503 02:00:44.624424 22299 consensus_queue.cc:1035] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:44.625296 22299 consensus_queue.cc:1035] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:44.625540 21887 consensus_queue.cc:1035] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:44.628808 21887 raft_consensus.cc:2932] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false replace: true } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:44.629019 22271 raft_consensus.cc:2932] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } attrs { replace: true } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false replace: true } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { promote: true } } }
I20240503 02:00:44.629182 17356 raft_consensus.cc:2932] T 3f67591d6a3744ab90a3e4d2375a59db P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false replace: true } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:44.631036 17353 raft_consensus.cc:1261] T 30abfada8ab44449b9a4237c57093b73 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:44.631561 19369 raft_consensus.cc:2932] T 30abfada8ab44449b9a4237c57093b73 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } attrs { replace: true } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false replace: true } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { promote: true } } }
I20240503 02:00:44.632443 17092 raft_consensus.cc:2932] T 3f67591d6a3744ab90a3e4d2375a59db P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false replace: true } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:44.633909 16957 raft_consensus.cc:2932] T 30abfada8ab44449b9a4237c57093b73 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } attrs { replace: true } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false replace: true } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { promote: true } } }
I20240503 02:00:44.635015 19108 raft_consensus.cc:2932] T 3f67591d6a3744ab90a3e4d2375a59db P 838ef968bb45484bb27f5af6f3989bae [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false replace: true } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:44.636926 22343 consensus_queue.cc:1035] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:44.636929 16812 catalog_manager.cc:5484] T 30abfada8ab44449b9a4237c57093b73 P 375d0f2ec11d4e25bd9d093bb6a5572d reported cstate change: config changed from index 8 to 9, NON_VOTER c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "89dd09f8aaee4b249ae3ce924a422a67" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } attrs { replace: true } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false replace: true } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { promote: true } } }
I20240503 02:00:44.637168 17752 raft_consensus.cc:2932] T 30abfada8ab44449b9a4237c57093b73 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } attrs { replace: true } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false replace: true } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { promote: true } } }
I20240503 02:00:44.639784 16814 catalog_manager.cc:5484] T 3f67591d6a3744ab90a3e4d2375a59db P 838ef968bb45484bb27f5af6f3989bae reported cstate change: config changed from index 7 to 8, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "16d437fe629643948c95d628926d5ffa" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false replace: true } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:44.640743 17092 raft_consensus.cc:2932] T 63962b169caf4f14888871e246c00075 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 5b8291b26a7f448b8636187d1caf86d0 (127.0.99.131) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { promote: true } } }
I20240503 02:00:44.640975 17353 raft_consensus.cc:2932] T 30abfada8ab44449b9a4237c57093b73 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } attrs { replace: true } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false replace: true } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { promote: true } } }
I20240503 02:00:44.642757 17487 raft_consensus.cc:1261] T 30abfada8ab44449b9a4237c57093b73 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:44.643270 22344 consensus_queue.cc:1035] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:44.644856 17487 raft_consensus.cc:2932] T 30abfada8ab44449b9a4237c57093b73 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } attrs { replace: true } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false replace: true } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { promote: true } } }
I20240503 02:00:44.645565 17222 raft_consensus.cc:1261] T 30abfada8ab44449b9a4237c57093b73 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:44.646057 16958 raft_consensus.cc:1261] T 3f67591d6a3744ab90a3e4d2375a59db P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:44.646706 22298 consensus_queue.cc:1035] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:44.647173 22344 consensus_queue.cc:1035] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:44.648741 17222 raft_consensus.cc:2932] T 30abfada8ab44449b9a4237c57093b73 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } attrs { replace: true } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false replace: true } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { promote: true } } }
I20240503 02:00:44.652787 16958 raft_consensus.cc:2932] T 3f67591d6a3744ab90a3e4d2375a59db P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false replace: true } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:44.655167 17751 raft_consensus.cc:2932] T 3f67591d6a3744ab90a3e4d2375a59db P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false replace: true } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:44.657001 17220 raft_consensus.cc:2932] T 3f67591d6a3744ab90a3e4d2375a59db P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false replace: true } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:44.658551 17092 consensus_queue.cc:237] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false replace: true } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } }
I20240503 02:00:44.661299 17353 raft_consensus.cc:1261] T 81b54f3bcf34485eb11b1bc954a2bd8f P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:44.661437 19108 raft_consensus.cc:1261] T 81b54f3bcf34485eb11b1bc954a2bd8f P 838ef968bb45484bb27f5af6f3989bae [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:44.661705 17751 raft_consensus.cc:1261] T 81b54f3bcf34485eb11b1bc954a2bd8f P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:44.661849 22341 consensus_queue.cc:1035] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:44.662101 22342 consensus_queue.cc:1035] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:44.662431 22344 consensus_queue.cc:1035] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:44.662712 17220 raft_consensus.cc:1261] T 81b54f3bcf34485eb11b1bc954a2bd8f P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:44.663715 22344 consensus_queue.cc:1035] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:44.664424 17487 raft_consensus.cc:1261] T 81b54f3bcf34485eb11b1bc954a2bd8f P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:44.666504 22341 raft_consensus.cc:2932] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false replace: true } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:44.667068 17220 raft_consensus.cc:2932] T 81b54f3bcf34485eb11b1bc954a2bd8f P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false replace: true } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:44.667353 22344 consensus_queue.cc:1035] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:44.668759 17487 raft_consensus.cc:2932] T 81b54f3bcf34485eb11b1bc954a2bd8f P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false replace: true } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:44.669268 17752 consensus_queue.cc:237] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false replace: true } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { promote: true } }
I20240503 02:00:44.670711 19108 raft_consensus.cc:2932] T 81b54f3bcf34485eb11b1bc954a2bd8f P 838ef968bb45484bb27f5af6f3989bae [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false replace: true } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:44.670755 17353 raft_consensus.cc:2932] T 81b54f3bcf34485eb11b1bc954a2bd8f P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false replace: true } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:44.671810 16816 catalog_manager.cc:5484] T 81b54f3bcf34485eb11b1bc954a2bd8f P 5b8291b26a7f448b8636187d1caf86d0 reported cstate change: config changed from index 7 to 8, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "89dd09f8aaee4b249ae3ce924a422a67" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false replace: true } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
W20240503 02:00:44.672017 17686 consensus_peers.cc:485] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer d43ed6b8663947d99a72cd922cc323aa (127.0.99.132:40023): Couldn't send request to peer d43ed6b8663947d99a72cd922cc323aa. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:44.672832 17686 consensus_peers.cc:485] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fdf0a4b9e17642849a4094d2f0b9a255. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:44.674010 17220 raft_consensus.cc:1261] T fdf0a4b9e17642849a4094d2f0b9a255 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:44.674399 22292 consensus_queue.cc:1035] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:44.674501 19369 raft_consensus.cc:1261] T fdf0a4b9e17642849a4094d2f0b9a255 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:44.675520 22292 consensus_queue.cc:1035] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240503 02:00:44.676206 17026 consensus_peers.cc:485] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 81b54f3bcf34485eb11b1bc954a2bd8f. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:44.676858 16958 raft_consensus.cc:1261] T 81b54f3bcf34485eb11b1bc954a2bd8f P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:44.678653 17751 raft_consensus.cc:2932] T 81b54f3bcf34485eb11b1bc954a2bd8f P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false replace: true } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:44.678851 17092 raft_consensus.cc:1261] T fdf0a4b9e17642849a4094d2f0b9a255 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:44.679047 22271 consensus_queue.cc:1035] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:44.679559 22295 consensus_queue.cc:1035] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:44.682201 22293 raft_consensus.cc:2932] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false replace: true } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { promote: true } } }
I20240503 02:00:44.682597 19368 raft_consensus.cc:2932] T fdf0a4b9e17642849a4094d2f0b9a255 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false replace: true } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { promote: true } } }
I20240503 02:00:44.686245 16817 catalog_manager.cc:5484] T fdf0a4b9e17642849a4094d2f0b9a255 P 375d0f2ec11d4e25bd9d093bb6a5572d reported cstate change: config changed from index 8 to 9, NON_VOTER d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false replace: true } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { promote: true } } }
I20240503 02:00:44.687780 16958 raft_consensus.cc:2932] T 81b54f3bcf34485eb11b1bc954a2bd8f P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false replace: true } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: true } } }
I20240503 02:00:44.690819 16957 raft_consensus.cc:1261] T fdf0a4b9e17642849a4094d2f0b9a255 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:44.691247 22292 consensus_queue.cc:1035] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:44.692163 16957 raft_consensus.cc:2932] T fdf0a4b9e17642849a4094d2f0b9a255 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false replace: true } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { promote: true } } }
I20240503 02:00:44.694254 17487 raft_consensus.cc:1261] T fdf0a4b9e17642849a4094d2f0b9a255 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:44.695122 22293 consensus_queue.cc:1035] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:44.696094 17220 raft_consensus.cc:2932] T fdf0a4b9e17642849a4094d2f0b9a255 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false replace: true } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { promote: true } } }
I20240503 02:00:44.696187 17487 raft_consensus.cc:2932] T fdf0a4b9e17642849a4094d2f0b9a255 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false replace: true } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { promote: true } } }
I20240503 02:00:44.698930 17620 raft_consensus.cc:1261] T fdf0a4b9e17642849a4094d2f0b9a255 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:44.699549 22295 consensus_queue.cc:1035] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:44.700466 17092 raft_consensus.cc:2932] T fdf0a4b9e17642849a4094d2f0b9a255 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false replace: true } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { promote: true } } }
I20240503 02:00:44.700587 17620 raft_consensus.cc:2932] T fdf0a4b9e17642849a4094d2f0b9a255 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false replace: true } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { promote: true } } }
I20240503 02:00:44.709946 17487 consensus_queue.cc:237] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { promote: true } }
I20240503 02:00:44.713390 19107 raft_consensus.cc:1261] T fbec07350ad04c49afd42e5d88dda6ad P 838ef968bb45484bb27f5af6f3989bae [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:00:44.713413 17353 raft_consensus.cc:1261] T fbec07350ad04c49afd42e5d88dda6ad P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:00:44.713629 16958 raft_consensus.cc:1261] T fbec07350ad04c49afd42e5d88dda6ad P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:00:44.713862 22272 consensus_queue.cc:1035] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:00:44.714118 22297 consensus_queue.cc:1035] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:00:44.714897 17752 raft_consensus.cc:1261] T fbec07350ad04c49afd42e5d88dda6ad P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:00:44.715288 22298 consensus_queue.cc:1035] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:00:44.715642 22298 consensus_queue.cc:1035] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240503 02:00:44.718938 17421 consensus_peers.cc:485] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa -> Peer c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134:44343): Couldn't send request to peer c843193b801b4f1a88f630ed3ed0f375. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:44.718962 17092 raft_consensus.cc:1261] T fbec07350ad04c49afd42e5d88dda6ad P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:00:44.719413 19237 raft_consensus.cc:1261] T fbec07350ad04c49afd42e5d88dda6ad P 9564311af29a4493831a8db658a085c4 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:00:44.719455 22298 consensus_queue.cc:1035] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:00:44.719480 21887 raft_consensus.cc:2932] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { promote: true } } }
I20240503 02:00:44.720054 22298 consensus_queue.cc:1035] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:00:44.720201 17752 raft_consensus.cc:2932] T fbec07350ad04c49afd42e5d88dda6ad P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { promote: true } } }
I20240503 02:00:44.720747 17092 raft_consensus.cc:2932] T fbec07350ad04c49afd42e5d88dda6ad P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { promote: true } } }
I20240503 02:00:44.721580 19237 raft_consensus.cc:2932] T fbec07350ad04c49afd42e5d88dda6ad P 9564311af29a4493831a8db658a085c4 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { promote: true } } }
I20240503 02:00:44.722522 19107 raft_consensus.cc:2932] T fbec07350ad04c49afd42e5d88dda6ad P 838ef968bb45484bb27f5af6f3989bae [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { promote: true } } }
I20240503 02:00:44.723655 16958 raft_consensus.cc:2932] T fbec07350ad04c49afd42e5d88dda6ad P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { promote: true } } }
I20240503 02:00:44.725432 16814 catalog_manager.cc:5484] T fbec07350ad04c49afd42e5d88dda6ad P 89dd09f8aaee4b249ae3ce924a422a67 reported cstate change: config changed from index 10 to 11, NON_VOTER c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "16d437fe629643948c95d628926d5ffa" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { promote: true } } }
I20240503 02:00:44.728209 17353 raft_consensus.cc:2932] T fbec07350ad04c49afd42e5d88dda6ad P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { promote: true } } }
I20240503 02:00:44.739933 17752 consensus_queue.cc:237] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { promote: true } }
I20240503 02:00:44.742414 17487 raft_consensus.cc:1261] T 78d106d705df4089a4dfae4d5dc533cc P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:00:44.742429 17620 raft_consensus.cc:1261] T 78d106d705df4089a4dfae4d5dc533cc P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:00:44.742558 19237 raft_consensus.cc:1261] T 78d106d705df4089a4dfae4d5dc533cc P 9564311af29a4493831a8db658a085c4 [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:00:44.742908 17092 raft_consensus.cc:1261] T 78d106d705df4089a4dfae4d5dc533cc P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:00:44.742861 22141 consensus_queue.cc:1035] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:00:44.742920 19107 raft_consensus.cc:1261] T 78d106d705df4089a4dfae4d5dc533cc P 838ef968bb45484bb27f5af6f3989bae [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:00:44.743026 22294 consensus_queue.cc:1035] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:00:44.743227 22430 consensus_queue.cc:1035] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:00:44.743477 17220 raft_consensus.cc:1261] T 78d106d705df4089a4dfae4d5dc533cc P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:00:44.743451 22291 consensus_queue.cc:1035] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240503 02:00:44.744203 17686 consensus_peers.cc:485] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer d43ed6b8663947d99a72cd922cc323aa (127.0.99.132:40023): Couldn't send request to peer d43ed6b8663947d99a72cd922cc323aa. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:44.744928 22294 consensus_queue.cc:1035] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:00:44.745282 22294 consensus_queue.cc:1035] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:00:44.750842 22291 raft_consensus.cc:2932] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { promote: true } } }
I20240503 02:00:44.751547 17092 raft_consensus.cc:2932] T 78d106d705df4089a4dfae4d5dc533cc P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { promote: true } } }
I20240503 02:00:44.754241 16816 catalog_manager.cc:5484] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 reported cstate change: config changed from index 10 to 11, NON_VOTER d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:44.755033 19237 raft_consensus.cc:2932] T 78d106d705df4089a4dfae4d5dc533cc P 9564311af29a4493831a8db658a085c4 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { promote: true } } }
I20240503 02:00:44.755440 17487 raft_consensus.cc:2932] T 78d106d705df4089a4dfae4d5dc533cc P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { promote: true } } }
I20240503 02:00:44.755452 17620 raft_consensus.cc:2932] T 78d106d705df4089a4dfae4d5dc533cc P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { promote: true } } }
I20240503 02:00:44.759649 17220 raft_consensus.cc:2932] T 78d106d705df4089a4dfae4d5dc533cc P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { promote: true } } }
I20240503 02:00:44.766286 19107 raft_consensus.cc:2932] T 78d106d705df4089a4dfae4d5dc533cc P 838ef968bb45484bb27f5af6f3989bae [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { promote: true } } }
I20240503 02:00:44.772392 17220 consensus_queue.cc:237] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } attrs { replace: true } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { promote: true } }
I20240503 02:00:44.775072 17752 raft_consensus.cc:1261] T 4782de9826d6460e9e993e01327034e4 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer 5b8291b26a7f448b8636187d1caf86d0: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:00:44.775494 22135 consensus_queue.cc:1035] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:00:44.775578 17620 raft_consensus.cc:1261] T 4782de9826d6460e9e993e01327034e4 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer 5b8291b26a7f448b8636187d1caf86d0: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:00:44.775754 19368 raft_consensus.cc:1261] T 4782de9826d6460e9e993e01327034e4 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 FOLLOWER]: Refusing update from remote peer 5b8291b26a7f448b8636187d1caf86d0: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:00:44.776028 19107 raft_consensus.cc:1261] T 4782de9826d6460e9e993e01327034e4 P 838ef968bb45484bb27f5af6f3989bae [term 1 FOLLOWER]: Refusing update from remote peer 5b8291b26a7f448b8636187d1caf86d0: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:00:44.776607 17487 raft_consensus.cc:1261] T 4782de9826d6460e9e993e01327034e4 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Refusing update from remote peer 5b8291b26a7f448b8636187d1caf86d0: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:00:44.776976 16958 raft_consensus.cc:1261] T 4782de9826d6460e9e993e01327034e4 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 5b8291b26a7f448b8636187d1caf86d0: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:00:44.777351 22135 consensus_queue.cc:1035] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240503 02:00:44.777765 17158 consensus_peers.cc:485] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 -> Peer d43ed6b8663947d99a72cd922cc323aa (127.0.99.132:40023): Couldn't send request to peer d43ed6b8663947d99a72cd922cc323aa. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:44.777762 22135 consensus_queue.cc:1035] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:00:44.778128 22135 consensus_queue.cc:1035] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:00:44.778465 22135 consensus_queue.cc:1035] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:00:44.778681 22474 consensus_queue.cc:1035] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:00:44.782090 22472 raft_consensus.cc:2932] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } attrs { replace: true } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { promote: true } } }
I20240503 02:00:44.782562 17487 raft_consensus.cc:2932] T 4782de9826d6460e9e993e01327034e4 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } attrs { replace: true } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { promote: true } } }
I20240503 02:00:44.784976 17752 raft_consensus.cc:2932] T 4782de9826d6460e9e993e01327034e4 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } attrs { replace: true } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { promote: true } } }
I20240503 02:00:44.785320 16958 raft_consensus.cc:2932] T 4782de9826d6460e9e993e01327034e4 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } attrs { replace: true } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { promote: true } } }
I20240503 02:00:44.787499 16816 catalog_manager.cc:5484] T 4782de9826d6460e9e993e01327034e4 P 16d437fe629643948c95d628926d5ffa reported cstate change: config changed from index 10 to 11, NON_VOTER d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "5b8291b26a7f448b8636187d1caf86d0" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } attrs { replace: true } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { promote: true } } }
I20240503 02:00:44.788290 19368 raft_consensus.cc:2932] T 4782de9826d6460e9e993e01327034e4 P 375d0f2ec11d4e25bd9d093bb6a5572d [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } attrs { replace: true } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { promote: true } } }
I20240503 02:00:44.794692 19107 raft_consensus.cc:2932] T 4782de9826d6460e9e993e01327034e4 P 838ef968bb45484bb27f5af6f3989bae [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } attrs { replace: true } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { promote: true } } }
I20240503 02:00:44.795333 17620 raft_consensus.cc:2932] T 4782de9826d6460e9e993e01327034e4 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } attrs { replace: true } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { promote: true } } }
I20240503 02:00:44.912791 22528 ts_tablet_manager.cc:902] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671: Initiating tablet copy from peer 9564311af29a4493831a8db658a085c4 (127.0.99.138:33181)
I20240503 02:00:44.913653 22528 tablet_copy_client.cc:308] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.138:33181
I20240503 02:00:44.914336 19259 tablet_copy_service.cc:140] P 9564311af29a4493831a8db658a085c4: Received BeginTabletCopySession request for tablet 885fe0c3a5e04f408e5b02c3c133248a from peer 6398c8993c3c45dcaab25b42142d5671 ({username='slave'} at 127.0.99.129:33959)
I20240503 02:00:44.914474 19259 tablet_copy_service.cc:161] P 9564311af29a4493831a8db658a085c4: Beginning new tablet copy session on tablet 885fe0c3a5e04f408e5b02c3c133248a from peer 6398c8993c3c45dcaab25b42142d5671 at {username='slave'} at 127.0.99.129:33959: session id = 6398c8993c3c45dcaab25b42142d5671-885fe0c3a5e04f408e5b02c3c133248a
I20240503 02:00:44.915763 22533 ts_tablet_manager.cc:902] T 42a900125241432a8bb31dad0b940968 P d43ed6b8663947d99a72cd922cc323aa: Initiating tablet copy from peer 16d437fe629643948c95d628926d5ffa (127.0.99.133:46581)
I20240503 02:00:44.915831 19259 tablet_copy_source_session.cc:205] T 885fe0c3a5e04f408e5b02c3c133248a P 9564311af29a4493831a8db658a085c4: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:44.916566 22533 tablet_copy_client.cc:308] T 42a900125241432a8bb31dad0b940968 P d43ed6b8663947d99a72cd922cc323aa: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.133:46581
I20240503 02:00:44.917186 22528 ts_tablet_manager.cc:1875] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 02:00:44.917212 17508 tablet_copy_service.cc:140] P 16d437fe629643948c95d628926d5ffa: Received BeginTabletCopySession request for tablet 42a900125241432a8bb31dad0b940968 from peer d43ed6b8663947d99a72cd922cc323aa ({username='slave'} at 127.0.99.132:55927)
I20240503 02:00:44.917335 17508 tablet_copy_service.cc:161] P 16d437fe629643948c95d628926d5ffa: Beginning new tablet copy session on tablet 42a900125241432a8bb31dad0b940968 from peer d43ed6b8663947d99a72cd922cc323aa at {username='slave'} at 127.0.99.132:55927: session id = d43ed6b8663947d99a72cd922cc323aa-42a900125241432a8bb31dad0b940968
I20240503 02:00:44.918470 17508 tablet_copy_source_session.cc:205] T 42a900125241432a8bb31dad0b940968 P 16d437fe629643948c95d628926d5ffa: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:44.919339 22528 ts_tablet_manager.cc:1888] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 2.10
I20240503 02:00:44.919489 22533 ts_tablet_manager.cc:1875] T 42a900125241432a8bb31dad0b940968 P d43ed6b8663947d99a72cd922cc323aa: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 02:00:44.919523 22528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 885fe0c3a5e04f408e5b02c3c133248a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:44.920701 22528 tablet_copy_client.cc:783] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:44.920881 22528 tablet_copy_client.cc:647] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:44.921944 22533 ts_tablet_manager.cc:1888] T 42a900125241432a8bb31dad0b940968 P d43ed6b8663947d99a72cd922cc323aa: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.6
I20240503 02:00:44.922099 22533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42a900125241432a8bb31dad0b940968. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:44.923172 22533 tablet_copy_client.cc:783] T 42a900125241432a8bb31dad0b940968 P d43ed6b8663947d99a72cd922cc323aa: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:44.923362 22533 tablet_copy_client.cc:647] T 42a900125241432a8bb31dad0b940968 P d43ed6b8663947d99a72cd922cc323aa: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:44.924798 22528 tablet_copy_client.cc:523] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:44.924834 22533 tablet_copy_client.cc:523] T 42a900125241432a8bb31dad0b940968 P d43ed6b8663947d99a72cd922cc323aa: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:44.926431 22528 tablet_bootstrap.cc:492] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671: Bootstrap starting.
I20240503 02:00:44.926702 22533 tablet_bootstrap.cc:492] T 42a900125241432a8bb31dad0b940968 P d43ed6b8663947d99a72cd922cc323aa: Bootstrap starting.
I20240503 02:00:44.955242 22533 tablet_bootstrap.cc:492] T 42a900125241432a8bb31dad0b940968 P d43ed6b8663947d99a72cd922cc323aa: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:44.955696 22533 tablet_bootstrap.cc:492] T 42a900125241432a8bb31dad0b940968 P d43ed6b8663947d99a72cd922cc323aa: Bootstrap complete.
I20240503 02:00:44.955899 22533 ts_tablet_manager.cc:1367] T 42a900125241432a8bb31dad0b940968 P d43ed6b8663947d99a72cd922cc323aa: Time spent bootstrapping tablet: real 0.029s user 0.028s sys 0.002s
I20240503 02:00:44.956429 22533 raft_consensus.cc:348] T 42a900125241432a8bb31dad0b940968 P d43ed6b8663947d99a72cd922cc323aa [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } attrs { replace: true } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { promote: true } }
I20240503 02:00:44.956595 22533 raft_consensus.cc:729] T 42a900125241432a8bb31dad0b940968 P d43ed6b8663947d99a72cd922cc323aa [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d43ed6b8663947d99a72cd922cc323aa, State: Initialized, Role: LEARNER
I20240503 02:00:44.956786 22533 consensus_queue.cc:260] T 42a900125241432a8bb31dad0b940968 P d43ed6b8663947d99a72cd922cc323aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } attrs { replace: true } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { promote: true } }
I20240503 02:00:44.957258 22533 ts_tablet_manager.cc:1398] T 42a900125241432a8bb31dad0b940968 P d43ed6b8663947d99a72cd922cc323aa: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:44.958164 17508 tablet_copy_service.cc:342] P 16d437fe629643948c95d628926d5ffa: Request end of tablet copy session d43ed6b8663947d99a72cd922cc323aa-42a900125241432a8bb31dad0b940968 received from {username='slave'} at 127.0.99.132:55927
I20240503 02:00:44.958287 22528 tablet_bootstrap.cc:492] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:44.958312 17508 tablet_copy_service.cc:433] P 16d437fe629643948c95d628926d5ffa: ending tablet copy session d43ed6b8663947d99a72cd922cc323aa-42a900125241432a8bb31dad0b940968 on tablet 42a900125241432a8bb31dad0b940968 with peer d43ed6b8663947d99a72cd922cc323aa
I20240503 02:00:44.958585 22528 tablet_bootstrap.cc:492] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671: Bootstrap complete.
I20240503 02:00:44.958752 22528 ts_tablet_manager.cc:1367] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671: Time spent bootstrapping tablet: real 0.032s user 0.022s sys 0.009s
I20240503 02:00:44.959245 22528 raft_consensus.cc:348] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } attrs { replace: true } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { promote: true } }
I20240503 02:00:44.959393 22528 raft_consensus.cc:729] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 6398c8993c3c45dcaab25b42142d5671, State: Initialized, Role: LEARNER
I20240503 02:00:44.959513 22528 consensus_queue.cc:260] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } attrs { replace: true } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { promote: true } }
I20240503 02:00:44.959990 22528 ts_tablet_manager.cc:1398] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240503 02:00:44.960618 19259 tablet_copy_service.cc:342] P 9564311af29a4493831a8db658a085c4: Request end of tablet copy session 6398c8993c3c45dcaab25b42142d5671-885fe0c3a5e04f408e5b02c3c133248a received from {username='slave'} at 127.0.99.129:33959
I20240503 02:00:44.960768 19259 tablet_copy_service.cc:433] P 9564311af29a4493831a8db658a085c4: ending tablet copy session 6398c8993c3c45dcaab25b42142d5671-885fe0c3a5e04f408e5b02c3c133248a on tablet 885fe0c3a5e04f408e5b02c3c133248a with peer 6398c8993c3c45dcaab25b42142d5671
I20240503 02:00:45.000128 22560 ts_tablet_manager.cc:902] T 30abfada8ab44449b9a4237c57093b73 P c843193b801b4f1a88f630ed3ed0f375: Initiating tablet copy from peer 89dd09f8aaee4b249ae3ce924a422a67 (127.0.99.130:38889)
I20240503 02:00:45.001062 22560 tablet_copy_client.cc:308] T 30abfada8ab44449b9a4237c57093b73 P c843193b801b4f1a88f630ed3ed0f375: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:38889
I20240503 02:00:45.001722 17111 tablet_copy_service.cc:140] P 89dd09f8aaee4b249ae3ce924a422a67: Received BeginTabletCopySession request for tablet 30abfada8ab44449b9a4237c57093b73 from peer c843193b801b4f1a88f630ed3ed0f375 ({username='slave'} at 127.0.99.134:42309)
I20240503 02:00:45.001825 17111 tablet_copy_service.cc:161] P 89dd09f8aaee4b249ae3ce924a422a67: Beginning new tablet copy session on tablet 30abfada8ab44449b9a4237c57093b73 from peer c843193b801b4f1a88f630ed3ed0f375 at {username='slave'} at 127.0.99.134:42309: session id = c843193b801b4f1a88f630ed3ed0f375-30abfada8ab44449b9a4237c57093b73
I20240503 02:00:45.002687 17111 tablet_copy_source_session.cc:205] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:45.003767 22560 ts_tablet_manager.cc:1875] T 30abfada8ab44449b9a4237c57093b73 P c843193b801b4f1a88f630ed3ed0f375: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 02:00:45.006376 22560 ts_tablet_manager.cc:1888] T 30abfada8ab44449b9a4237c57093b73 P c843193b801b4f1a88f630ed3ed0f375: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.6
I20240503 02:00:45.006531 22560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30abfada8ab44449b9a4237c57093b73. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:45.007470 22560 tablet_copy_client.cc:783] T 30abfada8ab44449b9a4237c57093b73 P c843193b801b4f1a88f630ed3ed0f375: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:45.007618 22560 tablet_copy_client.cc:647] T 30abfada8ab44449b9a4237c57093b73 P c843193b801b4f1a88f630ed3ed0f375: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:45.009122 22560 tablet_copy_client.cc:523] T 30abfada8ab44449b9a4237c57093b73 P c843193b801b4f1a88f630ed3ed0f375: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:45.011081 22560 tablet_bootstrap.cc:492] T 30abfada8ab44449b9a4237c57093b73 P c843193b801b4f1a88f630ed3ed0f375: Bootstrap starting.
I20240503 02:00:45.022023 22566 ts_tablet_manager.cc:902] T 30abfada8ab44449b9a4237c57093b73 P 838ef968bb45484bb27f5af6f3989bae: Initiating tablet copy from peer 89dd09f8aaee4b249ae3ce924a422a67 (127.0.99.130:38889)
I20240503 02:00:45.022616 22566 tablet_copy_client.cc:308] T 30abfada8ab44449b9a4237c57093b73 P 838ef968bb45484bb27f5af6f3989bae: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:38889
I20240503 02:00:45.023216 17111 tablet_copy_service.cc:140] P 89dd09f8aaee4b249ae3ce924a422a67: Received BeginTabletCopySession request for tablet 30abfada8ab44449b9a4237c57093b73 from peer 838ef968bb45484bb27f5af6f3989bae ({username='slave'} at 127.0.99.137:47267)
I20240503 02:00:45.023355 17111 tablet_copy_service.cc:161] P 89dd09f8aaee4b249ae3ce924a422a67: Beginning new tablet copy session on tablet 30abfada8ab44449b9a4237c57093b73 from peer 838ef968bb45484bb27f5af6f3989bae at {username='slave'} at 127.0.99.137:47267: session id = 838ef968bb45484bb27f5af6f3989bae-30abfada8ab44449b9a4237c57093b73
I20240503 02:00:45.024468 17111 tablet_copy_source_session.cc:205] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:45.025596 22566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30abfada8ab44449b9a4237c57093b73. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:45.028483 22566 tablet_copy_client.cc:783] T 30abfada8ab44449b9a4237c57093b73 P 838ef968bb45484bb27f5af6f3989bae: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:45.028761 22566 tablet_copy_client.cc:647] T 30abfada8ab44449b9a4237c57093b73 P 838ef968bb45484bb27f5af6f3989bae: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:45.031174 22566 tablet_copy_client.cc:523] T 30abfada8ab44449b9a4237c57093b73 P 838ef968bb45484bb27f5af6f3989bae: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:45.032575 22566 tablet_bootstrap.cc:492] T 30abfada8ab44449b9a4237c57093b73 P 838ef968bb45484bb27f5af6f3989bae: Bootstrap starting.
I20240503 02:00:45.038010 22560 tablet_bootstrap.cc:492] T 30abfada8ab44449b9a4237c57093b73 P c843193b801b4f1a88f630ed3ed0f375: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:45.038337 22560 tablet_bootstrap.cc:492] T 30abfada8ab44449b9a4237c57093b73 P c843193b801b4f1a88f630ed3ed0f375: Bootstrap complete.
I20240503 02:00:45.038518 22560 ts_tablet_manager.cc:1367] T 30abfada8ab44449b9a4237c57093b73 P c843193b801b4f1a88f630ed3ed0f375: Time spent bootstrapping tablet: real 0.028s user 0.019s sys 0.008s
I20240503 02:00:45.038993 22560 raft_consensus.cc:348] T 30abfada8ab44449b9a4237c57093b73 P c843193b801b4f1a88f630ed3ed0f375 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } attrs { replace: true } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false replace: true } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { promote: true } }
I20240503 02:00:45.039175 22560 raft_consensus.cc:729] T 30abfada8ab44449b9a4237c57093b73 P c843193b801b4f1a88f630ed3ed0f375 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c843193b801b4f1a88f630ed3ed0f375, State: Initialized, Role: LEARNER
I20240503 02:00:45.039291 22560 consensus_queue.cc:260] T 30abfada8ab44449b9a4237c57093b73 P c843193b801b4f1a88f630ed3ed0f375 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } attrs { replace: true } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false replace: true } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { promote: true } }
I20240503 02:00:45.039624 22560 ts_tablet_manager.cc:1398] T 30abfada8ab44449b9a4237c57093b73 P c843193b801b4f1a88f630ed3ed0f375: Time spent starting tablet: real 0.001s user 0.002s sys 0.001s
I20240503 02:00:45.040295 17111 tablet_copy_service.cc:342] P 89dd09f8aaee4b249ae3ce924a422a67: Request end of tablet copy session c843193b801b4f1a88f630ed3ed0f375-30abfada8ab44449b9a4237c57093b73 received from {username='slave'} at 127.0.99.134:42309
I20240503 02:00:45.040390 17111 tablet_copy_service.cc:433] P 89dd09f8aaee4b249ae3ce924a422a67: ending tablet copy session c843193b801b4f1a88f630ed3ed0f375-30abfada8ab44449b9a4237c57093b73 on tablet 30abfada8ab44449b9a4237c57093b73 with peer c843193b801b4f1a88f630ed3ed0f375
I20240503 02:00:45.060494 22566 tablet_bootstrap.cc:492] T 30abfada8ab44449b9a4237c57093b73 P 838ef968bb45484bb27f5af6f3989bae: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:45.060981 22566 tablet_bootstrap.cc:492] T 30abfada8ab44449b9a4237c57093b73 P 838ef968bb45484bb27f5af6f3989bae: Bootstrap complete.
I20240503 02:00:45.061214 22566 ts_tablet_manager.cc:1367] T 30abfada8ab44449b9a4237c57093b73 P 838ef968bb45484bb27f5af6f3989bae: Time spent bootstrapping tablet: real 0.029s user 0.025s sys 0.003s
I20240503 02:00:45.061764 22566 raft_consensus.cc:348] T 30abfada8ab44449b9a4237c57093b73 P 838ef968bb45484bb27f5af6f3989bae [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } attrs { replace: true } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false replace: true } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { promote: true } }
I20240503 02:00:45.061903 22566 raft_consensus.cc:729] T 30abfada8ab44449b9a4237c57093b73 P 838ef968bb45484bb27f5af6f3989bae [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 838ef968bb45484bb27f5af6f3989bae, State: Initialized, Role: LEARNER
I20240503 02:00:45.062018 22566 consensus_queue.cc:260] T 30abfada8ab44449b9a4237c57093b73 P 838ef968bb45484bb27f5af6f3989bae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } attrs { replace: true } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false replace: true } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { promote: true } }
I20240503 02:00:45.062386 22566 ts_tablet_manager.cc:1398] T 30abfada8ab44449b9a4237c57093b73 P 838ef968bb45484bb27f5af6f3989bae: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:45.062965 17111 tablet_copy_service.cc:342] P 89dd09f8aaee4b249ae3ce924a422a67: Request end of tablet copy session 838ef968bb45484bb27f5af6f3989bae-30abfada8ab44449b9a4237c57093b73 received from {username='slave'} at 127.0.99.137:47267
I20240503 02:00:45.063078 17111 tablet_copy_service.cc:433] P 89dd09f8aaee4b249ae3ce924a422a67: ending tablet copy session 838ef968bb45484bb27f5af6f3989bae-30abfada8ab44449b9a4237c57093b73 on tablet 30abfada8ab44449b9a4237c57093b73 with peer 838ef968bb45484bb27f5af6f3989bae
I20240503 02:00:45.132977 22612 ts_tablet_manager.cc:902] T 63962b169caf4f14888871e246c00075 P 5b8291b26a7f448b8636187d1caf86d0: Initiating tablet copy from peer d43ed6b8663947d99a72cd922cc323aa (127.0.99.132:40023)
I20240503 02:00:45.134197 22612 tablet_copy_client.cc:308] T 63962b169caf4f14888871e246c00075 P 5b8291b26a7f448b8636187d1caf86d0: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:40023
I20240503 02:00:45.135697 17376 tablet_copy_service.cc:140] P d43ed6b8663947d99a72cd922cc323aa: Received BeginTabletCopySession request for tablet 63962b169caf4f14888871e246c00075 from peer 5b8291b26a7f448b8636187d1caf86d0 ({username='slave'} at 127.0.99.131:51021)
I20240503 02:00:45.135821 17376 tablet_copy_service.cc:161] P d43ed6b8663947d99a72cd922cc323aa: Beginning new tablet copy session on tablet 63962b169caf4f14888871e246c00075 from peer 5b8291b26a7f448b8636187d1caf86d0 at {username='slave'} at 127.0.99.131:51021: session id = 5b8291b26a7f448b8636187d1caf86d0-63962b169caf4f14888871e246c00075
I20240503 02:00:45.136868 17376 tablet_copy_source_session.cc:205] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:45.138007 22612 ts_tablet_manager.cc:1875] T 63962b169caf4f14888871e246c00075 P 5b8291b26a7f448b8636187d1caf86d0: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 02:00:45.140110 22612 ts_tablet_manager.cc:1888] T 63962b169caf4f14888871e246c00075 P 5b8291b26a7f448b8636187d1caf86d0: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.9
I20240503 02:00:45.140290 22612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63962b169caf4f14888871e246c00075. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:45.141475 22612 tablet_copy_client.cc:783] T 63962b169caf4f14888871e246c00075 P 5b8291b26a7f448b8636187d1caf86d0: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:45.141633 22612 tablet_copy_client.cc:647] T 63962b169caf4f14888871e246c00075 P 5b8291b26a7f448b8636187d1caf86d0: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:45.143254 22612 tablet_copy_client.cc:523] T 63962b169caf4f14888871e246c00075 P 5b8291b26a7f448b8636187d1caf86d0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:45.144868 22612 tablet_bootstrap.cc:492] T 63962b169caf4f14888871e246c00075 P 5b8291b26a7f448b8636187d1caf86d0: Bootstrap starting.
I20240503 02:00:45.151330 22566 ts_tablet_manager.cc:902] T fdf0a4b9e17642849a4094d2f0b9a255 P 838ef968bb45484bb27f5af6f3989bae: Initiating tablet copy from peer 002cc3411acb4fdfa0b67c6f57f687a6 (127.0.99.135:38989)
F20240503 02:00:45.151571 22566 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-30abfada8ab44449b9a4237c57093b73.queue_time_us, tablet-download-pool-30abfada8ab44449b9a4237c57093b73.queue_time_us) (tablet-download-pool-30abfada8ab44449b9a4237c57093b73.run_cpu_time_us, tablet-download-pool-30abfada8ab44449b9a4237c57093b73.run_cpu_time_us) (tablet-download-pool-30abfada8ab44449b9a4237c57093b73.run_wall_time_us, tablet-download-pool-30abfada8ab44449b9a4237c57093b73.run_wall_time_us) (tablet-download-pool-3f67591d6a3744ab90a3e4d2375a59db.queue_time_us, tablet-download-pool-3f67591d6a3744ab90a3e4d2375a59db.queue_time_us) (tablet-download-pool-3f67591d6a3744ab90a3e4d2375a59db.run_cpu_time_us, tablet-download-pool-3f67591d6a3744ab90a3e4d2375a59db.run_cpu_time_us) (tablet-download-pool-3f67591d6a3744ab90a3e4d2375a59db.run_wall_time_us, tablet-download-pool-3f67591d6a3744ab90a3e4d2375a59db.run_wall_time_us) (tablet-download-pool-4782de9826d6460e9e993e01327034e4.queue_time_us, tablet-download-pool-4782de9826d6460e9e993e01327034e4.queue_time_us) (tablet-download-pool-4782de9826d6460e9e993e01327034e4.run_cpu_time_us, tablet-download-pool-4782de9826d6460e9e993e01327034e4.run_cpu_time_us) (tablet-download-pool-4782de9826d6460e9e993e01327034e4.run_wall_time_us, tablet-download-pool-4782de9826d6460e9e993e01327034e4.run_wall_time_us) (tablet-download-pool-5833b4f9a5434734817f154d660aa9f7.queue_time_us, tablet-download-pool-5833b4f9a5434734817f154d660aa9f7.queue_time_us) (tablet-download-pool-5833b4f9a5434734817f154d660aa9f7.run_cpu_time_us, tablet-download-pool-5833b4f9a5434734817f154d660aa9f7.run_cpu_time_us) (tablet-download-pool-5833b4f9a5434734817f154d660aa9f7.run_wall_time_us, tablet-download-pool-5833b4f9a5434734817f154d660aa9f7.run_wall_time_us) (tablet-download-pool-78d106d705df4089a4dfae4d5dc533cc.queue_time_us, tablet-download-pool-78d106d705df4089a4dfae4d5dc533cc.queue_time_us) (tablet-download-pool-78d106d705df4089a4dfae4d5dc533cc.run_cpu_time_us, tablet-download-pool-78d106d705df4089a4dfae4d5dc533cc.run_cpu_time_us) (tablet-download-pool-78d106d705df4089a4dfae4d5dc533cc.run_wall_time_us, tablet-download-pool-78d106d705df4089a4dfae4d5dc533cc.run_wall_time_us) (tablet-download-pool-78f81fe4c6234ff0b5fcabd974779a66.queue_time_us, tablet-download-pool-78f81fe4c6234ff0b5fcabd974779a66.queue_time_us) (tablet-download-pool-78f81fe4c6234ff0b5fcabd974779a66.run_cpu_time_us, tablet-download-pool-78f81fe4c6234ff0b5fcabd974779a66.run_cpu_time_us) (tablet-download-pool-78f81fe4c6234ff0b5fcabd974779a66.run_wall_time_us, tablet-download-pool-78f81fe4c6234ff0b5fcabd974779a66.run_wall_time_us) (tablet-download-pool-81b54f3bcf34485eb11b1bc954a2bd8f.queue_time_us, tablet-download-pool-81b54f3bcf34485eb11b1bc954a2bd8f.queue_time_us) (tablet-download-pool-81b54f3bcf34485eb11b1bc954a2bd8f.run_cpu_time_us, tablet-download-pool-81b54f3bcf34485eb11b1bc954a2bd8f.run_cpu_time_us) (tablet-download-pool-81b54f3bcf34485eb11b1bc954a2bd8f.run_wall_time_us, tablet-download-pool-81b54f3bcf34485eb11b1bc954a2bd8f.run_wall_time_us) (tablet-download-pool-a39f284167e942188c181147e8a9e74e.queue_time_us, tablet-download-pool-a39f284167e942188c181147e8a9e74e.queue_time_us) (tablet-download-pool-a39f284167e942188c181147e8a9e74e.run_cpu_time_us, tablet-download-pool-a39f284167e942188c181147e8a9e74e.run_cpu_time_us) (tablet-download-pool-a39f284167e942188c181147e8a9e74e.run_wall_time_us, tablet-download-pool-a39f284167e942188c181147e8a9e74e.run_wall_time_us) (tablet-download-pool-ab536115ac7a4ba193f718d421f4000d.queue_time_us, tablet-download-pool-ab536115ac7a4ba193f718d421f4000d.queue_time_us) (tablet-download-pool-ab536115ac7a4ba193f718d421f4000d.run_cpu_time_us, tablet-download-pool-ab536115ac7a4ba193f718d421f4000d.run_cpu_time_us) (tablet-download-pool-ab536115ac7a4ba193f718d421f4000d.run_wall_time_us, tablet-download-pool-ab536115ac7a4ba193f718d421f4000d.run_wall_time_us) (tablet-download-pool-acfd773c1f384c219d776878407d1144.queue_time_us, tablet-download-pool-acfd773c1f384c219d776878407d1144.queue_time_us) (tablet-download-pool-acfd773c1f384c219d776878407d1144.run_cpu_time_us, tablet-download-pool-acfd773c1f384c219d776878407d1144.run_cpu_time_us) (tablet-download-pool-acfd773c1f384c219d776878407d1144.run_wall_time_us, tablet-download-pool-acfd773c1f384c219d776878407d1144.run_wall_time_us) (tablet-download-pool-d81dcfefb4fd401b9cfd2ca36034a010.queue_time_us, tablet-download-pool-d81dcfefb4fd401b9cfd2ca36034a010.queue_time_us) (tablet-download-pool-d81dcfefb4fd401b9cfd2ca36034a010.run_cpu_time_us, tablet-download-pool-d81dcfefb4fd401b9cfd2ca36034a010.run_cpu_time_us) (tablet-download-pool-d81dcfefb4fd401b9cfd2ca36034a010.run_wall_time_us, tablet-download-pool-d81dcfefb4fd401b9cfd2ca36034a010.run_wall_time_us) (tablet-download-pool-e245c9cf642043cfbb3b87114beeb002.queue_time_us, tablet-download-pool-e245c9cf642043cfbb3b87114beeb002.queue_time_us) (tablet-download-pool-e245c9cf642043cfbb3b87114beeb002.run_cpu_time_us, tablet-download-pool-e245c9cf642043cfbb3b87114beeb002.run_cpu_time_us) (tablet-download-pool-e245c9cf642043cfbb3b87114beeb002.run_wall_time_us, tablet-download-pool-e245c9cf642043cfbb3b87114beeb002.run_wall_time_us) (tablet-download-pool-ec69ad86be3d4793b10aca98dc4c1736.queue_time_us, tablet-download-pool-ec69ad86be3d4793b10aca98dc4c1736.queue_time_us) (tablet-download-pool-ec69ad86be3d4793b10aca98dc4c1736.run_cpu_time_us, tablet-download-pool-ec69ad86be3d4793b10aca98dc4c1736.run_cpu_time_us) (tablet-download-pool-ec69ad86be3d4793b10aca98dc4c1736.run_wall_time_us, tablet-download-pool-ec69ad86be3d4793b10aca98dc4c1736.run_wall_time_us) (tablet-download-pool-fbec07350ad04c49afd42e5d88dda6ad.queue_time_us, tablet-download-pool-fbec07350ad04c49afd42e5d88dda6ad.queue_time_us) (tablet-download-pool-fbec07350ad04c49afd42e5d88dda6ad.run_cpu_time_us, tablet-download-pool-fbec07350ad04c49afd42e5d88dda6ad.run_cpu_time_us) (tablet-download-pool-fbec07350ad04c49afd42e5d88dda6ad.run_wall_time_us, tablet-download-pool-fbec07350ad04c49afd42e5d88dda6ad.run_wall_time_us) (tablet-download-pool-fdf0a4b9e17642849a4094d2f0b9a255.queue_time_us, tablet-download-pool-fdf0a4b9e17642849a4094d2f0b9a255.queue_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
@ 0x7f74233bfdcd google::LogMessage::Fail() at ??:0
@ 0x7f74233c3b93 google::LogMessage::SendToLog() at ??:0
I20240503 02:00:45.158488 22624 ts_tablet_manager.cc:902] T a39f284167e942188c181147e8a9e74e P 16d437fe629643948c95d628926d5ffa: Initiating tablet copy from peer d43ed6b8663947d99a72cd922cc323aa (127.0.99.132:40023)
I20240503 02:00:45.159515 22624 tablet_copy_client.cc:308] T a39f284167e942188c181147e8a9e74e P 16d437fe629643948c95d628926d5ffa: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:40023
@ 0x7f74233bf7cc google::LogMessage::Flush() at ??:0
I20240503 02:00:45.160187 17376 tablet_copy_service.cc:140] P d43ed6b8663947d99a72cd922cc323aa: Received BeginTabletCopySession request for tablet a39f284167e942188c181147e8a9e74e from peer 16d437fe629643948c95d628926d5ffa ({username='slave'} at 127.0.99.133:43879)
I20240503 02:00:45.160305 17376 tablet_copy_service.cc:161] P d43ed6b8663947d99a72cd922cc323aa: Beginning new tablet copy session on tablet a39f284167e942188c181147e8a9e74e from peer 16d437fe629643948c95d628926d5ffa at {username='slave'} at 127.0.99.133:43879: session id = 16d437fe629643948c95d628926d5ffa-a39f284167e942188c181147e8a9e74e
I20240503 02:00:45.161638 17376 tablet_copy_source_session.cc:205] T a39f284167e942188c181147e8a9e74e P d43ed6b8663947d99a72cd922cc323aa: Tablet Copy: opened 0 blocks and 1 log segments
@ 0x7f74233c0f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
I20240503 02:00:45.164216 17752 raft_consensus.cc:2932] T 63962b169caf4f14888871e246c00075 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 5b8291b26a7f448b8636187d1caf86d0 (127.0.99.131) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { promote: true } } }
I20240503 02:00:45.164698 22624 ts_tablet_manager.cc:1875] T a39f284167e942188c181147e8a9e74e P 16d437fe629643948c95d628926d5ffa: Deleting tablet data with delete state TABLET_DATA_COPYING
@ 0x7f7424829044 kudu::TraceMetrics::InternName() at ??:0
@ 0x7f742480df98 kudu::ThreadPool::ThreadPool() at ??:0
I20240503 02:00:45.169571 22624 ts_tablet_manager.cc:1888] T a39f284167e942188c181147e8a9e74e P 16d437fe629643948c95d628926d5ffa: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.6
I20240503 02:00:45.170498 22624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a39f284167e942188c181147e8a9e74e. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7f742480befc kudu::ThreadPoolBuilder::Build() at ??:0
I20240503 02:00:45.173408 22624 tablet_copy_client.cc:783] T a39f284167e942188c181147e8a9e74e P 16d437fe629643948c95d628926d5ffa: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:45.173605 22624 tablet_copy_client.cc:647] T a39f284167e942188c181147e8a9e74e P 16d437fe629643948c95d628926d5ffa: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:45.174157 22631 ts_tablet_manager.cc:902] T 81b54f3bcf34485eb11b1bc954a2bd8f P 375d0f2ec11d4e25bd9d093bb6a5572d: Initiating tablet copy from peer 89dd09f8aaee4b249ae3ce924a422a67 (127.0.99.130:38889)
F20240503 02:00:45.174350 22631 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-30abfada8ab44449b9a4237c57093b73.queue_time_us, tablet-download-pool-30abfada8ab44449b9a4237c57093b73.queue_time_us) (tablet-download-pool-30abfada8ab44449b9a4237c57093b73.run_cpu_time_us, tablet-download-pool-30abfada8ab44449b9a4237c57093b73.run_cpu_time_us) (tablet-download-pool-30abfada8ab44449b9a4237c57093b73.run_wall_time_us, tablet-download-pool-30abfada8ab44449b9a4237c57093b73.run_wall_time_us) (tablet-download-pool-42a900125241432a8bb31dad0b940968.queue_time_us, tablet-download-pool-42a900125241432a8bb31dad0b940968.queue_time_us) (tablet-download-pool-42a900125241432a8bb31dad0b940968.run_cpu_time_us, tablet-download-pool-42a900125241432a8bb31dad0b940968.run_cpu_time_us) (tablet-download-pool-42a900125241432a8bb31dad0b940968.run_wall_time_us, tablet-download-pool-42a900125241432a8bb31dad0b940968.run_wall_time_us) (tablet-download-pool-4782de9826d6460e9e993e01327034e4.queue_time_us, tablet-download-pool-4782de9826d6460e9e993e01327034e4.queue_time_us) (tablet-download-pool-4782de9826d6460e9e993e01327034e4.run_cpu_time_us, tablet-download-pool-4782de9826d6460e9e993e01327034e4.run_cpu_time_us) (tablet-download-pool-4782de9826d6460e9e993e01327034e4.run_wall_time_us, tablet-download-pool-4782de9826d6460e9e993e01327034e4.run_wall_time_us) (tablet-download-pool-5833b4f9a5434734817f154d660aa9f7.queue_time_us, tablet-download-pool-5833b4f9a5434734817f154d660aa9f7.queue_time_us) (tablet-download-pool-5833b4f9a5434734817f154d660aa9f7.run_cpu_time_us, tablet-download-pool-5833b4f9a5434734817f154d660aa9f7.run_cpu_time_us) (tablet-download-pool-5833b4f9a5434734817f154d660aa9f7.run_wall_time_us, tablet-download-pool-5833b4f9a5434734817f154d660aa9f7.run_wall_time_us) (tablet-download-pool-5d2104698d5f452c8350cce10afd3898.queue_time_us, tablet-download-pool-5d2104698d5f452c8350cce10afd3898.queue_time_us) (tablet-download-pool-5d2104698d5f452c8350cce10afd3898.run_cpu_time_us, tablet-download-pool-5d2104698d5f452c8350cce10afd3898.run_cpu_time_us) (tablet-download-pool-5d2104698d5f452c8350cce10afd3898.run_wall_time_us, tablet-download-pool-5d2104698d5f452c8350cce10afd3898.run_wall_time_us) (tablet-download-pool-63962b169caf4f14888871e246c00075.queue_time_us, tablet-download-pool-63962b169caf4f14888871e246c00075.queue_time_us) (tablet-download-pool-63962b169caf4f14888871e246c00075.run_cpu_time_us, tablet-download-pool-63962b169caf4f14888871e246c00075.run_cpu_time_us) (tablet-download-pool-63962b169caf4f14888871e246c00075.run_wall_time_us, tablet-download-pool-63962b169caf4f14888871e246c00075.run_wall_time_us) (tablet-download-pool-78f81fe4c6234ff0b5fcabd974779a66.queue_time_us, tablet-download-pool-78f81fe4c6234ff0b5fcabd974779a66.queue_time_us) (tablet-download-pool-78f81fe4c6234ff0b5fcabd974779a66.run_cpu_time_us, tablet-download-pool-78f81fe4c6234ff0b5fcabd974779a66.run_cpu_time_us) (tablet-download-pool-78f81fe4c6234ff0b5fcabd974779a66.run_wall_time_us, tablet-download-pool-78f81fe4c6234ff0b5fcabd974779a66.run_wall_time_us) (tablet-download-pool-81b54f3bcf34485eb11b1bc954a2bd8f.queue_time_us, tablet-download-pool-81b54f3bcf34485eb11b1bc954a2bd8f.queue_time_us) (tablet-download-pool-885fe0c3a5e04f408e5b02c3c133248a.queue_time_us, tablet-download-pool-885fe0c3a5e04f408e5b02c3c133248a.queue_time_us) (tablet-download-pool-885fe0c3a5e04f408e5b02c3c133248a.run_cpu_time_us, tablet-download-pool-885fe0c3a5e04f408e5b02c3c133248a.run_cpu_time_us) (tablet-download-pool-885fe0c3a5e04f408e5b02c3c133248a.run_wall_time_us, tablet-download-pool-885fe0c3a5e04f408e5b02c3c133248a.run_wall_time_us) (tablet-download-pool-acfd773c1f384c219d776878407d1144.queue_time_us, tablet-download-pool-acfd773c1f384c219d776878407d1144.queue_time_us) (tablet-download-pool-acfd773c1f384c219d776878407d1144.run_cpu_time_us, tablet-download-pool-acfd773c1f384c219d776878407d1144.run_cpu_time_us) (tablet-download-pool-acfd773c1f384c219d776878407d1144.run_wall_time_us, tablet-download-pool-acfd773c1f384c219d776878407d1144.run_wall_time_us) (tablet-download-pool-ae4259dd3dc745869256269632886f8a.queue_time_us, tablet-download-pool-ae4259dd3dc745869256269632886f8a.queue_time_us) (tablet-download-pool-ae4259dd3dc745869256269632886f8a.run_cpu_time_us, tablet-download-pool-ae4259dd3dc745869256269632886f8a.run_cpu_time_us) (tablet-download-pool-ae4259dd3dc745869256269632886f8a.run_wall_time_us, tablet-download-pool-ae4259dd3dc745869256269632886f8a.run_wall_time_us) (tablet-download-pool-d52afe1281064d6480427554ca4bb2f6.queue_time_us, tablet-download-pool-d52afe1281064d6480427554ca4bb2f6.queue_time_us) (tablet-download-pool-d52afe1281064d6480427554ca4bb2f6.run_cpu_time_us, tablet-download-pool-d52afe1281064d6480427554ca4bb2f6.run_cpu_time_us) (tablet-download-pool-d52afe1281064d6480427554ca4bb2f6.run_wall_time_us, tablet-download-pool-d52afe1281064d6480427554ca4bb2f6.run_wall_time_us) (tablet-download-pool-d81dcfefb4fd401b9cfd2ca36034a010.queue_time_us, tablet-download-pool-d81dcfefb4fd401b9cfd2ca36034a010.queue_time_us) (tablet-download-pool-d81dcfefb4fd401b9cfd2ca36034a010.run_cpu_time_us, tablet-download-pool-d81dcfefb4fd401b9cfd2ca36034a010.run_cpu_time_us) (tablet-download-pool-d81dcfefb4fd401b9cfd2ca36034a010.run_wall_time_us, tablet-download-pool-d81dcfefb4fd401b9cfd2ca36034a010.run_wall_time_us) (tablet-download-pool-ec69ad86be3d4793b10aca98dc4c1736.queue_time_us, tablet-download-pool-ec69ad86be3d4793b10aca98dc4c1736.queue_time_us) (tablet-download-pool-ec69ad86be3d4793b10aca98dc4c1736.run_cpu_time_us, tablet-download-pool-ec69ad86be3d4793b10aca98dc4c1736.run_cpu_time_us) (tablet-download-pool-ec69ad86be3d4793b10aca98dc4c1736.run_wall_time_us, tablet-download-pool-ec69ad86be3d4793b10aca98dc4c1736.run_wall_time_us) (tablet-download-pool-fdf0a4b9e17642849a4094d2f0b9a255.queue_time_us, tablet-download-pool-fdf0a4b9e17642849a4094d2f0b9a255.queue_time_us) (tablet-download-pool-fdf0a4b9e17642849a4094d2f0b9a255.run_cpu_time_us, tablet-download-pool-fdf0a4b9e17642849a4094d2f0b9a255.run_cpu_time_us) (tablet-download-pool-fdf0a4b9e17642849a4094d2f0b9a255.run_wall_time_us, tablet-download-pool-fdf0a4b9e17642849a4094d2f0b9a255.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
I20240503 02:00:45.176362 22624 tablet_copy_client.cc:523] T a39f284167e942188c181147e8a9e74e P 16d437fe629643948c95d628926d5ffa: tablet copy: Tablet Copy complete. Replacing tablet superblock.
@ 0x7f8fbb194dcd google::LogMessage::Fail() at ??:0
I20240503 02:00:45.180626 22624 tablet_bootstrap.cc:492] T a39f284167e942188c181147e8a9e74e P 16d437fe629643948c95d628926d5ffa: Bootstrap starting.
@ 0x7f74273be77a kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7f74273c7a45 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
I20240503 02:00:45.187423 22633 ts_tablet_manager.cc:902] T 3f67591d6a3744ab90a3e4d2375a59db P 375d0f2ec11d4e25bd9d093bb6a5572d: Initiating tablet copy from peer 16d437fe629643948c95d628926d5ffa (127.0.99.133:46581)
@ 0x7f8fbb198b93 google::LogMessage::SendToLog() at ??:0
I20240503 02:00:45.191597 22612 tablet_bootstrap.cc:492] T 63962b169caf4f14888871e246c00075 P 5b8291b26a7f448b8636187d1caf86d0: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
@ 0x7f8fbb1947cc google::LogMessage::Flush() at ??:0
I20240503 02:00:45.191955 22612 tablet_bootstrap.cc:492] T 63962b169caf4f14888871e246c00075 P 5b8291b26a7f448b8636187d1caf86d0: Bootstrap complete.
I20240503 02:00:45.192166 22612 ts_tablet_manager.cc:1367] T 63962b169caf4f14888871e246c00075 P 5b8291b26a7f448b8636187d1caf86d0: Time spent bootstrapping tablet: real 0.047s user 0.030s sys 0.000s
I20240503 02:00:45.192801 22612 raft_consensus.cc:348] T 63962b169caf4f14888871e246c00075 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { promote: true } }
I20240503 02:00:45.193022 22612 raft_consensus.cc:729] T 63962b169caf4f14888871e246c00075 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5b8291b26a7f448b8636187d1caf86d0, State: Initialized, Role: LEARNER
I20240503 02:00:45.193161 22612 consensus_queue.cc:260] T 63962b169caf4f14888871e246c00075 P 5b8291b26a7f448b8636187d1caf86d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { promote: true } }
@ 0x7f742744876f kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7f8fbb195f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
I20240503 02:00:45.195806 22612 ts_tablet_manager.cc:1398] T 63962b169caf4f14888871e246c00075 P 5b8291b26a7f448b8636187d1caf86d0: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20240503 02:00:45.198374 17376 tablet_copy_service.cc:342] P d43ed6b8663947d99a72cd922cc323aa: Request end of tablet copy session 5b8291b26a7f448b8636187d1caf86d0-63962b169caf4f14888871e246c00075 received from {username='slave'} at 127.0.99.131:51021
I20240503 02:00:45.198467 17376 tablet_copy_service.cc:433] P d43ed6b8663947d99a72cd922cc323aa: ending tablet copy session 5b8291b26a7f448b8636187d1caf86d0-63962b169caf4f14888871e246c00075 on tablet 63962b169caf4f14888871e246c00075 with peer 5b8291b26a7f448b8636187d1caf86d0
@ 0x7f8fbc5fe044 kudu::TraceMetrics::InternName() at ??:0
@ 0x7f742745c4f6 kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7f8fbc5e2f98 kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7f8fbc5e0efc kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7f7427447226 _ZZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS_6StatusENS0_24TabletServerErrorPB_CodeEEEENKUlvE_clEv at ??:0
@ 0x7f8fbf19377a kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7f7427456c0a _ZNSt17_Function_handlerIFvvEZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x55f5167bf75c std::function<>::operator()() at ??:0
@ 0x7f8fbf19ca45 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7f7424810e06 kudu::ThreadPool::DispatchThread() at ??:0
I20240503 02:00:45.224797 22533 ts_tablet_manager.cc:902] T 78d106d705df4089a4dfae4d5dc533cc P d43ed6b8663947d99a72cd922cc323aa: Initiating tablet copy from peer 002cc3411acb4fdfa0b67c6f57f687a6 (127.0.99.135:38989)
I20240503 02:00:45.225363 22533 tablet_copy_client.cc:308] T 78d106d705df4089a4dfae4d5dc533cc P d43ed6b8663947d99a72cd922cc323aa: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.135:38989
I20240503 02:00:45.226049 17772 tablet_copy_service.cc:140] P 002cc3411acb4fdfa0b67c6f57f687a6: Received BeginTabletCopySession request for tablet 78d106d705df4089a4dfae4d5dc533cc from peer d43ed6b8663947d99a72cd922cc323aa ({username='slave'} at 127.0.99.132:51549)
I20240503 02:00:45.226203 17772 tablet_copy_service.cc:161] P 002cc3411acb4fdfa0b67c6f57f687a6: Beginning new tablet copy session on tablet 78d106d705df4089a4dfae4d5dc533cc from peer d43ed6b8663947d99a72cd922cc323aa at {username='slave'} at 127.0.99.132:51549: session id = d43ed6b8663947d99a72cd922cc323aa-78d106d705df4089a4dfae4d5dc533cc
@ 0x7f8fbf21d76f kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
I20240503 02:00:45.227490 17772 tablet_copy_source_session.cc:205] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6: Tablet Copy: opened 0 blocks and 1 log segments
@ 0x7f74248116ef _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
I20240503 02:00:45.228444 22533 ts_tablet_manager.cc:1875] T 78d106d705df4089a4dfae4d5dc533cc P d43ed6b8663947d99a72cd922cc323aa: Deleting tablet data with delete state TABLET_DATA_COPYING
@ 0x7f742481303b _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f8fbf2314f6 kudu::tserver::TabletCopyRunnable::Run() at ??:0
I20240503 02:00:45.231755 22533 ts_tablet_manager.cc:1888] T 78d106d705df4089a4dfae4d5dc533cc P d43ed6b8663947d99a72cd922cc323aa: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.6
I20240503 02:00:45.231987 22533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78d106d705df4089a4dfae4d5dc533cc. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x55f5167bf75c std::function<>::operator()() at ??:0
I20240503 02:00:45.233886 22533 tablet_copy_client.cc:783] T 78d106d705df4089a4dfae4d5dc533cc P d43ed6b8663947d99a72cd922cc323aa: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:45.234184 22533 tablet_copy_client.cc:647] T 78d106d705df4089a4dfae4d5dc533cc P d43ed6b8663947d99a72cd922cc323aa: tablet copy: Starting download of 1 WAL segments...
@ 0x7f8fbf21c226 _ZZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS_6StatusENS0_24TabletServerErrorPB_CodeEEEENKUlvE_clEv at ??:0
@ 0x7f7424800807 kudu::Thread::SuperviseThread() at ??:0
I20240503 02:00:45.238915 22533 tablet_copy_client.cc:523] T 78d106d705df4089a4dfae4d5dc533cc P d43ed6b8663947d99a72cd922cc323aa: tablet copy: Tablet Copy complete. Replacing tablet superblock.
@ 0x7f8fbf22bc0a _ZNSt17_Function_handlerIFvvEZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
I20240503 02:00:45.240690 22533 tablet_bootstrap.cc:492] T 78d106d705df4089a4dfae4d5dc533cc P d43ed6b8663947d99a72cd922cc323aa: Bootstrap starting.
@ 0x562a76c3e75c std::function<>::operator()() at ??:0
@ 0x7f742503e6db start_thread at ??:0
@ 0x7f8fbc5e5e06 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f742257671f clone at ??:0
@ 0x7f8fbc5e66ef _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f8fbc5e803b _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x562a76c3e75c std::function<>::operator()() at ??:0
@ 0x7f8fbc5d5807 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f8fbce136db start_thread at ??:0
@ 0x7f8fba34b71f clone at ??:0
I20240503 02:00:45.264890 22624 tablet_bootstrap.cc:492] T a39f284167e942188c181147e8a9e74e P 16d437fe629643948c95d628926d5ffa: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:45.265828 22624 tablet_bootstrap.cc:492] T a39f284167e942188c181147e8a9e74e P 16d437fe629643948c95d628926d5ffa: Bootstrap complete.
I20240503 02:00:45.265985 22624 ts_tablet_manager.cc:1367] T a39f284167e942188c181147e8a9e74e P 16d437fe629643948c95d628926d5ffa: Time spent bootstrapping tablet: real 0.085s user 0.044s sys 0.004s
I20240503 02:00:45.266405 22624 raft_consensus.cc:348] T a39f284167e942188c181147e8a9e74e P 16d437fe629643948c95d628926d5ffa [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } attrs { replace: true } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { promote: true } }
I20240503 02:00:45.266686 22624 raft_consensus.cc:729] T a39f284167e942188c181147e8a9e74e P 16d437fe629643948c95d628926d5ffa [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 16d437fe629643948c95d628926d5ffa, State: Initialized, Role: LEARNER
I20240503 02:00:45.266880 22624 consensus_queue.cc:260] T a39f284167e942188c181147e8a9e74e P 16d437fe629643948c95d628926d5ffa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } attrs { replace: true } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { promote: true } }
I20240503 02:00:45.267228 22624 ts_tablet_manager.cc:1398] T a39f284167e942188c181147e8a9e74e P 16d437fe629643948c95d628926d5ffa: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:45.267758 17376 tablet_copy_service.cc:342] P d43ed6b8663947d99a72cd922cc323aa: Request end of tablet copy session 16d437fe629643948c95d628926d5ffa-a39f284167e942188c181147e8a9e74e received from {username='slave'} at 127.0.99.133:43879
I20240503 02:00:45.267834 17376 tablet_copy_service.cc:433] P d43ed6b8663947d99a72cd922cc323aa: ending tablet copy session 16d437fe629643948c95d628926d5ffa-a39f284167e942188c181147e8a9e74e on tablet a39f284167e942188c181147e8a9e74e with peer 16d437fe629643948c95d628926d5ffa
I20240503 02:00:45.267860 22664 ts_tablet_manager.cc:902] T fdf0a4b9e17642849a4094d2f0b9a255 P d43ed6b8663947d99a72cd922cc323aa: Initiating tablet copy from peer 002cc3411acb4fdfa0b67c6f57f687a6 (127.0.99.135:38989)
I20240503 02:00:45.268636 22664 tablet_copy_client.cc:308] T fdf0a4b9e17642849a4094d2f0b9a255 P d43ed6b8663947d99a72cd922cc323aa: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.135:38989
I20240503 02:00:45.269181 17772 tablet_copy_service.cc:140] P 002cc3411acb4fdfa0b67c6f57f687a6: Received BeginTabletCopySession request for tablet fdf0a4b9e17642849a4094d2f0b9a255 from peer d43ed6b8663947d99a72cd922cc323aa ({username='slave'} at 127.0.99.132:51549)
I20240503 02:00:45.269295 17772 tablet_copy_service.cc:161] P 002cc3411acb4fdfa0b67c6f57f687a6: Beginning new tablet copy session on tablet fdf0a4b9e17642849a4094d2f0b9a255 from peer d43ed6b8663947d99a72cd922cc323aa at {username='slave'} at 127.0.99.132:51549: session id = d43ed6b8663947d99a72cd922cc323aa-fdf0a4b9e17642849a4094d2f0b9a255
I20240503 02:00:45.270272 17772 tablet_copy_source_session.cc:205] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:45.270983 22664 ts_tablet_manager.cc:1875] T fdf0a4b9e17642849a4094d2f0b9a255 P d43ed6b8663947d99a72cd922cc323aa: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 02:00:45.272859 22664 ts_tablet_manager.cc:1888] T fdf0a4b9e17642849a4094d2f0b9a255 P d43ed6b8663947d99a72cd922cc323aa: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.6
I20240503 02:00:45.273031 22664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdf0a4b9e17642849a4094d2f0b9a255. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:45.274377 22664 tablet_copy_client.cc:783] T fdf0a4b9e17642849a4094d2f0b9a255 P d43ed6b8663947d99a72cd922cc323aa: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:45.274518 22664 tablet_copy_client.cc:647] T fdf0a4b9e17642849a4094d2f0b9a255 P d43ed6b8663947d99a72cd922cc323aa: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:45.275599 22664 tablet_copy_client.cc:523] T fdf0a4b9e17642849a4094d2f0b9a255 P d43ed6b8663947d99a72cd922cc323aa: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:45.276613 22664 tablet_bootstrap.cc:492] T fdf0a4b9e17642849a4094d2f0b9a255 P d43ed6b8663947d99a72cd922cc323aa: Bootstrap starting.
I20240503 02:00:45.308036 22560 ts_tablet_manager.cc:902] T fbec07350ad04c49afd42e5d88dda6ad P c843193b801b4f1a88f630ed3ed0f375: Initiating tablet copy from peer 16d437fe629643948c95d628926d5ffa (127.0.99.133:46581)
I20240503 02:00:45.308534 22560 tablet_copy_client.cc:308] T fbec07350ad04c49afd42e5d88dda6ad P c843193b801b4f1a88f630ed3ed0f375: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.133:46581
I20240503 02:00:45.309137 17508 tablet_copy_service.cc:140] P 16d437fe629643948c95d628926d5ffa: Received BeginTabletCopySession request for tablet fbec07350ad04c49afd42e5d88dda6ad from peer c843193b801b4f1a88f630ed3ed0f375 ({username='slave'} at 127.0.99.134:57603)
I20240503 02:00:45.309250 17508 tablet_copy_service.cc:161] P 16d437fe629643948c95d628926d5ffa: Beginning new tablet copy session on tablet fbec07350ad04c49afd42e5d88dda6ad from peer c843193b801b4f1a88f630ed3ed0f375 at {username='slave'} at 127.0.99.134:57603: session id = c843193b801b4f1a88f630ed3ed0f375-fbec07350ad04c49afd42e5d88dda6ad
I20240503 02:00:45.310153 17508 tablet_copy_source_session.cc:205] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:45.310981 22560 ts_tablet_manager.cc:1875] T fbec07350ad04c49afd42e5d88dda6ad P c843193b801b4f1a88f630ed3ed0f375: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 02:00:45.313464 22560 ts_tablet_manager.cc:1888] T fbec07350ad04c49afd42e5d88dda6ad P c843193b801b4f1a88f630ed3ed0f375: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.6
I20240503 02:00:45.313652 22560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbec07350ad04c49afd42e5d88dda6ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:45.314733 22560 tablet_copy_client.cc:783] T fbec07350ad04c49afd42e5d88dda6ad P c843193b801b4f1a88f630ed3ed0f375: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:45.314874 22560 tablet_copy_client.cc:647] T fbec07350ad04c49afd42e5d88dda6ad P c843193b801b4f1a88f630ed3ed0f375: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:45.317878 22664 tablet_bootstrap.cc:492] T fdf0a4b9e17642849a4094d2f0b9a255 P d43ed6b8663947d99a72cd922cc323aa: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:45.319804 22664 tablet_bootstrap.cc:492] T fdf0a4b9e17642849a4094d2f0b9a255 P d43ed6b8663947d99a72cd922cc323aa: Bootstrap complete.
I20240503 02:00:45.319998 22664 ts_tablet_manager.cc:1367] T fdf0a4b9e17642849a4094d2f0b9a255 P d43ed6b8663947d99a72cd922cc323aa: Time spent bootstrapping tablet: real 0.043s user 0.026s sys 0.004s
I20240503 02:00:45.320547 22664 raft_consensus.cc:348] T fdf0a4b9e17642849a4094d2f0b9a255 P d43ed6b8663947d99a72cd922cc323aa [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false replace: true } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { promote: true } }
I20240503 02:00:45.320698 22664 raft_consensus.cc:729] T fdf0a4b9e17642849a4094d2f0b9a255 P d43ed6b8663947d99a72cd922cc323aa [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d43ed6b8663947d99a72cd922cc323aa, State: Initialized, Role: LEARNER
I20240503 02:00:45.320787 22664 consensus_queue.cc:260] T fdf0a4b9e17642849a4094d2f0b9a255 P d43ed6b8663947d99a72cd922cc323aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false replace: true } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { promote: true } }
I20240503 02:00:45.321149 22664 ts_tablet_manager.cc:1398] T fdf0a4b9e17642849a4094d2f0b9a255 P d43ed6b8663947d99a72cd922cc323aa: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:45.321759 17772 tablet_copy_service.cc:342] P 002cc3411acb4fdfa0b67c6f57f687a6: Request end of tablet copy session d43ed6b8663947d99a72cd922cc323aa-fdf0a4b9e17642849a4094d2f0b9a255 received from {username='slave'} at 127.0.99.132:51549
I20240503 02:00:45.321833 17772 tablet_copy_service.cc:433] P 002cc3411acb4fdfa0b67c6f57f687a6: ending tablet copy session d43ed6b8663947d99a72cd922cc323aa-fdf0a4b9e17642849a4094d2f0b9a255 on tablet fdf0a4b9e17642849a4094d2f0b9a255 with peer d43ed6b8663947d99a72cd922cc323aa
I20240503 02:00:45.333570 22533 tablet_bootstrap.cc:492] T 78d106d705df4089a4dfae4d5dc533cc P d43ed6b8663947d99a72cd922cc323aa: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:45.334049 22533 tablet_bootstrap.cc:492] T 78d106d705df4089a4dfae4d5dc533cc P d43ed6b8663947d99a72cd922cc323aa: Bootstrap complete.
I20240503 02:00:45.334249 22533 ts_tablet_manager.cc:1367] T 78d106d705df4089a4dfae4d5dc533cc P d43ed6b8663947d99a72cd922cc323aa: Time spent bootstrapping tablet: real 0.094s user 0.033s sys 0.004s
I20240503 02:00:45.334648 22533 raft_consensus.cc:348] T 78d106d705df4089a4dfae4d5dc533cc P d43ed6b8663947d99a72cd922cc323aa [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { promote: true } }
I20240503 02:00:45.334838 22533 raft_consensus.cc:729] T 78d106d705df4089a4dfae4d5dc533cc P d43ed6b8663947d99a72cd922cc323aa [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d43ed6b8663947d99a72cd922cc323aa, State: Initialized, Role: LEARNER
I20240503 02:00:45.334920 22533 consensus_queue.cc:260] T 78d106d705df4089a4dfae4d5dc533cc P d43ed6b8663947d99a72cd922cc323aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { promote: true } }
I20240503 02:00:45.335551 22533 ts_tablet_manager.cc:1398] T 78d106d705df4089a4dfae4d5dc533cc P d43ed6b8663947d99a72cd922cc323aa: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:45.336575 17772 tablet_copy_service.cc:342] P 002cc3411acb4fdfa0b67c6f57f687a6: Request end of tablet copy session d43ed6b8663947d99a72cd922cc323aa-78d106d705df4089a4dfae4d5dc533cc received from {username='slave'} at 127.0.99.132:51549
I20240503 02:00:45.338505 17772 tablet_copy_service.cc:433] P 002cc3411acb4fdfa0b67c6f57f687a6: ending tablet copy session d43ed6b8663947d99a72cd922cc323aa-78d106d705df4089a4dfae4d5dc533cc on tablet 78d106d705df4089a4dfae4d5dc533cc with peer d43ed6b8663947d99a72cd922cc323aa
I20240503 02:00:45.338742 22560 tablet_copy_client.cc:523] T fbec07350ad04c49afd42e5d88dda6ad P c843193b801b4f1a88f630ed3ed0f375: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:45.340751 22560 tablet_bootstrap.cc:492] T fbec07350ad04c49afd42e5d88dda6ad P c843193b801b4f1a88f630ed3ed0f375: Bootstrap starting.
I20240503 02:00:45.357411 22533 ts_tablet_manager.cc:902] T 4782de9826d6460e9e993e01327034e4 P d43ed6b8663947d99a72cd922cc323aa: Initiating tablet copy from peer 5b8291b26a7f448b8636187d1caf86d0 (127.0.99.131:44479)
I20240503 02:00:45.357806 22533 tablet_copy_client.cc:308] T 4782de9826d6460e9e993e01327034e4 P d43ed6b8663947d99a72cd922cc323aa: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:44479
I20240503 02:00:45.358283 17244 tablet_copy_service.cc:140] P 5b8291b26a7f448b8636187d1caf86d0: Received BeginTabletCopySession request for tablet 4782de9826d6460e9e993e01327034e4 from peer d43ed6b8663947d99a72cd922cc323aa ({username='slave'} at 127.0.99.132:39351)
I20240503 02:00:45.358410 17244 tablet_copy_service.cc:161] P 5b8291b26a7f448b8636187d1caf86d0: Beginning new tablet copy session on tablet 4782de9826d6460e9e993e01327034e4 from peer d43ed6b8663947d99a72cd922cc323aa at {username='slave'} at 127.0.99.132:39351: session id = d43ed6b8663947d99a72cd922cc323aa-4782de9826d6460e9e993e01327034e4
I20240503 02:00:45.359411 17244 tablet_copy_source_session.cc:205] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:45.360200 22533 ts_tablet_manager.cc:1875] T 4782de9826d6460e9e993e01327034e4 P d43ed6b8663947d99a72cd922cc323aa: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 02:00:45.362761 22533 ts_tablet_manager.cc:1888] T 4782de9826d6460e9e993e01327034e4 P d43ed6b8663947d99a72cd922cc323aa: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.6
I20240503 02:00:45.362921 22533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4782de9826d6460e9e993e01327034e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:45.364238 22533 tablet_copy_client.cc:783] T 4782de9826d6460e9e993e01327034e4 P d43ed6b8663947d99a72cd922cc323aa: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:45.364388 22533 tablet_copy_client.cc:647] T 4782de9826d6460e9e993e01327034e4 P d43ed6b8663947d99a72cd922cc323aa: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:45.365712 22533 tablet_copy_client.cc:523] T 4782de9826d6460e9e993e01327034e4 P d43ed6b8663947d99a72cd922cc323aa: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:45.366766 22533 tablet_bootstrap.cc:492] T 4782de9826d6460e9e993e01327034e4 P d43ed6b8663947d99a72cd922cc323aa: Bootstrap starting.
I20240503 02:00:45.397532 22560 tablet_bootstrap.cc:492] T fbec07350ad04c49afd42e5d88dda6ad P c843193b801b4f1a88f630ed3ed0f375: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:45.397861 22560 tablet_bootstrap.cc:492] T fbec07350ad04c49afd42e5d88dda6ad P c843193b801b4f1a88f630ed3ed0f375: Bootstrap complete.
I20240503 02:00:45.398197 22560 ts_tablet_manager.cc:1367] T fbec07350ad04c49afd42e5d88dda6ad P c843193b801b4f1a88f630ed3ed0f375: Time spent bootstrapping tablet: real 0.058s user 0.032s sys 0.000s
I20240503 02:00:45.398723 22560 raft_consensus.cc:348] T fbec07350ad04c49afd42e5d88dda6ad P c843193b801b4f1a88f630ed3ed0f375 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { promote: true } }
I20240503 02:00:45.398887 22560 raft_consensus.cc:729] T fbec07350ad04c49afd42e5d88dda6ad P c843193b801b4f1a88f630ed3ed0f375 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c843193b801b4f1a88f630ed3ed0f375, State: Initialized, Role: LEARNER
I20240503 02:00:45.399010 22560 consensus_queue.cc:260] T fbec07350ad04c49afd42e5d88dda6ad P c843193b801b4f1a88f630ed3ed0f375 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { promote: true } }
I20240503 02:00:45.399377 22560 ts_tablet_manager.cc:1398] T fbec07350ad04c49afd42e5d88dda6ad P c843193b801b4f1a88f630ed3ed0f375: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:45.400815 17508 tablet_copy_service.cc:342] P 16d437fe629643948c95d628926d5ffa: Request end of tablet copy session c843193b801b4f1a88f630ed3ed0f375-fbec07350ad04c49afd42e5d88dda6ad received from {username='slave'} at 127.0.99.134:57603
I20240503 02:00:45.400913 17508 tablet_copy_service.cc:433] P 16d437fe629643948c95d628926d5ffa: ending tablet copy session c843193b801b4f1a88f630ed3ed0f375-fbec07350ad04c49afd42e5d88dda6ad on tablet fbec07350ad04c49afd42e5d88dda6ad with peer c843193b801b4f1a88f630ed3ed0f375
I20240503 02:00:45.415656 22533 tablet_bootstrap.cc:492] T 4782de9826d6460e9e993e01327034e4 P d43ed6b8663947d99a72cd922cc323aa: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:45.416023 22533 tablet_bootstrap.cc:492] T 4782de9826d6460e9e993e01327034e4 P d43ed6b8663947d99a72cd922cc323aa: Bootstrap complete.
I20240503 02:00:45.416224 22533 ts_tablet_manager.cc:1367] T 4782de9826d6460e9e993e01327034e4 P d43ed6b8663947d99a72cd922cc323aa: Time spent bootstrapping tablet: real 0.049s user 0.029s sys 0.002s
I20240503 02:00:45.416918 22533 raft_consensus.cc:348] T 4782de9826d6460e9e993e01327034e4 P d43ed6b8663947d99a72cd922cc323aa [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } attrs { replace: true } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { promote: true } }
I20240503 02:00:45.417073 22533 raft_consensus.cc:729] T 4782de9826d6460e9e993e01327034e4 P d43ed6b8663947d99a72cd922cc323aa [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d43ed6b8663947d99a72cd922cc323aa, State: Initialized, Role: LEARNER
I20240503 02:00:45.417222 22533 consensus_queue.cc:260] T 4782de9826d6460e9e993e01327034e4 P d43ed6b8663947d99a72cd922cc323aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } attrs { replace: true } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { promote: true } }
I20240503 02:00:45.417583 22533 ts_tablet_manager.cc:1398] T 4782de9826d6460e9e993e01327034e4 P d43ed6b8663947d99a72cd922cc323aa: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20240503 02:00:45.418516 17244 tablet_copy_service.cc:342] P 5b8291b26a7f448b8636187d1caf86d0: Request end of tablet copy session d43ed6b8663947d99a72cd922cc323aa-4782de9826d6460e9e993e01327034e4 received from {username='slave'} at 127.0.99.132:39351
I20240503 02:00:45.418601 17244 tablet_copy_service.cc:433] P 5b8291b26a7f448b8636187d1caf86d0: ending tablet copy session d43ed6b8663947d99a72cd922cc323aa-4782de9826d6460e9e993e01327034e4 on tablet 4782de9826d6460e9e993e01327034e4 with peer d43ed6b8663947d99a72cd922cc323aa
I20240503 02:00:45.489341 16958 raft_consensus.cc:1203] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.11->[2.12-2.12] Dedup: 2.12->[]
I20240503 02:00:45.534641 17353 raft_consensus.cc:1203] T 42a900125241432a8bb31dad0b940968 P d43ed6b8663947d99a72cd922cc323aa [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
W20240503 02:00:45.570843 17554 connection.cc:535] client connection to 127.0.99.137:41887 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:00:45.571004 17554 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:00:45.571982 17422 connection.cc:535] client connection to 127.0.99.137:41887 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:00:45.572091 17026 connection.cc:535] client connection to 127.0.99.137:41887 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:00:45.572186 17422 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:00:45.572209 17026 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:00:45.572615 17290 connection.cc:535] client connection to 127.0.99.137:41887 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:00:45.572733 17290 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:00:45.573144 17686 connection.cc:535] client connection to 127.0.99.137:41887 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:00:45.573274 17686 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:00:45.574507 17686 consensus_peers.cc:485] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:45.575898 17686 consensus_peers.cc:462] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111)
W20240503 02:00:45.579490 17290 consensus_peers.cc:485] T a39f284167e942188c181147e8a9e74e P d43ed6b8663947d99a72cd922cc323aa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:45.582345 17554 consensus_peers.cc:485] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:45.582540 17290 consensus_peers.cc:485] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:45.580618 22777 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 02:00:45.573107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.99.137:41887 (local address 127.0.99.133:48331)
0503 02:00:45.579830 (+ 6723us) negotiation.cc:107] Waiting for socket to connect
0503 02:00:45.579921 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":4256,"thread_start_us":95,"threads_started":1}
W20240503 02:00:45.582739 17422 consensus_peers.cc:485] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:45.583133 17026 consensus_peers.cc:485] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:45.583195 17026 consensus_peers.cc:485] T e245c9cf642043cfbb3b87114beeb002 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:45.583245 17026 consensus_peers.cc:485] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:45.583277 17026 consensus_peers.cc:485] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:45.597410 17222 raft_consensus.cc:1203] T 63962b169caf4f14888871e246c00075 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.10->[1.11-1.11] Dedup: 1.11->[]
W20240503 02:00:45.599401 19173 connection.cc:535] client connection to 127.0.99.136:40167 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:00:45.599457 17554 connection.cc:535] client connection to 127.0.99.136:40167 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:00:45.599577 19173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:00:45.599844 17422 connection.cc:535] client connection to 127.0.99.136:40167 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:00:45.600112 17686 connection.cc:535] client connection to 127.0.99.136:40167 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:00:45.600459 17026 connection.cc:535] client connection to 127.0.99.136:40167 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:00:45.600601 17158 connection.cc:535] client connection to 127.0.99.136:40167 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:00:45.600749 17158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:00:45.601655 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:00:45.601714 17422 consensus_peers.cc:485] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:45.603191 19173 consensus_peers.cc:485] T 885fe0c3a5e04f408e5b02c3c133248a P 9564311af29a4493831a8db658a085c4 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:45.604295 17620 raft_consensus.cc:1203] T 30abfada8ab44449b9a4237c57093b73 P c843193b801b4f1a88f630ed3ed0f375 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
W20240503 02:00:45.605773 17554 consensus_peers.cc:485] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:45.605980 17686 consensus_peers.cc:485] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:45.606067 17686 consensus_peers.cc:485] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:45.606207 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:00:45.606272 17026 consensus_peers.cc:485] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:45.606314 17158 consensus_peers.cc:485] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:45.648242 17686 consensus_peers.cc:462] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111)
W20240503 02:00:45.658658 17290 consensus_peers.cc:485] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:45.659530 17422 consensus_peers.cc:485] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:45.678181 17026 consensus_peers.cc:485] T ae4259dd3dc745869256269632886f8a P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:45.684535 17422 consensus_peers.cc:485] T 42a900125241432a8bb31dad0b940968 P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:45.690229 17026 consensus_peers.cc:485] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:45.691514 17486 raft_consensus.cc:1203] T a39f284167e942188c181147e8a9e74e P 16d437fe629643948c95d628926d5ffa [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
W20240503 02:00:45.698307 17422 consensus_peers.cc:485] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:45.699615 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:00:45.700325 17158 consensus_peers.cc:485] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:45.709887 17422 consensus_peers.cc:485] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:45.740888 17353 raft_consensus.cc:1203] T 4782de9826d6460e9e993e01327034e4 P d43ed6b8663947d99a72cd922cc323aa [term 1 LEARNER]: Deduplicated request from leader. Original: 1.10->[1.11-1.11] Dedup: 1.11->[]
W20240503 02:00:45.767563 17422 consensus_peers.cc:485] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:45.778857 17356 raft_consensus.cc:1203] T 78d106d705df4089a4dfae4d5dc533cc P d43ed6b8663947d99a72cd922cc323aa [term 1 LEARNER]: Deduplicated request from leader. Original: 1.10->[1.11-1.11] Dedup: 1.11->[]
I20240503 02:00:45.795804 17356 raft_consensus.cc:1203] T fdf0a4b9e17642849a4094d2f0b9a255 P d43ed6b8663947d99a72cd922cc323aa [term 1 LEARNER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
W20240503 02:00:45.803299 17026 consensus_peers.cc:485] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:45.831895 17422 consensus_peers.cc:485] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:45.860304 17620 raft_consensus.cc:1203] T fbec07350ad04c49afd42e5d88dda6ad P c843193b801b4f1a88f630ed3ed0f375 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.10->[1.11-1.11] Dedup: 1.11->[]
I20240503 02:00:45.948012 22297 raft_consensus.cc:1050] T 42a900125241432a8bb31dad0b940968 P 16d437fe629643948c95d628926d5ffa: attempting to promote NON_VOTER d43ed6b8663947d99a72cd922cc323aa to VOTER
I20240503 02:00:45.948617 22297 consensus_queue.cc:237] T 42a900125241432a8bb31dad0b940968 P 16d437fe629643948c95d628926d5ffa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } attrs { replace: true } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { promote: false } }
I20240503 02:00:45.951203 16958 raft_consensus.cc:1261] T 42a900125241432a8bb31dad0b940968 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:45.951328 17620 raft_consensus.cc:1261] T 42a900125241432a8bb31dad0b940968 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:45.951758 17092 raft_consensus.cc:1261] T 42a900125241432a8bb31dad0b940968 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:45.951715 22932 consensus_queue.cc:1035] T 42a900125241432a8bb31dad0b940968 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:45.952196 22297 consensus_queue.cc:1035] T 42a900125241432a8bb31dad0b940968 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:45.953064 17222 raft_consensus.cc:1261] T 42a900125241432a8bb31dad0b940968 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:45.953210 22297 consensus_queue.cc:1035] T 42a900125241432a8bb31dad0b940968 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:45.953512 17752 raft_consensus.cc:1261] T 42a900125241432a8bb31dad0b940968 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:45.954012 22933 consensus_queue.cc:1035] T 42a900125241432a8bb31dad0b940968 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:45.954123 17356 raft_consensus.cc:1261] T 42a900125241432a8bb31dad0b940968 P d43ed6b8663947d99a72cd922cc323aa [term 1 LEARNER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
W20240503 02:00:45.954260 17422 consensus_peers.cc:485] T 42a900125241432a8bb31dad0b940968 P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:45.954349 22933 consensus_queue.cc:1035] T 42a900125241432a8bb31dad0b940968 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:45.955547 22933 consensus_queue.cc:1035] T 42a900125241432a8bb31dad0b940968 P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:45.957844 22297 raft_consensus.cc:2932] T 42a900125241432a8bb31dad0b940968 P 16d437fe629643948c95d628926d5ffa [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } attrs { replace: true } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { promote: false } } }
I20240503 02:00:45.958271 17092 raft_consensus.cc:2932] T 42a900125241432a8bb31dad0b940968 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } attrs { replace: true } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { promote: false } } }
I20240503 02:00:45.958271 17356 raft_consensus.cc:2932] T 42a900125241432a8bb31dad0b940968 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } attrs { replace: true } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { promote: false } } }
I20240503 02:00:45.958585 17752 raft_consensus.cc:2932] T 42a900125241432a8bb31dad0b940968 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } attrs { replace: true } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { promote: false } } }
I20240503 02:00:45.960914 16958 raft_consensus.cc:2932] T 42a900125241432a8bb31dad0b940968 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } attrs { replace: true } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { promote: false } } }
I20240503 02:00:45.961810 16813 catalog_manager.cc:5484] T 42a900125241432a8bb31dad0b940968 P d43ed6b8663947d99a72cd922cc323aa reported cstate change: config changed from index 8 to 9, d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "16d437fe629643948c95d628926d5ffa" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } attrs { replace: true } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { promote: false } } }
I20240503 02:00:45.963343 17222 raft_consensus.cc:2932] T 42a900125241432a8bb31dad0b940968 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } attrs { replace: true } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { promote: false } } }
I20240503 02:00:45.966012 17620 raft_consensus.cc:2932] T 42a900125241432a8bb31dad0b940968 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } attrs { replace: true } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { promote: false } } }
I20240503 02:00:45.979121 22309 raft_consensus.cc:1050] T 885fe0c3a5e04f408e5b02c3c133248a P 9564311af29a4493831a8db658a085c4: attempting to promote NON_VOTER 6398c8993c3c45dcaab25b42142d5671 to VOTER
I20240503 02:00:45.979849 22309 consensus_queue.cc:237] T 885fe0c3a5e04f408e5b02c3c133248a P 9564311af29a4493831a8db658a085c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 9, Current term: 2, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } attrs { replace: true } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { promote: false } }
I20240503 02:00:45.982152 16958 raft_consensus.cc:1261] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671 [term 2 LEARNER]: Refusing update from remote peer 9564311af29a4493831a8db658a085c4: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240503 02:00:45.982600 17222 raft_consensus.cc:1261] T 885fe0c3a5e04f408e5b02c3c133248a P 5b8291b26a7f448b8636187d1caf86d0 [term 2 FOLLOWER]: Refusing update from remote peer 9564311af29a4493831a8db658a085c4: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240503 02:00:45.982688 22949 consensus_queue.cc:1035] T 885fe0c3a5e04f408e5b02c3c133248a P 9564311af29a4493831a8db658a085c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 02:00:45.983047 17486 raft_consensus.cc:1261] T 885fe0c3a5e04f408e5b02c3c133248a P 16d437fe629643948c95d628926d5ffa [term 2 FOLLOWER]: Refusing update from remote peer 9564311af29a4493831a8db658a085c4: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240503 02:00:45.983325 17620 raft_consensus.cc:1261] T 885fe0c3a5e04f408e5b02c3c133248a P c843193b801b4f1a88f630ed3ed0f375 [term 2 FOLLOWER]: Refusing update from remote peer 9564311af29a4493831a8db658a085c4: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240503 02:00:45.983462 22949 consensus_queue.cc:1035] T 885fe0c3a5e04f408e5b02c3c133248a P 9564311af29a4493831a8db658a085c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 02:00:45.983788 22948 consensus_queue.cc:1035] T 885fe0c3a5e04f408e5b02c3c133248a P 9564311af29a4493831a8db658a085c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 02:00:45.983847 17092 raft_consensus.cc:1261] T 885fe0c3a5e04f408e5b02c3c133248a P 89dd09f8aaee4b249ae3ce924a422a67 [term 2 FOLLOWER]: Refusing update from remote peer 9564311af29a4493831a8db658a085c4: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240503 02:00:45.984365 17752 raft_consensus.cc:1261] T 885fe0c3a5e04f408e5b02c3c133248a P 002cc3411acb4fdfa0b67c6f57f687a6 [term 2 FOLLOWER]: Refusing update from remote peer 9564311af29a4493831a8db658a085c4: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240503 02:00:45.984845 22949 consensus_queue.cc:1035] T 885fe0c3a5e04f408e5b02c3c133248a P 9564311af29a4493831a8db658a085c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 02:00:45.985252 22288 consensus_queue.cc:1035] T 885fe0c3a5e04f408e5b02c3c133248a P 9564311af29a4493831a8db658a085c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 02:00:45.986092 22288 consensus_queue.cc:1035] T 885fe0c3a5e04f408e5b02c3c133248a P 9564311af29a4493831a8db658a085c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20240503 02:00:45.986603 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
I20240503 02:00:45.987712 22288 raft_consensus.cc:2932] T 885fe0c3a5e04f408e5b02c3c133248a P 9564311af29a4493831a8db658a085c4 [term 2 LEADER]: Committing config change with OpId 2.13: config changed from index 12 to 13, 6398c8993c3c45dcaab25b42142d5671 (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } attrs { replace: true } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { promote: false } } }
I20240503 02:00:45.988036 17620 raft_consensus.cc:2932] T 885fe0c3a5e04f408e5b02c3c133248a P c843193b801b4f1a88f630ed3ed0f375 [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, 6398c8993c3c45dcaab25b42142d5671 (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } attrs { replace: true } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { promote: false } } }
W20240503 02:00:45.988515 19173 consensus_peers.cc:485] T 885fe0c3a5e04f408e5b02c3c133248a P 9564311af29a4493831a8db658a085c4 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:45.989941 16958 raft_consensus.cc:2932] T 885fe0c3a5e04f408e5b02c3c133248a P 6398c8993c3c45dcaab25b42142d5671 [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, 6398c8993c3c45dcaab25b42142d5671 (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } attrs { replace: true } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { promote: false } } }
I20240503 02:00:45.991276 17486 raft_consensus.cc:2932] T 885fe0c3a5e04f408e5b02c3c133248a P 16d437fe629643948c95d628926d5ffa [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, 6398c8993c3c45dcaab25b42142d5671 (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } attrs { replace: true } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { promote: false } } }
I20240503 02:00:45.992445 16817 catalog_manager.cc:5484] T 885fe0c3a5e04f408e5b02c3c133248a P 9564311af29a4493831a8db658a085c4 reported cstate change: config changed from index 12 to 13, 6398c8993c3c45dcaab25b42142d5671 (127.0.99.129) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "9564311af29a4493831a8db658a085c4" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:00:45.992873 17092 raft_consensus.cc:2932] T 885fe0c3a5e04f408e5b02c3c133248a P 89dd09f8aaee4b249ae3ce924a422a67 [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, 6398c8993c3c45dcaab25b42142d5671 (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } attrs { replace: true } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { promote: false } } }
I20240503 02:00:45.998179 17222 raft_consensus.cc:2932] T 885fe0c3a5e04f408e5b02c3c133248a P 5b8291b26a7f448b8636187d1caf86d0 [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, 6398c8993c3c45dcaab25b42142d5671 (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } attrs { replace: true } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { promote: false } } }
I20240503 02:00:46.010561 17752 raft_consensus.cc:2932] T 885fe0c3a5e04f408e5b02c3c133248a P 002cc3411acb4fdfa0b67c6f57f687a6 [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, 6398c8993c3c45dcaab25b42142d5671 (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } attrs { replace: true } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { promote: false } } }
I20240503 02:00:46.130558 22355 raft_consensus.cc:1050] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa: attempting to promote NON_VOTER 5b8291b26a7f448b8636187d1caf86d0 to VOTER
I20240503 02:00:46.131441 22355 consensus_queue.cc:237] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { promote: false } }
I20240503 02:00:46.135866 17752 raft_consensus.cc:1261] T 63962b169caf4f14888871e246c00075 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 02:00:46.136466 19237 raft_consensus.cc:1261] T 63962b169caf4f14888871e246c00075 P 9564311af29a4493831a8db658a085c4 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 02:00:46.136452 22355 consensus_queue.cc:1035] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:00:46.136943 17092 raft_consensus.cc:1261] T 63962b169caf4f14888871e246c00075 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 02:00:46.137487 16958 raft_consensus.cc:1261] T 63962b169caf4f14888871e246c00075 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 02:00:46.137960 17486 raft_consensus.cc:1261] T 63962b169caf4f14888871e246c00075 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 02:00:46.138689 22629 consensus_queue.cc:1035] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:00:46.139056 22629 consensus_queue.cc:1035] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:00:46.140791 22629 consensus_queue.cc:1035] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:00:46.141115 22629 consensus_queue.cc:1035] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
W20240503 02:00:46.141242 17290 consensus_peers.cc:485] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:46.141959 17222 raft_consensus.cc:1261] T 63962b169caf4f14888871e246c00075 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 LEARNER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 02:00:46.142554 22797 consensus_queue.cc:1035] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:00:46.144868 22979 raft_consensus.cc:2932] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 5b8291b26a7f448b8636187d1caf86d0 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { promote: false } } }
I20240503 02:00:46.147677 19237 raft_consensus.cc:2932] T 63962b169caf4f14888871e246c00075 P 9564311af29a4493831a8db658a085c4 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 5b8291b26a7f448b8636187d1caf86d0 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { promote: false } } }
I20240503 02:00:46.147744 17222 raft_consensus.cc:2932] T 63962b169caf4f14888871e246c00075 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 5b8291b26a7f448b8636187d1caf86d0 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { promote: false } } }
I20240503 02:00:46.147804 16958 raft_consensus.cc:2932] T 63962b169caf4f14888871e246c00075 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 5b8291b26a7f448b8636187d1caf86d0 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { promote: false } } }
I20240503 02:00:46.151110 16818 catalog_manager.cc:5484] T 63962b169caf4f14888871e246c00075 P 5b8291b26a7f448b8636187d1caf86d0 reported cstate change: config changed from index 11 to 12, 5b8291b26a7f448b8636187d1caf86d0 (127.0.99.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d43ed6b8663947d99a72cd922cc323aa" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { promote: false } } }
I20240503 02:00:46.152036 17486 raft_consensus.cc:2932] T 63962b169caf4f14888871e246c00075 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 5b8291b26a7f448b8636187d1caf86d0 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { promote: false } } }
I20240503 02:00:46.152204 17092 raft_consensus.cc:2932] T 63962b169caf4f14888871e246c00075 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 5b8291b26a7f448b8636187d1caf86d0 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { promote: false } } }
I20240503 02:00:46.155879 17752 raft_consensus.cc:2932] T 63962b169caf4f14888871e246c00075 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 5b8291b26a7f448b8636187d1caf86d0 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } attrs { replace: true } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } attrs { promote: false } } }
W20240503 02:00:46.159183 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
I20240503 02:00:46.163822 22355 raft_consensus.cc:1050] T a39f284167e942188c181147e8a9e74e P d43ed6b8663947d99a72cd922cc323aa: attempting to promote NON_VOTER 16d437fe629643948c95d628926d5ffa to VOTER
I20240503 02:00:46.164419 22355 consensus_queue.cc:237] T a39f284167e942188c181147e8a9e74e P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } attrs { replace: true } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { promote: false } }
I20240503 02:00:46.166734 16958 raft_consensus.cc:1261] T a39f284167e942188c181147e8a9e74e P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:46.166977 17222 raft_consensus.cc:1261] T a39f284167e942188c181147e8a9e74e P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:46.167269 22797 consensus_queue.cc:1035] T a39f284167e942188c181147e8a9e74e P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:46.167670 17752 raft_consensus.cc:1261] T a39f284167e942188c181147e8a9e74e P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:46.167665 22797 consensus_queue.cc:1035] T a39f284167e942188c181147e8a9e74e P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:46.167809 17620 raft_consensus.cc:1261] T a39f284167e942188c181147e8a9e74e P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
W20240503 02:00:46.168198 17290 consensus_peers.cc:485] T a39f284167e942188c181147e8a9e74e P d43ed6b8663947d99a72cd922cc323aa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:46.168344 22976 consensus_queue.cc:1035] T a39f284167e942188c181147e8a9e74e P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:46.168776 22979 consensus_queue.cc:1035] T a39f284167e942188c181147e8a9e74e P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:46.169396 17092 raft_consensus.cc:1261] T a39f284167e942188c181147e8a9e74e P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:46.172302 22976 consensus_queue.cc:1035] T a39f284167e942188c181147e8a9e74e P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:46.176731 17486 raft_consensus.cc:1261] T a39f284167e942188c181147e8a9e74e P 16d437fe629643948c95d628926d5ffa [term 1 LEARNER]: Refusing update from remote peer d43ed6b8663947d99a72cd922cc323aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:46.177281 22797 consensus_queue.cc:1035] T a39f284167e942188c181147e8a9e74e P d43ed6b8663947d99a72cd922cc323aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:46.180887 22344 raft_consensus.cc:1050] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67: attempting to promote NON_VOTER c843193b801b4f1a88f630ed3ed0f375 to VOTER
I20240503 02:00:46.181582 22344 consensus_queue.cc:237] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } attrs { replace: true } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false replace: true } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { promote: false } }
I20240503 02:00:46.182833 22977 raft_consensus.cc:2932] T a39f284167e942188c181147e8a9e74e P d43ed6b8663947d99a72cd922cc323aa [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 16d437fe629643948c95d628926d5ffa (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } attrs { replace: true } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { promote: false } } }
I20240503 02:00:46.185213 16957 raft_consensus.cc:1261] T 30abfada8ab44449b9a4237c57093b73 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:00:46.186117 17487 raft_consensus.cc:1261] T 30abfada8ab44449b9a4237c57093b73 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:00:46.186436 17486 raft_consensus.cc:2932] T a39f284167e942188c181147e8a9e74e P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 16d437fe629643948c95d628926d5ffa (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } attrs { replace: true } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { promote: false } } }
I20240503 02:00:46.186687 17220 raft_consensus.cc:1261] T 30abfada8ab44449b9a4237c57093b73 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:00:46.186563 17092 raft_consensus.cc:2932] T a39f284167e942188c181147e8a9e74e P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 16d437fe629643948c95d628926d5ffa (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } attrs { replace: true } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { promote: false } } }
I20240503 02:00:46.187180 17356 raft_consensus.cc:1261] T 30abfada8ab44449b9a4237c57093b73 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:00:46.187932 22271 consensus_queue.cc:1035] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:00:46.188856 17752 raft_consensus.cc:1261] T 30abfada8ab44449b9a4237c57093b73 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:00:46.189713 17620 raft_consensus.cc:1261] T 30abfada8ab44449b9a4237c57093b73 P c843193b801b4f1a88f630ed3ed0f375 [term 1 LEARNER]: Refusing update from remote peer 89dd09f8aaee4b249ae3ce924a422a67: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:00:46.190419 22345 consensus_queue.cc:1035] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:00:46.190711 22345 consensus_queue.cc:1035] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:00:46.191179 17620 raft_consensus.cc:2932] T a39f284167e942188c181147e8a9e74e P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 16d437fe629643948c95d628926d5ffa (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } attrs { replace: true } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { promote: false } } }
I20240503 02:00:46.191869 16810 catalog_manager.cc:5484] T a39f284167e942188c181147e8a9e74e P d43ed6b8663947d99a72cd922cc323aa reported cstate change: config changed from index 8 to 9, 16d437fe629643948c95d628926d5ffa (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d43ed6b8663947d99a72cd922cc323aa" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:00:46.194711 22271 consensus_queue.cc:1035] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:00:46.194967 22345 consensus_queue.cc:1035] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:00:46.195267 22345 consensus_queue.cc:1035] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240503 02:00:46.199220 17026 consensus_peers.cc:485] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:46.199324 17026 consensus_peers.cc:485] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:46.200959 17751 raft_consensus.cc:2932] T a39f284167e942188c181147e8a9e74e P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 16d437fe629643948c95d628926d5ffa (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } attrs { replace: true } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { promote: false } } }
I20240503 02:00:46.202746 16957 raft_consensus.cc:2932] T a39f284167e942188c181147e8a9e74e P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 16d437fe629643948c95d628926d5ffa (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } attrs { replace: true } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { promote: false } } }
I20240503 02:00:46.207286 17619 raft_consensus.cc:2932] T 30abfada8ab44449b9a4237c57093b73 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } attrs { replace: true } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false replace: true } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { promote: false } } }
I20240503 02:00:46.208401 23019 raft_consensus.cc:2932] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } attrs { replace: true } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false replace: true } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { promote: false } } }
W20240503 02:00:46.209219 17686 consensus_peers.cc:462] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111)
I20240503 02:00:46.209755 17752 raft_consensus.cc:2932] T 30abfada8ab44449b9a4237c57093b73 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } attrs { replace: true } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false replace: true } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { promote: false } } }
I20240503 02:00:46.210107 16818 catalog_manager.cc:5484] T 30abfada8ab44449b9a4237c57093b73 P c843193b801b4f1a88f630ed3ed0f375 reported cstate change: config changed from index 9 to 10, c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "89dd09f8aaee4b249ae3ce924a422a67" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } attrs { replace: true } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false replace: true } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { promote: false } } }
I20240503 02:00:46.211800 17356 raft_consensus.cc:2932] T 30abfada8ab44449b9a4237c57093b73 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } attrs { replace: true } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false replace: true } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { promote: false } } }
I20240503 02:00:46.215781 17220 raft_consensus.cc:2932] T 30abfada8ab44449b9a4237c57093b73 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } attrs { replace: true } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false replace: true } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { promote: false } } }
I20240503 02:00:46.216292 17222 raft_consensus.cc:2932] T a39f284167e942188c181147e8a9e74e P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 16d437fe629643948c95d628926d5ffa (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } attrs { replace: true } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { promote: false } } }
I20240503 02:00:46.223688 22863 raft_consensus.cc:1050] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6: attempting to promote NON_VOTER d43ed6b8663947d99a72cd922cc323aa to VOTER
I20240503 02:00:46.224135 16957 raft_consensus.cc:2932] T 30abfada8ab44449b9a4237c57093b73 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } attrs { replace: true } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false replace: true } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { promote: false } } }
I20240503 02:00:46.224381 22863 consensus_queue.cc:237] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false replace: true } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { promote: false } }
I20240503 02:00:46.227830 17092 raft_consensus.cc:1261] T fdf0a4b9e17642849a4094d2f0b9a255 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:00:46.229352 22860 consensus_queue.cc:1035] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:00:46.228636 17356 raft_consensus.cc:1261] T fdf0a4b9e17642849a4094d2f0b9a255 P d43ed6b8663947d99a72cd922cc323aa [term 1 LEARNER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:00:46.231000 17487 raft_consensus.cc:1261] T fdf0a4b9e17642849a4094d2f0b9a255 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:00:46.231410 22860 consensus_queue.cc:1035] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:00:46.232422 17619 raft_consensus.cc:1261] T fdf0a4b9e17642849a4094d2f0b9a255 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:00:46.232427 22860 consensus_queue.cc:1035] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:00:46.232944 22860 consensus_queue.cc:1035] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:00:46.233642 17486 raft_consensus.cc:2932] T 30abfada8ab44449b9a4237c57093b73 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } attrs { replace: true } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false replace: true } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { promote: false } } }
I20240503 02:00:46.234458 17222 raft_consensus.cc:1261] T fdf0a4b9e17642849a4094d2f0b9a255 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:00:46.234943 16957 raft_consensus.cc:1261] T fdf0a4b9e17642849a4094d2f0b9a255 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:00:46.235499 22141 consensus_queue.cc:1035] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:00:46.236666 22863 consensus_queue.cc:1035] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240503 02:00:46.238555 17686 consensus_peers.cc:485] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:46.238715 17686 consensus_peers.cc:485] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:46.239315 22860 raft_consensus.cc:2932] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false replace: true } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { promote: false } } }
I20240503 02:00:46.241205 17619 raft_consensus.cc:2932] T fdf0a4b9e17642849a4094d2f0b9a255 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false replace: true } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { promote: false } } }
I20240503 02:00:46.241612 17356 raft_consensus.cc:2932] T fdf0a4b9e17642849a4094d2f0b9a255 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false replace: true } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { promote: false } } }
I20240503 02:00:46.242262 17092 raft_consensus.cc:2932] T fdf0a4b9e17642849a4094d2f0b9a255 P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false replace: true } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { promote: false } } }
I20240503 02:00:46.242375 16957 raft_consensus.cc:2932] T fdf0a4b9e17642849a4094d2f0b9a255 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false replace: true } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { promote: false } } }
I20240503 02:00:46.244238 16814 catalog_manager.cc:5484] T fdf0a4b9e17642849a4094d2f0b9a255 P c843193b801b4f1a88f630ed3ed0f375 reported cstate change: config changed from index 9 to 10, d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false replace: true } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { promote: false } } }
I20240503 02:00:46.246901 17486 raft_consensus.cc:2932] T fdf0a4b9e17642849a4094d2f0b9a255 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false replace: true } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { promote: false } } }
I20240503 02:00:46.253604 17222 raft_consensus.cc:2932] T fdf0a4b9e17642849a4094d2f0b9a255 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false replace: true } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: true } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { promote: false } } }
I20240503 02:00:46.304026 22862 raft_consensus.cc:1050] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0: attempting to promote NON_VOTER d43ed6b8663947d99a72cd922cc323aa to VOTER
I20240503 02:00:46.304798 22862 consensus_queue.cc:237] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } attrs { replace: true } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { promote: false } }
I20240503 02:00:46.307392 16957 raft_consensus.cc:1261] T 4782de9826d6460e9e993e01327034e4 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 5b8291b26a7f448b8636187d1caf86d0: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 02:00:46.307469 17619 raft_consensus.cc:1261] T 4782de9826d6460e9e993e01327034e4 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer 5b8291b26a7f448b8636187d1caf86d0: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 02:00:46.307950 17486 raft_consensus.cc:1261] T 4782de9826d6460e9e993e01327034e4 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Refusing update from remote peer 5b8291b26a7f448b8636187d1caf86d0: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 02:00:46.308388 22861 consensus_queue.cc:1035] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:00:46.308621 17752 raft_consensus.cc:1261] T 4782de9826d6460e9e993e01327034e4 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer 5b8291b26a7f448b8636187d1caf86d0: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 02:00:46.308782 22861 consensus_queue.cc:1035] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:00:46.309536 22473 consensus_queue.cc:1035] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:00:46.309818 22473 consensus_queue.cc:1035] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:00:46.313288 22473 raft_consensus.cc:2932] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } attrs { replace: true } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { promote: false } } }
I20240503 02:00:46.314375 17356 raft_consensus.cc:1261] T 4782de9826d6460e9e993e01327034e4 P d43ed6b8663947d99a72cd922cc323aa [term 1 LEARNER]: Refusing update from remote peer 5b8291b26a7f448b8636187d1caf86d0: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 02:00:46.315001 22473 consensus_queue.cc:1035] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:00:46.315944 16957 raft_consensus.cc:2932] T 4782de9826d6460e9e993e01327034e4 P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } attrs { replace: true } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { promote: false } } }
I20240503 02:00:46.316147 17752 raft_consensus.cc:2932] T 4782de9826d6460e9e993e01327034e4 P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } attrs { replace: true } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { promote: false } } }
I20240503 02:00:46.316613 17486 raft_consensus.cc:2932] T 4782de9826d6460e9e993e01327034e4 P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } attrs { replace: true } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { promote: false } } }
I20240503 02:00:46.316994 16817 catalog_manager.cc:5484] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 reported cstate change: config changed from index 11 to 12, d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5b8291b26a7f448b8636187d1caf86d0" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:46.317741 17620 raft_consensus.cc:2932] T 4782de9826d6460e9e993e01327034e4 P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } attrs { replace: true } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { promote: false } } }
W20240503 02:00:46.319315 17158 consensus_peers.cc:485] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:46.319419 17158 consensus_peers.cc:485] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:46.320489 17353 raft_consensus.cc:2932] T 4782de9826d6460e9e993e01327034e4 P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } attrs { replace: true } } peers { permanent_uuid: "375d0f2ec11d4e25bd9d093bb6a5572d" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 40167 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { promote: false } } }
I20240503 02:00:46.323908 23038 raft_consensus.cc:1050] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6: attempting to promote NON_VOTER d43ed6b8663947d99a72cd922cc323aa to VOTER
I20240503 02:00:46.324515 23038 consensus_queue.cc:237] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { promote: false } }
I20240503 02:00:46.326856 19237 raft_consensus.cc:1261] T 78d106d705df4089a4dfae4d5dc533cc P 9564311af29a4493831a8db658a085c4 [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 02:00:46.326951 17620 raft_consensus.cc:1261] T 78d106d705df4089a4dfae4d5dc533cc P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 02:00:46.327049 17092 raft_consensus.cc:1261] T 78d106d705df4089a4dfae4d5dc533cc P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 02:00:46.327616 23034 consensus_queue.cc:1035] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:00:46.327972 23037 consensus_queue.cc:1035] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:00:46.328161 23036 consensus_queue.cc:1035] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:00:46.331645 17222 raft_consensus.cc:1261] T 78d106d705df4089a4dfae4d5dc533cc P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 02:00:46.331776 17353 raft_consensus.cc:1261] T 78d106d705df4089a4dfae4d5dc533cc P d43ed6b8663947d99a72cd922cc323aa [term 1 LEARNER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 02:00:46.332280 23036 consensus_queue.cc:1035] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:00:46.332417 23034 consensus_queue.cc:1035] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
W20240503 02:00:46.333139 17686 consensus_peers.cc:485] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:46.334717 17486 raft_consensus.cc:1261] T 78d106d705df4089a4dfae4d5dc533cc P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Refusing update from remote peer 002cc3411acb4fdfa0b67c6f57f687a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 02:00:46.334710 23036 raft_consensus.cc:2932] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { promote: false } } }
I20240503 02:00:46.335191 23037 consensus_queue.cc:1035] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:00:46.336475 17486 raft_consensus.cc:2932] T 78d106d705df4089a4dfae4d5dc533cc P 16d437fe629643948c95d628926d5ffa [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { promote: false } } }
I20240503 02:00:46.337592 17353 raft_consensus.cc:2932] T 78d106d705df4089a4dfae4d5dc533cc P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { promote: false } } }
I20240503 02:00:46.339394 17620 raft_consensus.cc:2932] T 78d106d705df4089a4dfae4d5dc533cc P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { promote: false } } }
I20240503 02:00:46.339896 17092 raft_consensus.cc:2932] T 78d106d705df4089a4dfae4d5dc533cc P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { promote: false } } }
I20240503 02:00:46.340514 16814 catalog_manager.cc:5484] T 78d106d705df4089a4dfae4d5dc533cc P d43ed6b8663947d99a72cd922cc323aa reported cstate change: config changed from index 11 to 12, d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { promote: false } } }
I20240503 02:00:46.341849 17222 raft_consensus.cc:2932] T 78d106d705df4089a4dfae4d5dc533cc P 5b8291b26a7f448b8636187d1caf86d0 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { promote: false } } }
I20240503 02:00:46.342849 19237 raft_consensus.cc:2932] T 78d106d705df4089a4dfae4d5dc533cc P 9564311af29a4493831a8db658a085c4 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, d43ed6b8663947d99a72cd922cc323aa (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } attrs { replace: true } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "5b8291b26a7f448b8636187d1caf86d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44479 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } attrs { promote: false } } }
I20240503 02:00:46.347512 22934 raft_consensus.cc:1050] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa: attempting to promote NON_VOTER c843193b801b4f1a88f630ed3ed0f375 to VOTER
I20240503 02:00:46.348014 22934 consensus_queue.cc:237] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { promote: false } }
I20240503 02:00:46.349968 19237 raft_consensus.cc:1261] T fbec07350ad04c49afd42e5d88dda6ad P 9564311af29a4493831a8db658a085c4 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 02:00:46.350126 17092 raft_consensus.cc:1261] T fbec07350ad04c49afd42e5d88dda6ad P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 02:00:46.350461 17620 raft_consensus.cc:1261] T fbec07350ad04c49afd42e5d88dda6ad P c843193b801b4f1a88f630ed3ed0f375 [term 1 LEARNER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 02:00:46.350490 17353 raft_consensus.cc:1261] T fbec07350ad04c49afd42e5d88dda6ad P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 02:00:46.350903 22932 consensus_queue.cc:1035] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:00:46.351114 22298 consensus_queue.cc:1035] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:00:46.351200 22932 consensus_queue.cc:1035] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:00:46.351387 22272 consensus_queue.cc:1035] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
W20240503 02:00:46.351907 17422 consensus_peers.cc:485] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:46.352751 16957 raft_consensus.cc:1261] T fbec07350ad04c49afd42e5d88dda6ad P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 02:00:46.353216 22298 consensus_queue.cc:1035] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:00:46.354018 17752 raft_consensus.cc:1261] T fbec07350ad04c49afd42e5d88dda6ad P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Refusing update from remote peer 16d437fe629643948c95d628926d5ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 02:00:46.354483 22298 consensus_queue.cc:1035] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:00:46.355810 22935 raft_consensus.cc:2932] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { promote: false } } }
I20240503 02:00:46.356261 16957 raft_consensus.cc:2932] T fbec07350ad04c49afd42e5d88dda6ad P 6398c8993c3c45dcaab25b42142d5671 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { promote: false } } }
I20240503 02:00:46.356976 19237 raft_consensus.cc:2932] T fbec07350ad04c49afd42e5d88dda6ad P 9564311af29a4493831a8db658a085c4 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { promote: false } } }
I20240503 02:00:46.357997 17353 raft_consensus.cc:2932] T fbec07350ad04c49afd42e5d88dda6ad P d43ed6b8663947d99a72cd922cc323aa [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { promote: false } } }
I20240503 02:00:46.359189 17620 raft_consensus.cc:2932] T fbec07350ad04c49afd42e5d88dda6ad P c843193b801b4f1a88f630ed3ed0f375 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { promote: false } } }
I20240503 02:00:46.359651 16810 catalog_manager.cc:5484] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa reported cstate change: config changed from index 11 to 12, c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "16d437fe629643948c95d628926d5ffa" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:00:46.361922 17752 raft_consensus.cc:2932] T fbec07350ad04c49afd42e5d88dda6ad P 002cc3411acb4fdfa0b67c6f57f687a6 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { promote: false } } }
I20240503 02:00:46.361922 17092 raft_consensus.cc:2932] T fbec07350ad04c49afd42e5d88dda6ad P 89dd09f8aaee4b249ae3ce924a422a67 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, c843193b801b4f1a88f630ed3ed0f375 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "6398c8993c3c45dcaab25b42142d5671" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36983 } } peers { permanent_uuid: "89dd09f8aaee4b249ae3ce924a422a67" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38889 } } peers { permanent_uuid: "d43ed6b8663947d99a72cd922cc323aa" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40023 } } peers { permanent_uuid: "002cc3411acb4fdfa0b67c6f57f687a6" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38989 } attrs { replace: true } } peers { permanent_uuid: "16d437fe629643948c95d628926d5ffa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46581 } } peers { permanent_uuid: "9564311af29a4493831a8db658a085c4" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33181 } attrs { promote: false } } peers { permanent_uuid: "838ef968bb45484bb27f5af6f3989bae" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 41887 } attrs { promote: false } } peers { permanent_uuid: "c843193b801b4f1a88f630ed3ed0f375" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44343 } attrs { promote: false } } }
W20240503 02:00:46.565711 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:00:46.761721 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:00:47.141889 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:00:47.328702 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:00:47.607467 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:00:47.708427 17026 consensus_peers.cc:485] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:47.728816 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:00:47.831784 17554 consensus_peers.cc:485] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:47.837987 17026 consensus_peers.cc:485] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:47.848743 17686 consensus_peers.cc:485] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:47.856515 17026 consensus_peers.cc:485] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:47.878309 17554 consensus_peers.cc:485] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:47.880406 17026 consensus_peers.cc:485] T e245c9cf642043cfbb3b87114beeb002 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:47.901438 17026 consensus_peers.cc:485] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:47.968056 17422 consensus_peers.cc:485] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:47.981936 17422 consensus_peers.cc:485] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:47.985689 17290 consensus_peers.cc:485] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:47.999208 17422 consensus_peers.cc:485] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:48.011257 17422 consensus_peers.cc:485] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:48.019155 17422 consensus_peers.cc:485] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:48.100876 17422 consensus_peers.cc:485] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:48.131218 17026 consensus_peers.cc:485] T ae4259dd3dc745869256269632886f8a P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:48.175556 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:00:48.262118 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:00:48.306244 19173 consensus_peers.cc:485] T 885fe0c3a5e04f408e5b02c3c133248a P 9564311af29a4493831a8db658a085c4 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:48.366195 17026 consensus_peers.cc:485] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:48.377048 17290 consensus_peers.cc:485] T a39f284167e942188c181147e8a9e74e P d43ed6b8663947d99a72cd922cc323aa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:48.496910 17686 consensus_peers.cc:485] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:48.527283 17026 consensus_peers.cc:485] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:48.590487 17686 consensus_peers.cc:485] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:48.635221 17158 consensus_peers.cc:485] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:48.656690 17422 consensus_peers.cc:485] T 42a900125241432a8bb31dad0b940968 P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:48.660949 17686 consensus_peers.cc:485] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:48.665772 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:00:48.749804 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:00:48.757220 17422 consensus_peers.cc:485] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:48.791684 17290 consensus_peers.cc:485] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:48.885617 17158 consensus_peers.cc:485] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:49.057709 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:00:49.200383 17026 consensus_peers.cc:485] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:49.314530 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:00:49.499585 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:00:49.690675 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:00:49.846541 17026 consensus_peers.cc:485] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:49.945614 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:00:50.042891 17422 consensus_peers.cc:485] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:50.113423 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:00:50.162971 17026 consensus_peers.cc:485] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:50.230005 17026 consensus_peers.cc:485] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:50.243844 17554 consensus_peers.cc:485] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:50.244997 17026 consensus_peers.cc:485] T e245c9cf642043cfbb3b87114beeb002 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:50.276708 17290 consensus_peers.cc:485] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:50.289410 17554 consensus_peers.cc:485] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:50.358264 17026 consensus_peers.cc:485] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:50.474303 17686 consensus_peers.cc:485] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:50.483582 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:00:50.492100 17422 consensus_peers.cc:485] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:50.525728 17422 consensus_peers.cc:485] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:50.532207 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:00:50.551743 19173 consensus_peers.cc:485] T 885fe0c3a5e04f408e5b02c3c133248a P 9564311af29a4493831a8db658a085c4 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:50.622102 17554 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240503 02:00:50.636268 17422 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111) [suppressed 94 similar messages]
W20240503 02:00:50.641566 17026 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111) [suppressed 107 similar messages]
W20240503 02:00:50.642920 17686 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111) [suppressed 45 similar messages]
W20240503 02:00:50.649003 17422 consensus_peers.cc:485] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:50.671227 17026 consensus_peers.cc:485] T ae4259dd3dc745869256269632886f8a P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:50.681674 17290 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111) [suppressed 32 similar messages]
W20240503 02:00:50.840600 17026 consensus_peers.cc:485] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:50.853869 17422 consensus_peers.cc:485] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:50.857932 17158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111) [suppressed 23 similar messages]
W20240503 02:00:50.858750 17158 consensus_peers.cc:485] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:50.882512 17290 consensus_peers.cc:485] T a39f284167e942188c181147e8a9e74e P d43ed6b8663947d99a72cd922cc323aa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:50.885468 17422 consensus_peers.cc:485] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:51.022315 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:00:51.030318 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:00:51.030980 19173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111) [suppressed 12 similar messages]
W20240503 02:00:51.040566 17422 consensus_peers.cc:485] T 42a900125241432a8bb31dad0b940968 P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:51.041747 17686 consensus_peers.cc:485] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:51.048568 17686 consensus_peers.cc:485] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:51.166543 17290 consensus_peers.cc:485] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:51.312150 17686 consensus_peers.cc:485] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:51.386204 17158 consensus_peers.cc:485] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:51.393025 17026 consensus_peers.cc:485] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:51.397281 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:00:51.465574 17422 consensus_peers.cc:485] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:51.622970 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:00:51.886689 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:00:51.898087 17026 consensus_peers.cc:485] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:52.136626 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:00:52.265151 17554 consensus_peers.cc:485] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:52.302978 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:00:52.343132 17026 consensus_peers.cc:485] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:52.471261 17422 consensus_peers.cc:485] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:52.601122 17026 consensus_peers.cc:485] T e245c9cf642043cfbb3b87114beeb002 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:52.626180 17026 consensus_peers.cc:485] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:52.679955 17026 consensus_peers.cc:485] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:52.695920 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:00:52.702381 17422 consensus_peers.cc:485] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:52.768981 17026 consensus_peers.cc:485] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:52.819067 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:00:52.830474 17554 consensus_peers.cc:485] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:52.876132 17290 consensus_peers.cc:485] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:53.029701 17422 consensus_peers.cc:485] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:53.077546 19173 consensus_peers.cc:485] T 885fe0c3a5e04f408e5b02c3c133248a P 9564311af29a4493831a8db658a085c4 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:53.085096 17422 consensus_peers.cc:485] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:53.088024 17686 consensus_peers.cc:485] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:53.135247 17026 consensus_peers.cc:485] T ae4259dd3dc745869256269632886f8a P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:53.154776 17158 consensus_peers.cc:485] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:53.155922 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:00:53.173501 17422 consensus_peers.cc:485] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:53.230792 17026 consensus_peers.cc:485] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:53.399142 17686 consensus_peers.cc:485] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:53.410681 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:00:53.410742 17422 consensus_peers.cc:485] T 42a900125241432a8bb31dad0b940968 P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:53.491029 17422 consensus_peers.cc:485] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:53.568753 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:00:53.637878 17290 consensus_peers.cc:485] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:53.751457 17686 consensus_peers.cc:485] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:53.776793 17290 consensus_peers.cc:485] T a39f284167e942188c181147e8a9e74e P d43ed6b8663947d99a72cd922cc323aa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:53.782946 17686 consensus_peers.cc:485] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:53.825449 17422 consensus_peers.cc:485] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:53.858232 17158 consensus_peers.cc:485] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:53.933187 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:00:53.977241 17026 consensus_peers.cc:485] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:53.992062 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:00:54.241617 17026 consensus_peers.cc:485] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:54.469640 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:00:54.582192 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:00:54.674394 17026 consensus_peers.cc:485] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:54.835098 17554 consensus_peers.cc:485] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:54.904032 17422 consensus_peers.cc:485] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:54.952126 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:00:55.022109 17290 consensus_peers.cc:485] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:55.090179 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:00:55.098652 17026 consensus_peers.cc:485] T e245c9cf642043cfbb3b87114beeb002 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:55.121353 17026 consensus_peers.cc:485] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:55.127199 17554 consensus_peers.cc:485] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:55.127452 17026 consensus_peers.cc:485] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:55.171526 17422 consensus_peers.cc:485] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:55.207208 17026 consensus_peers.cc:485] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:55.359853 19173 consensus_peers.cc:485] T 885fe0c3a5e04f408e5b02c3c133248a P 9564311af29a4493831a8db658a085c4 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:55.401118 17686 consensus_peers.cc:485] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:55.424937 17422 consensus_peers.cc:485] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:55.532223 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:00:55.628940 17158 consensus_peers.cc:485] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:55.640976 17026 consensus_peers.cc:485] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:55.677532 17422 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240503 02:00:55.677564 17554 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 02:00:55.678236 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:00:55.680426 17026 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111) [suppressed 104 similar messages]
W20240503 02:00:55.681030 17026 consensus_peers.cc:485] T ae4259dd3dc745869256269632886f8a P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:55.689713 17422 consensus_peers.cc:485] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:55.737798 17290 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 02:00:55.792539 17686 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 02:00:55.793232 17686 consensus_peers.cc:485] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:55.903983 17422 consensus_peers.cc:485] T 42a900125241432a8bb31dad0b940968 P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:55.929817 17290 consensus_peers.cc:485] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:55.939464 17422 consensus_peers.cc:485] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:56.093487 17422 consensus_peers.cc:485] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:56.095726 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:00:56.123108 17686 consensus_peers.cc:485] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:56.194619 17422 consensus_peers.cc:485] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:56.195153 17290 consensus_peers.cc:485] T a39f284167e942188c181147e8a9e74e P d43ed6b8663947d99a72cd922cc323aa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:56.199532 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:00:56.219223 17158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240503 02:00:56.359586 17158 consensus_peers.cc:485] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:56.426584 19173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 02:00:56.436406 17686 consensus_peers.cc:485] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:56.650449 17026 consensus_peers.cc:485] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:56.655571 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:00:56.720480 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:00:56.789199 17026 consensus_peers.cc:485] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:56.922129 17026 consensus_peers.cc:485] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:57.084081 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:00:57.216322 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:00:57.256767 17554 consensus_peers.cc:485] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:57.362483 17422 consensus_peers.cc:485] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:57.441432 17026 consensus_peers.cc:485] T e245c9cf642043cfbb3b87114beeb002 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:57.482251 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:00:57.527223 17026 consensus_peers.cc:485] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:57.603211 17026 consensus_peers.cc:485] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:57.611871 17554 consensus_peers.cc:485] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:57.645290 17026 consensus_peers.cc:485] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:57.767639 17422 consensus_peers.cc:485] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:57.796805 17422 consensus_peers.cc:485] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:57.810724 17290 consensus_peers.cc:485] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:57.829588 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:00:57.876605 19173 consensus_peers.cc:485] T 885fe0c3a5e04f408e5b02c3c133248a P 9564311af29a4493831a8db658a085c4 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:57.880232 17686 consensus_peers.cc:485] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:57.893190 17422 consensus_peers.cc:485] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:57.917145 17158 consensus_peers.cc:485] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:57.932397 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:00:58.191506 17026 consensus_peers.cc:485] T ae4259dd3dc745869256269632886f8a P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:58.294775 17422 consensus_peers.cc:485] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:58.299795 17686 consensus_peers.cc:485] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:58.301896 17422 consensus_peers.cc:485] T 42a900125241432a8bb31dad0b940968 P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:58.339113 17026 consensus_peers.cc:485] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:58.348582 17686 consensus_peers.cc:485] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:58.396420 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:00:58.410826 17290 consensus_peers.cc:485] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:58.469518 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:00:58.633168 17422 consensus_peers.cc:485] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:58.772445 17290 consensus_peers.cc:485] T a39f284167e942188c181147e8a9e74e P d43ed6b8663947d99a72cd922cc323aa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:58.788401 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:00:58.858448 17422 consensus_peers.cc:485] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:58.983866 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:00:58.997071 17158 consensus_peers.cc:485] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:59.156091 17686 consensus_peers.cc:485] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:59.212112 17026 consensus_peers.cc:485] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:59.244952 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:00:59.323048 17026 consensus_peers.cc:485] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:59.323577 17026 consensus_peers.cc:485] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:59.409969 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:00:59.664840 17026 consensus_peers.cc:485] T e245c9cf642043cfbb3b87114beeb002 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:59.669559 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:00:59.714193 17422 consensus_peers.cc:485] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:59.771982 17554 consensus_peers.cc:485] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:59.800374 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:00.155663 17026 consensus_peers.cc:485] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:01:00.177574 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:00.184823 19173 consensus_peers.cc:485] T 885fe0c3a5e04f408e5b02c3c133248a P 9564311af29a4493831a8db658a085c4 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:01:00.209949 17026 consensus_peers.cc:485] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:01:00.231592 17554 consensus_peers.cc:485] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:01:00.256913 17686 consensus_peers.cc:485] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:01:00.265431 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:00.348429 17026 consensus_peers.cc:485] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:01:00.365234 17422 consensus_peers.cc:485] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:01:00.418789 17290 consensus_peers.cc:485] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:01:00.519995 17422 consensus_peers.cc:485] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:01:00.533128 17422 consensus_peers.cc:485] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:01:00.596422 17158 consensus_peers.cc:485] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:01:00.613516 17686 consensus_peers.cc:485] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:01:00.622256 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:00.631656 17422 consensus_peers.cc:485] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:01:00.687474 17422 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111) [suppressed 93 similar messages]
W20240503 02:01:00.733023 17026 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111) [suppressed 95 similar messages]
W20240503 02:01:00.791936 17290 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240503 02:01:00.792744 17290 consensus_peers.cc:485] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:01:00.794090 17422 consensus_peers.cc:485] T 42a900125241432a8bb31dad0b940968 P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:01:00.812466 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:00.840430 17686 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 02:01:00.841256 17686 consensus_peers.cc:485] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:01:00.852571 17554 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 02:01:00.868724 17026 consensus_peers.cc:485] T ae4259dd3dc745869256269632886f8a P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:01:00.939210 17026 consensus_peers.cc:485] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:01:01.199630 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:01.259828 17422 consensus_peers.cc:485] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:01:01.283650 17422 consensus_peers.cc:485] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:01:01.286777 17158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 02:01:01.287556 17158 consensus_peers.cc:485] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:01:01.363070 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:01.385784 17290 consensus_peers.cc:485] T a39f284167e942188c181147e8a9e74e P d43ed6b8663947d99a72cd922cc323aa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:01:01.421164 17686 consensus_peers.cc:485] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:01:01.648100 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:01.691164 17026 consensus_peers.cc:485] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:01:01.746078 19173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 02:01:01.812598 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:01.888208 17026 consensus_peers.cc:485] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:01:01.947649 17026 consensus_peers.cc:485] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:01:02.130746 17422 consensus_peers.cc:485] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:01:02.169947 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:02.262238 17554 consensus_peers.cc:485] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:01:02.307498 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:02.367970 17026 consensus_peers.cc:485] T e245c9cf642043cfbb3b87114beeb002 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:01:02.615236 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:02.769774 17422 consensus_peers.cc:485] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:01:02.772282 19173 consensus_peers.cc:485] T 885fe0c3a5e04f408e5b02c3c133248a P 9564311af29a4493831a8db658a085c4 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:01:02.811118 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:02.830648 17026 consensus_peers.cc:485] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:01:02.878366 17686 consensus_peers.cc:485] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:01:02.885941 17422 consensus_peers.cc:485] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:01:02.965152 17026 consensus_peers.cc:485] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:01:03.012482 17554 consensus_peers.cc:485] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:01:03.032984 17422 consensus_peers.cc:485] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:01:03.033890 17026 consensus_peers.cc:485] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:01:03.060912 17686 consensus_peers.cc:485] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:01:03.066090 17686 consensus_peers.cc:485] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:01:03.084466 17290 consensus_peers.cc:485] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:01:03.177470 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:03.216153 17158 consensus_peers.cc:485] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:01:03.219883 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:03.272331 17026 consensus_peers.cc:485] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:01:03.290704 17422 consensus_peers.cc:485] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:01:03.380190 17026 consensus_peers.cc:485] T ae4259dd3dc745869256269632886f8a P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:01:03.389144 17422 consensus_peers.cc:485] T 42a900125241432a8bb31dad0b940968 P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:01:03.490391 17422 consensus_peers.cc:485] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:01:03.509222 17422 consensus_peers.cc:485] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:01:03.550642 17290 consensus_peers.cc:485] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:01:03.677982 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:03.745512 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:03.923190 17026 consensus_peers.cc:485] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:01:04.029302 17686 consensus_peers.cc:485] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:01:04.055846 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:04.166389 17158 consensus_peers.cc:485] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:01:04.197860 17290 consensus_peers.cc:485] T a39f284167e942188c181147e8a9e74e P d43ed6b8663947d99a72cd922cc323aa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:01:04.259806 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:04.318116 17026 consensus_peers.cc:485] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:01:04.570731 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:04.664134 17026 consensus_peers.cc:485] T e245c9cf642043cfbb3b87114beeb002 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:01:04.680070 17422 consensus_peers.cc:485] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:01:04.693413 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:04.723474 17554 consensus_peers.cc:485] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:01:04.751662 17026 consensus_peers.cc:485] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:01:04.968602 17026 consensus_peers.cc:485] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:01:05.128296 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:05.135005 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:05.148249 19173 consensus_peers.cc:485] T 885fe0c3a5e04f408e5b02c3c133248a P 9564311af29a4493831a8db658a085c4 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:01:05.161823 17422 consensus_peers.cc:485] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:01:05.207389 17686 consensus_peers.cc:485] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:01:05.286562 17026 consensus_peers.cc:485] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:01:05.501196 17554 consensus_peers.cc:485] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:01:05.590602 17422 consensus_peers.cc:485] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:01:05.612377 17422 consensus_peers.cc:485] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:01:05.612342 17026 consensus_peers.cc:485] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:01:05.621291 17290 consensus_peers.cc:485] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:01:05.622041 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:05.625921 17686 consensus_peers.cc:485] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:01:05.628935 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:05.685782 17422 consensus_peers.cc:485] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:01:05.703058 17422 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111) [suppressed 91 similar messages]
W20240503 02:01:05.703955 17422 consensus_peers.cc:485] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:01:05.735752 17026 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111) [suppressed 102 similar messages]
W20240503 02:01:05.736471 17026 consensus_peers.cc:485] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:01:05.767906 17686 consensus_peers.cc:485] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:01:05.796209 17422 consensus_peers.cc:485] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:01:05.849332 17158 consensus_peers.cc:485] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:01:05.891328 17422 consensus_peers.cc:485] T 42a900125241432a8bb31dad0b940968 P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:01:05.910913 17554 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 02:01:05.994764 17026 consensus_peers.cc:485] T ae4259dd3dc745869256269632886f8a P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:01:06.012959 17686 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240503 02:01:06.039870 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:06.085187 17290 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240503 02:01:06.086040 17290 consensus_peers.cc:485] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:01:06.098857 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:06.452869 17158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20240503 02:01:06.536063 17026 consensus_peers.cc:485] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:01:06.536558 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:06.538288 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:06.562608 17026 consensus_peers.cc:485] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:01:06.617571 17290 consensus_peers.cc:485] T a39f284167e942188c181147e8a9e74e P d43ed6b8663947d99a72cd922cc323aa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:01:06.660866 17158 consensus_peers.cc:485] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:01:06.758530 17686 consensus_peers.cc:485] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:01:06.766678 19173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 02:01:06.915203 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:06.987944 17026 consensus_peers.cc:485] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:01:06.988787 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:07.075163 17026 consensus_peers.cc:485] T e245c9cf642043cfbb3b87114beeb002 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:01:07.183319 17554 consensus_peers.cc:485] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:01:07.202659 17026 consensus_peers.cc:485] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:01:07.294070 17422 consensus_peers.cc:485] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:01:07.372812 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:07.459230 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:07.630606 17422 consensus_peers.cc:485] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:01:07.714392 17686 consensus_peers.cc:485] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:01:07.808389 19173 consensus_peers.cc:485] T 885fe0c3a5e04f408e5b02c3c133248a P 9564311af29a4493831a8db658a085c4 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:01:07.816176 17026 consensus_peers.cc:485] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:01:07.858013 17686 consensus_peers.cc:485] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:01:07.887679 17554 consensus_peers.cc:485] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:01:07.931617 17422 consensus_peers.cc:485] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:01:07.945732 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:07.980415 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:08.000257 17422 consensus_peers.cc:485] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:01:08.125425 17026 consensus_peers.cc:485] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:01:08.167570 17422 consensus_peers.cc:485] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:01:08.249729 17290 consensus_peers.cc:485] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:01:08.264266 17026 consensus_peers.cc:485] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:01:08.310818 17422 consensus_peers.cc:485] T 42a900125241432a8bb31dad0b940968 P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:01:08.319628 17422 consensus_peers.cc:485] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:01:08.356763 17026 consensus_peers.cc:485] T ae4259dd3dc745869256269632886f8a P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:01:08.370486 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:08.373989 17686 consensus_peers.cc:485] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:01:08.390333 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:08.424000 17290 consensus_peers.cc:485] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:01:08.495019 17422 consensus_peers.cc:485] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:01:08.610292 17158 consensus_peers.cc:485] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:01:08.819671 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:08.849167 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:08.972333 17026 consensus_peers.cc:485] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:01:09.122170 17290 consensus_peers.cc:485] T a39f284167e942188c181147e8a9e74e P d43ed6b8663947d99a72cd922cc323aa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:01:09.164002 17026 consensus_peers.cc:485] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:01:09.242688 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:09.396596 17686 consensus_peers.cc:485] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:01:09.458868 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:09.464151 17026 consensus_peers.cc:485] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:01:09.482293 17026 consensus_peers.cc:485] T e245c9cf642043cfbb3b87114beeb002 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:01:09.562165 17026 consensus_peers.cc:485] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:01:09.582458 17158 consensus_peers.cc:485] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:01:09.623047 17554 consensus_peers.cc:485] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:01:09.637240 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:09.949174 17422 consensus_peers.cc:485] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:01:10.016705 17026 consensus_peers.cc:485] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:01:10.043069 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:10.100324 17686 consensus_peers.cc:485] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:01:10.117049 17422 consensus_peers.cc:485] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:01:10.156983 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:10.250881 17554 consensus_peers.cc:485] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:01:10.305900 19173 consensus_peers.cc:485] T 885fe0c3a5e04f408e5b02c3c133248a P 9564311af29a4493831a8db658a085c4 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:01:10.375792 17422 consensus_peers.cc:485] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:01:10.441560 17686 consensus_peers.cc:485] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:01:10.487350 17026 consensus_peers.cc:485] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:01:10.541198 17026 consensus_peers.cc:485] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:01:10.550575 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:10.605412 17422 consensus_peers.cc:485] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:01:10.651535 17422 consensus_peers.cc:485] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:01:10.706444 17422 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111) [suppressed 89 similar messages]
W20240503 02:01:10.707152 17422 consensus_peers.cc:485] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:01:10.708089 17422 consensus_peers.cc:485] T 42a900125241432a8bb31dad0b940968 P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:01:10.745801 17026 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111) [suppressed 103 similar messages]
W20240503 02:01:10.746654 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:10.898067 17686 consensus_peers.cc:485] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:01:10.989048 17026 consensus_peers.cc:485] T ae4259dd3dc745869256269632886f8a P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:01:11.002697 17290 consensus_peers.cc:485] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:01:11.043165 17422 consensus_peers.cc:485] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:01:11.053459 17554 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 02:01:11.106575 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:11.110472 17290 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 02:01:11.111230 17290 consensus_peers.cc:485] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:01:11.137208 17686 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 02:01:11.153059 17158 consensus_peers.cc:485] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:01:11.350888 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:11.415452 17290 consensus_peers.cc:485] T a39f284167e942188c181147e8a9e74e P d43ed6b8663947d99a72cd922cc323aa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:01:11.482623 17026 consensus_peers.cc:485] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:01:11.609345 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:11.646884 17158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20240503 02:01:11.761323 17026 consensus_peers.cc:485] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:01:11.787962 17686 consensus_peers.cc:485] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:01:11.886134 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:12.023656 19173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 02:01:12.054757 17026 consensus_peers.cc:485] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:01:12.106110 17026 consensus_peers.cc:485] T e245c9cf642043cfbb3b87114beeb002 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:01:12.107966 17026 consensus_peers.cc:485] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:01:12.126470 17554 consensus_peers.cc:485] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:01:12.156459 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:12.160995 17158 consensus_peers.cc:485] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:01:12.207820 17422 consensus_peers.cc:485] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:01:12.233127 17422 consensus_peers.cc:485] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:01:12.324445 17026 consensus_peers.cc:485] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:01:12.362120 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:12.419725 19173 consensus_peers.cc:485] T 885fe0c3a5e04f408e5b02c3c133248a P 9564311af29a4493831a8db658a085c4 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:01:12.592372 17554 consensus_peers.cc:485] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:01:12.610625 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:12.769763 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:12.796501 17686 consensus_peers.cc:485] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:01:12.898334 17422 consensus_peers.cc:485] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:01:12.942309 17686 consensus_peers.cc:485] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:01:12.952396 17026 consensus_peers.cc:485] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:01:13.127220 17422 consensus_peers.cc:485] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:01:13.128255 17026 consensus_peers.cc:485] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:01:13.199990 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:13.306526 17422 consensus_peers.cc:485] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:01:13.328940 17422 consensus_peers.cc:485] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:01:13.351784 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:13.364681 17422 consensus_peers.cc:485] T 42a900125241432a8bb31dad0b940968 P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:01:13.398221 17290 consensus_peers.cc:485] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:01:13.447340 17290 consensus_peers.cc:485] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:01:13.510999 17686 consensus_peers.cc:485] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:01:13.569919 17026 consensus_peers.cc:485] T ae4259dd3dc745869256269632886f8a P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:01:13.635052 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:13.715687 17158 consensus_peers.cc:485] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:01:13.753580 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:13.763474 17290 consensus_peers.cc:485] T a39f284167e942188c181147e8a9e74e P d43ed6b8663947d99a72cd922cc323aa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:01:13.869712 17026 consensus_peers.cc:485] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:01:13.877805 17422 consensus_peers.cc:485] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:01:14.061280 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:14.183707 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:14.228834 17686 consensus_peers.cc:485] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:01:14.382015 17026 consensus_peers.cc:485] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:01:14.451578 17026 consensus_peers.cc:485] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:01:14.455463 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:14.474284 17026 consensus_peers.cc:485] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:01:14.621126 17158 consensus_peers.cc:485] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:01:14.621738 17554 consensus_peers.cc:485] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:01:14.649926 17422 consensus_peers.cc:485] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:01:14.708002 17026 consensus_peers.cc:485] T e245c9cf642043cfbb3b87114beeb002 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:01:14.730150 17422 consensus_peers.cc:485] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:01:14.778740 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:14.831457 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:14.885833 17026 consensus_peers.cc:485] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:01:15.033821 19173 consensus_peers.cc:485] T 885fe0c3a5e04f408e5b02c3c133248a P 9564311af29a4493831a8db658a085c4 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:01:15.287597 17686 consensus_peers.cc:485] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:01:15.318225 17554 consensus_peers.cc:485] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:01:15.372793 17026 consensus_peers.cc:485] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:01:15.401605 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:15.445223 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:15.539661 17422 consensus_peers.cc:485] T 42a900125241432a8bb31dad0b940968 P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:01:15.562623 17686 consensus_peers.cc:485] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:01:15.567410 17422 consensus_peers.cc:485] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:01:15.601737 17422 consensus_peers.cc:485] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:01:15.734421 17422 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240503 02:01:15.735334 17422 consensus_peers.cc:485] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:01:15.756903 17026 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111) [suppressed 96 similar messages]
W20240503 02:01:15.778450 17026 consensus_peers.cc:485] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:01:15.862749 17686 consensus_peers.cc:485] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:01:15.902405 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:15.930131 17422 consensus_peers.cc:485] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:01:15.932251 17290 consensus_peers.cc:485] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:01:16.003167 17290 consensus_peers.cc:485] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:01:16.012202 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:16.058127 17554 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 02:01:16.118909 17290 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 02:01:16.119593 17290 consensus_peers.cc:485] T a39f284167e942188c181147e8a9e74e P d43ed6b8663947d99a72cd922cc323aa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:01:16.214531 17026 consensus_peers.cc:485] T ae4259dd3dc745869256269632886f8a P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:01:16.257920 17686 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 02:01:16.356563 17422 consensus_peers.cc:485] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:01:16.479310 17026 consensus_peers.cc:485] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:01:16.505232 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:16.527416 17158 consensus_peers.cc:485] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:01:16.589365 17686 consensus_peers.cc:485] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:01:16.594717 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:16.681720 17158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 02:01:16.721294 17026 consensus_peers.cc:485] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:01:16.730126 17026 consensus_peers.cc:485] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:01:16.895756 17026 consensus_peers.cc:485] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:01:16.960222 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:17.027613 19173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 02:01:17.039655 17554 consensus_peers.cc:485] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:01:17.042271 17422 consensus_peers.cc:485] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:01:17.109179 17158 consensus_peers.cc:485] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:01:17.139226 17422 consensus_peers.cc:485] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:01:17.147437 17026 consensus_peers.cc:485] T e245c9cf642043cfbb3b87114beeb002 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:01:17.157354 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:17.389765 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:17.444574 17026 consensus_peers.cc:485] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:01:17.555696 19173 consensus_peers.cc:485] T 885fe0c3a5e04f408e5b02c3c133248a P 9564311af29a4493831a8db658a085c4 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:01:17.648077 17554 consensus_peers.cc:485] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:01:17.667182 17686 consensus_peers.cc:485] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:01:17.692896 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:17.849761 17026 consensus_peers.cc:485] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:01:17.864913 17026 consensus_peers.cc:485] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:01:17.990620 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:18.049057 17422 consensus_peers.cc:485] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:01:18.078270 17422 consensus_peers.cc:485] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:01:18.099812 17290 consensus_peers.cc:485] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:01:18.122396 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:18.135279 17686 consensus_peers.cc:485] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:01:18.144359 17686 consensus_peers.cc:485] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:01:18.179430 17422 consensus_peers.cc:485] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:01:18.218071 17422 consensus_peers.cc:485] T 42a900125241432a8bb31dad0b940968 P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:01:18.405462 17422 consensus_peers.cc:485] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:01:18.423615 17290 consensus_peers.cc:485] T a39f284167e942188c181147e8a9e74e P d43ed6b8663947d99a72cd922cc323aa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:01:18.571059 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:18.577885 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:18.608330 17026 consensus_peers.cc:485] T ae4259dd3dc745869256269632886f8a P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:01:18.710088 17290 consensus_peers.cc:485] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:01:18.732731 17422 consensus_peers.cc:485] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:01:18.932502 17158 consensus_peers.cc:485] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:01:19.089042 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:19.165746 17686 consensus_peers.cc:485] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:01:19.165874 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:19.192929 17026 consensus_peers.cc:485] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:01:19.204025 17026 consensus_peers.cc:485] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:01:19.227638 17026 consensus_peers.cc:485] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:01:19.246217 17026 consensus_peers.cc:485] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:01:19.318919 17554 consensus_peers.cc:485] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:01:19.425773 17026 consensus_peers.cc:485] T e245c9cf642043cfbb3b87114beeb002 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:01:19.521132 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:19.575752 17422 consensus_peers.cc:485] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:01:19.655016 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:19.737375 17158 consensus_peers.cc:485] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:01:19.818876 17026 consensus_peers.cc:485] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:01:19.862314 17422 consensus_peers.cc:485] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:01:20.069815 17686 consensus_peers.cc:485] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:01:20.104153 19173 consensus_peers.cc:485] T 885fe0c3a5e04f408e5b02c3c133248a P 9564311af29a4493831a8db658a085c4 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:01:20.119666 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:20.147363 17554 consensus_peers.cc:485] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:01:20.273144 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:20.310810 17026 consensus_peers.cc:485] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:01:20.413647 17422 consensus_peers.cc:485] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:01:20.454741 17026 consensus_peers.cc:485] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:01:20.472834 17422 consensus_peers.cc:485] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:01:20.500902 17290 consensus_peers.cc:485] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:01:20.564992 17422 consensus_peers.cc:485] T 42a900125241432a8bb31dad0b940968 P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:01:20.602463 17686 consensus_peers.cc:485] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:01:20.695585 17686 consensus_peers.cc:485] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:01:20.723750 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:20.761389 17026 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111) [suppressed 101 similar messages]
W20240503 02:01:20.770303 17422 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111) [suppressed 89 similar messages]
W20240503 02:01:20.784775 17422 consensus_peers.cc:485] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:01:20.792500 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:20.854745 17290 consensus_peers.cc:485] T a39f284167e942188c181147e8a9e74e P d43ed6b8663947d99a72cd922cc323aa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:01:20.879097 17422 consensus_peers.cc:485] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:01:21.008368 17422 consensus_peers.cc:485] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:01:21.060755 17026 consensus_peers.cc:485] T ae4259dd3dc745869256269632886f8a P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:01:21.176872 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:21.225036 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:21.232395 17290 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 02:01:21.249468 17158 consensus_peers.cc:485] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:01:21.259773 17554 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 02:01:21.336905 17290 consensus_peers.cc:485] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:01:21.556089 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:21.576189 17026 consensus_peers.cc:485] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:01:21.608422 17686 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240503 02:01:21.631419 17026 consensus_peers.cc:485] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:01:21.681560 17686 consensus_peers.cc:485] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:01:21.753720 17158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 02:01:21.774582 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:21.810851 17026 consensus_peers.cc:485] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:01:21.883975 17554 consensus_peers.cc:485] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:01:21.897174 17026 consensus_peers.cc:485] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:01:21.945652 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:21.970636 17026 consensus_peers.cc:485] T e245c9cf642043cfbb3b87114beeb002 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:01:22.108160 19173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 02:01:22.170379 17422 consensus_peers.cc:485] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:01:22.195279 17422 consensus_peers.cc:485] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:01:22.280529 17026 consensus_peers.cc:485] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:01:22.304388 17158 consensus_peers.cc:485] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:01:22.324079 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:22.338340 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:22.495307 19173 consensus_peers.cc:485] T 885fe0c3a5e04f408e5b02c3c133248a P 9564311af29a4493831a8db658a085c4 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:01:22.630744 17686 consensus_peers.cc:485] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:01:22.722071 17422 consensus_peers.cc:485] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:01:22.764410 17422 consensus_peers.cc:485] T 42a900125241432a8bb31dad0b940968 P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:01:22.797535 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:22.840611 17026 consensus_peers.cc:485] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:01:22.848685 17290 consensus_peers.cc:485] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:01:22.858934 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:22.916329 17554 consensus_peers.cc:485] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:01:23.015484 17026 consensus_peers.cc:485] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:01:23.055991 17686 consensus_peers.cc:485] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:01:23.212684 17422 consensus_peers.cc:485] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:01:23.215840 17686 consensus_peers.cc:485] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:01:23.250805 17422 consensus_peers.cc:485] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:01:23.322644 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:23.422111 17422 consensus_peers.cc:485] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:01:23.437186 17290 consensus_peers.cc:485] T a39f284167e942188c181147e8a9e74e P d43ed6b8663947d99a72cd922cc323aa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:01:23.475703 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:23.493921 17422 consensus_peers.cc:485] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:01:23.513705 17026 consensus_peers.cc:485] T ae4259dd3dc745869256269632886f8a P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:01:23.666409 17158 consensus_peers.cc:485] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:01:23.864071 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:23.889261 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:24.018383 17026 consensus_peers.cc:485] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:01:24.026191 17026 consensus_peers.cc:485] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:24.139238 17290 consensus_peers.cc:485] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:01:24.174809 17026 consensus_peers.cc:485] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:24.355055 17686 consensus_peers.cc:485] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:01:24.359849 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:24.373028 17026 consensus_peers.cc:485] T e245c9cf642043cfbb3b87114beeb002 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:24.374001 17554 consensus_peers.cc:485] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:24.413435 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:24.486677 17026 consensus_peers.cc:485] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:01:24.517869 17158 consensus_peers.cc:485] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:01:24.636731 17422 consensus_peers.cc:485] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:24.731160 17422 consensus_peers.cc:485] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:24.779348 19173 consensus_peers.cc:485] T 885fe0c3a5e04f408e5b02c3c133248a P 9564311af29a4493831a8db658a085c4 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:24.862036 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:24.871662 17422 consensus_peers.cc:485] T 42a900125241432a8bb31dad0b940968 P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:24.989284 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:25.027972 17290 consensus_peers.cc:485] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:25.133848 17026 consensus_peers.cc:485] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:25.297418 17554 consensus_peers.cc:485] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:25.307868 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:25.358462 17422 consensus_peers.cc:485] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:25.408567 17686 consensus_peers.cc:485] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:25.490501 17026 consensus_peers.cc:485] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:25.508270 17422 consensus_peers.cc:485] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:25.525049 17686 consensus_peers.cc:485] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:25.596817 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:25.672178 17026 consensus_peers.cc:485] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:25.805591 17422 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111) [suppressed 91 similar messages]
W20240503 02:01:25.806507 17026 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111) [suppressed 100 similar messages]
W20240503 02:01:25.806696 17422 consensus_peers.cc:485] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:25.894199 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:25.983104 17290 consensus_peers.cc:485] T a39f284167e942188c181147e8a9e74e P d43ed6b8663947d99a72cd922cc323aa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:25.989980 17686 consensus_peers.cc:485] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:26.011600 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:26.050426 17026 consensus_peers.cc:485] T ae4259dd3dc745869256269632886f8a P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:26.083817 17422 consensus_peers.cc:485] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:26.084229 17422 consensus_peers.cc:485] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:26.308501 17554 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 02:01:26.325723 17290 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240503 02:01:26.336256 17158 consensus_peers.cc:485] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:26.344985 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:26.425303 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:26.485893 17026 consensus_peers.cc:485] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:26.511065 17026 consensus_peers.cc:485] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:26.622529 17026 consensus_peers.cc:485] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:26.777271 17554 consensus_peers.cc:485] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:26.783666 17686 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 02:01:26.784394 17686 consensus_peers.cc:485] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:26.812103 17026 consensus_peers.cc:485] T e245c9cf642043cfbb3b87114beeb002 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:26.878523 17026 consensus_peers.cc:485] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:26.884275 17290 consensus_peers.cc:485] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:26.885840 17158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 02:01:26.887284 17422 consensus_peers.cc:485] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:26.891664 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:26.999576 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:27.115859 19173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 02:01:27.116596 19173 consensus_peers.cc:485] T 885fe0c3a5e04f408e5b02c3c133248a P 9564311af29a4493831a8db658a085c4 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:27.274677 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:27.281019 17026 consensus_peers.cc:485] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:27.337704 17158 consensus_peers.cc:485] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:27.418722 17422 consensus_peers.cc:485] T 42a900125241432a8bb31dad0b940968 P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:27.422850 17422 consensus_peers.cc:485] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:27.429100 17290 consensus_peers.cc:485] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:27.562999 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:27.616155 17422 consensus_peers.cc:485] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:27.698995 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:27.853356 17026 consensus_peers.cc:485] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:27.927856 17554 consensus_peers.cc:485] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:27.968703 17686 consensus_peers.cc:485] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:27.997539 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:28.121053 17686 consensus_peers.cc:485] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:28.206645 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:28.277913 17026 consensus_peers.cc:485] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:28.308957 17422 consensus_peers.cc:485] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:28.315517 17422 consensus_peers.cc:485] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:28.316190 17422 consensus_peers.cc:485] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:28.383759 17026 consensus_peers.cc:485] T ae4259dd3dc745869256269632886f8a P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:28.499042 17686 consensus_peers.cc:485] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:28.509850 17290 consensus_peers.cc:485] T a39f284167e942188c181147e8a9e74e P d43ed6b8663947d99a72cd922cc323aa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:28.573407 17422 consensus_peers.cc:485] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:28.577080 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:28.657742 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:28.845566 17158 consensus_peers.cc:485] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:29.122798 17026 consensus_peers.cc:485] T e245c9cf642043cfbb3b87114beeb002 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:29.122946 17026 consensus_peers.cc:485] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:29.142146 17026 consensus_peers.cc:485] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:29.176187 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:29.213074 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:29.216889 17290 consensus_peers.cc:485] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:29.243244 17026 consensus_peers.cc:485] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:29.325697 17422 consensus_peers.cc:485] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:29.361315 17554 consensus_peers.cc:485] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:29.406108 17026 consensus_peers.cc:485] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:29.647619 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:29.671247 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:29.672493 17686 consensus_peers.cc:485] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:29.673322 17290 consensus_peers.cc:485] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:29.674777 19173 consensus_peers.cc:485] T 885fe0c3a5e04f408e5b02c3c133248a P 9564311af29a4493831a8db658a085c4 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:29.725503 17158 consensus_peers.cc:485] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:29.830495 17422 consensus_peers.cc:485] T 42a900125241432a8bb31dad0b940968 P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:29.883881 17026 consensus_peers.cc:485] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:29.884091 17422 consensus_peers.cc:485] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:30.184834 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:30.251998 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:30.256222 17554 consensus_peers.cc:485] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:30.318580 17422 consensus_peers.cc:485] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:30.322551 17026 consensus_peers.cc:485] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:30.641093 17686 consensus_peers.cc:485] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:30.671597 17686 consensus_peers.cc:485] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:30.754263 17422 consensus_peers.cc:485] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:30.804653 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:30.853112 17026 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111) [suppressed 99 similar messages]
W20240503 02:01:30.854038 17026 consensus_peers.cc:485] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:30.861755 17026 consensus_peers.cc:485] T ae4259dd3dc745869256269632886f8a P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:30.872310 17422 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111) [suppressed 91 similar messages]
W20240503 02:01:30.876436 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:30.888085 17422 consensus_peers.cc:485] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:31.038995 17422 consensus_peers.cc:485] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:31.059197 17290 consensus_peers.cc:485] T a39f284167e942188c181147e8a9e74e P d43ed6b8663947d99a72cd922cc323aa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:31.060300 17422 consensus_peers.cc:485] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:31.164098 17686 consensus_peers.cc:485] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:31.198056 17158 consensus_peers.cc:485] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:31.272313 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:31.273245 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:31.311044 17554 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 02:01:31.460453 17290 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 02:01:31.480139 17026 consensus_peers.cc:485] T e245c9cf642043cfbb3b87114beeb002 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:31.527936 17026 consensus_peers.cc:485] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:31.562155 17422 consensus_peers.cc:485] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:31.680544 17026 consensus_peers.cc:485] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:31.681224 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:31.744719 17026 consensus_peers.cc:485] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:31.777891 17554 consensus_peers.cc:485] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:31.791312 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:31.794425 17026 consensus_peers.cc:485] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:31.923312 17290 consensus_peers.cc:485] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:31.969250 17422 consensus_peers.cc:485] T 42a900125241432a8bb31dad0b940968 P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:32.028168 17686 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 02:01:32.054790 19173 consensus_peers.cc:485] T 885fe0c3a5e04f408e5b02c3c133248a P 9564311af29a4493831a8db658a085c4 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:32.062922 17290 consensus_peers.cc:485] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:32.070220 17158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 02:01:32.261022 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:32.265094 17158 consensus_peers.cc:485] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:32.280396 17686 consensus_peers.cc:485] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:32.404011 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:32.446835 17422 consensus_peers.cc:485] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:32.526054 19173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 02:01:32.565105 17026 consensus_peers.cc:485] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:32.806013 17026 consensus_peers.cc:485] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:32.807498 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:32.813830 17554 consensus_peers.cc:485] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:32.929123 17422 consensus_peers.cc:485] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:33.001731 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:33.153939 17686 consensus_peers.cc:485] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:33.208832 17290 consensus_peers.cc:485] T a39f284167e942188c181147e8a9e74e P d43ed6b8663947d99a72cd922cc323aa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:33.303972 17686 consensus_peers.cc:485] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:33.367964 17422 consensus_peers.cc:485] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:33.378834 17158 consensus_peers.cc:485] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:33.404595 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:33.442862 17026 consensus_peers.cc:485] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:33.546330 17026 consensus_peers.cc:485] T ae4259dd3dc745869256269632886f8a P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:33.554036 17686 consensus_peers.cc:485] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:33.571740 17422 consensus_peers.cc:485] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:33.588987 17422 consensus_peers.cc:485] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:33.604694 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:33.915599 17422 consensus_peers.cc:485] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:33.962468 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:34.095947 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:34.117835 17026 consensus_peers.cc:485] T e245c9cf642043cfbb3b87114beeb002 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:34.235864 17026 consensus_peers.cc:485] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:34.268162 17554 consensus_peers.cc:485] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:34.308313 17290 consensus_peers.cc:485] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:34.337939 17026 consensus_peers.cc:485] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:34.341107 17026 consensus_peers.cc:485] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:34.349443 17290 consensus_peers.cc:485] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:34.392493 17422 consensus_peers.cc:485] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:34.474449 19173 consensus_peers.cc:485] T 885fe0c3a5e04f408e5b02c3c133248a P 9564311af29a4493831a8db658a085c4 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:34.511878 17026 consensus_peers.cc:485] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:34.520223 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:34.631086 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:34.643647 17026 consensus_peers.cc:485] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:34.701386 17422 consensus_peers.cc:485] T 42a900125241432a8bb31dad0b940968 P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:34.811007 17686 consensus_peers.cc:485] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:34.861403 17158 consensus_peers.cc:485] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:34.912442 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:35.107777 17422 consensus_peers.cc:485] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:35.229135 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:35.469204 17026 consensus_peers.cc:485] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:35.520009 17422 consensus_peers.cc:485] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:35.523948 17554 consensus_peers.cc:485] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:35.535250 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:35.613579 17026 consensus_peers.cc:485] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:35.670428 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:35.693791 17158 consensus_peers.cc:485] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:35.722666 17290 consensus_peers.cc:485] T a39f284167e942188c181147e8a9e74e P d43ed6b8663947d99a72cd922cc323aa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:35.822217 17686 consensus_peers.cc:485] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:35.884711 17422 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111) [suppressed 86 similar messages]
W20240503 02:01:35.885430 17422 consensus_peers.cc:485] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:35.891538 17422 consensus_peers.cc:485] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:35.932406 17026 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111) [suppressed 99 similar messages]
W20240503 02:01:36.016506 17686 consensus_peers.cc:485] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:36.021747 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:36.028766 17686 consensus_peers.cc:485] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:36.109035 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:36.152993 17422 consensus_peers.cc:485] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:36.158898 17026 consensus_peers.cc:485] T ae4259dd3dc745869256269632886f8a P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:36.261353 17422 consensus_peers.cc:485] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:36.567317 17290 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111) [suppressed 32 similar messages]
W20240503 02:01:36.568015 17290 consensus_peers.cc:485] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:36.575343 17026 consensus_peers.cc:485] T e245c9cf642043cfbb3b87114beeb002 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:36.592334 17026 consensus_peers.cc:485] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:36.618950 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:36.667485 17554 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 02:01:36.690981 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:36.708012 17554 consensus_peers.cc:485] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:36.716969 17026 consensus_peers.cc:485] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:36.894194 17026 consensus_peers.cc:485] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:36.957881 19173 consensus_peers.cc:485] T 885fe0c3a5e04f408e5b02c3c133248a P 9564311af29a4493831a8db658a085c4 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:36.979959 17290 consensus_peers.cc:485] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:37.032243 17422 consensus_peers.cc:485] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:37.110687 17686 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 02:01:37.125638 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:37.166669 17026 consensus_peers.cc:485] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:37.178007 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:37.200893 17422 consensus_peers.cc:485] T 42a900125241432a8bb31dad0b940968 P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:37.311949 17026 consensus_peers.cc:485] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:37.332791 17686 consensus_peers.cc:485] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:37.341082 17158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 02:01:37.341737 17158 consensus_peers.cc:485] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:37.646509 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:37.680688 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:37.753955 17422 consensus_peers.cc:485] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:37.909341 19173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 02:01:37.972669 17026 consensus_peers.cc:485] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:38.002816 17422 consensus_peers.cc:485] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:38.060349 17026 consensus_peers.cc:485] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:38.152681 17554 consensus_peers.cc:485] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:38.158012 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:38.249686 17158 consensus_peers.cc:485] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:38.279892 17422 consensus_peers.cc:485] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:38.305064 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:38.313810 17686 consensus_peers.cc:485] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:38.330612 17290 consensus_peers.cc:485] T a39f284167e942188c181147e8a9e74e P d43ed6b8663947d99a72cd922cc323aa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:38.383077 17422 consensus_peers.cc:485] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:38.415412 17422 consensus_peers.cc:485] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:38.416045 17422 consensus_peers.cc:485] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:38.555361 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:38.627861 17026 consensus_peers.cc:485] T ae4259dd3dc745869256269632886f8a P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:38.645473 17686 consensus_peers.cc:485] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:38.652489 17686 consensus_peers.cc:485] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:38.696054 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:38.949931 17026 consensus_peers.cc:485] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:38.983367 17026 consensus_peers.cc:485] T e245c9cf642043cfbb3b87114beeb002 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:39.077473 17290 consensus_peers.cc:485] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:39.136533 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:39.181314 17026 consensus_peers.cc:485] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:39.269739 17554 consensus_peers.cc:485] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:39.291674 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:39.350658 17026 consensus_peers.cc:485] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:39.394273 17422 consensus_peers.cc:485] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:39.400285 17290 consensus_peers.cc:485] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:39.511415 19173 consensus_peers.cc:485] T 885fe0c3a5e04f408e5b02c3c133248a P 9564311af29a4493831a8db658a085c4 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:39.522886 17026 consensus_peers.cc:485] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:39.583243 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:39.670715 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:39.672348 17686 consensus_peers.cc:485] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:39.722494 17422 consensus_peers.cc:485] T 42a900125241432a8bb31dad0b940968 P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:39.757151 17026 consensus_peers.cc:485] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:39.867801 17158 consensus_peers.cc:485] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:40.149276 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:40.251828 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:40.313954 17422 consensus_peers.cc:485] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:40.470476 17422 consensus_peers.cc:485] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:40.666581 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:40.695230 17026 consensus_peers.cc:485] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:40.726320 17290 consensus_peers.cc:485] T a39f284167e942188c181147e8a9e74e P d43ed6b8663947d99a72cd922cc323aa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:40.730471 17026 consensus_peers.cc:485] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:40.741922 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:40.766997 17686 consensus_peers.cc:485] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:40.804910 17422 consensus_peers.cc:485] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:40.809132 17422 consensus_peers.cc:485] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:40.818130 17422 consensus_peers.cc:485] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:40.841116 17554 consensus_peers.cc:485] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:40.926288 17158 consensus_peers.cc:485] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:40.954396 17026 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111) [suppressed 100 similar messages]
W20240503 02:01:41.010116 17422 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111) [suppressed 93 similar messages]
W20240503 02:01:41.105160 17026 consensus_peers.cc:485] T ae4259dd3dc745869256269632886f8a P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:41.148113 17686 consensus_peers.cc:485] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:41.148861 17422 consensus_peers.cc:485] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:41.160322 17686 consensus_peers.cc:485] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:41.179257 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:41.293071 17026 consensus_peers.cc:485] T e245c9cf642043cfbb3b87114beeb002 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:41.345837 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:41.463650 17026 consensus_peers.cc:485] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:41.615921 17026 consensus_peers.cc:485] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:41.617883 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:41.664765 17290 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240503 02:01:41.782297 17422 consensus_peers.cc:485] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:41.804613 17290 consensus_peers.cc:485] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:41.819089 17026 consensus_peers.cc:485] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:41.838232 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:41.845585 17554 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 02:01:41.949824 17422 consensus_peers.cc:485] T 42a900125241432a8bb31dad0b940968 P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:41.959266 19173 consensus_peers.cc:485] T 885fe0c3a5e04f408e5b02c3c133248a P 9564311af29a4493831a8db658a085c4 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:42.026170 17554 consensus_peers.cc:485] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:42.030246 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:42.038182 17686 consensus_peers.cc:485] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:42.060127 17290 consensus_peers.cc:485] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:42.070281 17026 consensus_peers.cc:485] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:42.133910 17686 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 02:01:42.232599 17026 consensus_peers.cc:485] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:42.359771 17158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 02:01:42.360755 17158 consensus_peers.cc:485] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:42.455830 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:42.520970 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:42.666042 17422 consensus_peers.cc:485] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:42.899784 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:42.972870 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:43.084048 17422 consensus_peers.cc:485] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:43.101419 17290 consensus_peers.cc:485] T a39f284167e942188c181147e8a9e74e P d43ed6b8663947d99a72cd922cc323aa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:43.119608 17026 consensus_peers.cc:485] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:43.149933 17026 consensus_peers.cc:485] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:43.159263 19173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 02:01:43.205428 17422 consensus_peers.cc:485] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:43.229705 17422 consensus_peers.cc:485] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:43.255728 17554 consensus_peers.cc:485] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:43.348088 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:43.360616 17686 consensus_peers.cc:485] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:43.430140 17422 consensus_peers.cc:485] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:43.472411 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:43.497576 17158 consensus_peers.cc:485] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:43.563803 17026 consensus_peers.cc:485] T ae4259dd3dc745869256269632886f8a P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:43.618039 17422 consensus_peers.cc:485] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:43.656450 17686 consensus_peers.cc:485] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:43.694725 17686 consensus_peers.cc:485] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:43.885361 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:43.904158 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:43.920094 17026 consensus_peers.cc:485] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:43.994700 17026 consensus_peers.cc:485] T e245c9cf642043cfbb3b87114beeb002 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:44.216523 17026 consensus_peers.cc:485] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:44.218256 17026 consensus_peers.cc:485] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:44.241407 17554 consensus_peers.cc:485] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:44.248772 17422 consensus_peers.cc:485] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:44.280807 17290 consensus_peers.cc:485] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:44.391819 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:44.480221 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:44.508520 17422 consensus_peers.cc:485] T 42a900125241432a8bb31dad0b940968 P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:44.529726 17686 consensus_peers.cc:485] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:44.575546 17290 consensus_peers.cc:485] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:44.642272 17026 consensus_peers.cc:485] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:44.739008 17026 consensus_peers.cc:485] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:44.740278 19173 consensus_peers.cc:485] T 885fe0c3a5e04f408e5b02c3c133248a P 9564311af29a4493831a8db658a085c4 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:44.788352 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:45.038030 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:45.347375 17422 consensus_peers.cc:485] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:45.361753 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:45.494868 17422 consensus_peers.cc:485] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:45.530537 17422 consensus_peers.cc:485] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:45.531090 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:45.547138 17422 consensus_peers.cc:485] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:45.594053 17158 consensus_peers.cc:485] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:45.706501 17026 consensus_peers.cc:485] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:45.715392 17026 consensus_peers.cc:485] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:45.743351 17290 consensus_peers.cc:485] T a39f284167e942188c181147e8a9e74e P d43ed6b8663947d99a72cd922cc323aa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:45.798837 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:45.882488 17554 consensus_peers.cc:485] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:45.896481 17686 consensus_peers.cc:485] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:45.979574 17422 consensus_peers.cc:485] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:45.995219 17158 consensus_peers.cc:485] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:45.999310 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:46.063066 17422 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240503 02:01:46.096385 17686 consensus_peers.cc:485] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:46.122865 17026 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111) [suppressed 102 similar messages]
W20240503 02:01:46.194324 17422 consensus_peers.cc:485] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:46.209740 17026 consensus_peers.cc:485] T ae4259dd3dc745869256269632886f8a P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:46.224485 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:46.324263 17686 consensus_peers.cc:485] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:46.426774 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:46.452728 17554 consensus_peers.cc:485] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:01:46.551674 17026 consensus_peers.cc:485] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:01:46.708031 17290 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240503 02:01:46.712473 17026 consensus_peers.cc:485] T e245c9cf642043cfbb3b87114beeb002 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:01:46.714659 17026 consensus_peers.cc:485] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:46.806341 17026 consensus_peers.cc:485] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:46.807637 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:46.846001 17290 consensus_peers.cc:485] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:01:46.894166 17026 consensus_peers.cc:485] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:01:46.896785 17290 consensus_peers.cc:485] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:46.909296 17422 consensus_peers.cc:485] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:01:46.962852 17554 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 02:01:46.969592 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:47.111578 17686 consensus_peers.cc:485] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:47.201743 17422 consensus_peers.cc:485] T 42a900125241432a8bb31dad0b940968 P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:01:47.240363 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:47.282258 17686 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 02:01:47.288408 17026 consensus_peers.cc:485] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:01:47.339524 19173 consensus_peers.cc:485] T 885fe0c3a5e04f408e5b02c3c133248a P 9564311af29a4493831a8db658a085c4 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:01:47.535379 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:47.586122 17158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 02:01:47.626369 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:47.800632 17422 consensus_peers.cc:485] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:01:47.923413 17422 consensus_peers.cc:485] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:01:48.017648 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:48.030596 17158 consensus_peers.cc:485] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:48.074803 17026 consensus_peers.cc:485] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:01:48.077586 17422 consensus_peers.cc:485] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:01:48.126256 17422 consensus_peers.cc:485] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:01:48.164551 17026 consensus_peers.cc:485] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:01:48.204026 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:48.206039 19173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 02:01:48.231734 17290 consensus_peers.cc:485] T a39f284167e942188c181147e8a9e74e P d43ed6b8663947d99a72cd922cc323aa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:01:48.368319 17554 consensus_peers.cc:485] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:01:48.403790 17422 consensus_peers.cc:485] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:01:48.409018 17158 consensus_peers.cc:485] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:01:48.415416 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:48.496699 17686 consensus_peers.cc:485] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:01:48.591286 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:48.692714 17422 consensus_peers.cc:485] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:01:48.763219 17026 consensus_peers.cc:485] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:01:48.778877 17026 consensus_peers.cc:485] T ae4259dd3dc745869256269632886f8a P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:01:48.807386 17686 consensus_peers.cc:485] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:01:48.876677 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:48.881636 17686 consensus_peers.cc:485] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:01:49.015818 17554 consensus_peers.cc:485] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:01:49.063738 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:49.181886 17290 consensus_peers.cc:485] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:01:49.182466 17026 consensus_peers.cc:485] T e245c9cf642043cfbb3b87114beeb002 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:01:49.226831 17290 consensus_peers.cc:485] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:01:49.232395 17026 consensus_peers.cc:485] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:01:49.337990 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:49.394542 17422 consensus_peers.cc:485] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:01:49.409118 17422 consensus_peers.cc:485] T 42a900125241432a8bb31dad0b940968 P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:01:49.437479 17026 consensus_peers.cc:485] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:01:49.480115 17026 consensus_peers.cc:485] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:01:49.488580 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:49.825191 17686 consensus_peers.cc:485] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:01:49.872004 19173 consensus_peers.cc:485] T 885fe0c3a5e04f408e5b02c3c133248a P 9564311af29a4493831a8db658a085c4 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:01:49.905819 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:49.911705 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:49.969411 17026 consensus_peers.cc:485] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:01:50.344429 17422 consensus_peers.cc:485] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:01:50.371493 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:50.390431 17422 consensus_peers.cc:485] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:01:50.489193 17158 consensus_peers.cc:485] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:01:50.495779 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:50.631625 17026 consensus_peers.cc:485] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:01:50.681828 17422 consensus_peers.cc:485] T d52afe1281064d6480427554ca4bb2f6 P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:01:50.792287 17554 consensus_peers.cc:485] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:01:50.858609 17290 consensus_peers.cc:485] T a39f284167e942188c181147e8a9e74e P d43ed6b8663947d99a72cd922cc323aa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:01:50.862264 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:50.879317 17158 consensus_peers.cc:485] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:01:50.914904 17026 consensus_peers.cc:485] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:01:50.929868 17422 consensus_peers.cc:485] T fbec07350ad04c49afd42e5d88dda6ad P 16d437fe629643948c95d628926d5ffa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:01:50.942014 17422 consensus_peers.cc:485] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:01:50.992314 17026 consensus_peers.cc:485] T 78f81fe4c6234ff0b5fcabd974779a66 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:01:51.052160 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:51.073927 17422 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240503 02:01:51.127707 17686 consensus_peers.cc:485] T 78d106d705df4089a4dfae4d5dc533cc P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:01:51.136232 17026 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111) [suppressed 101 similar messages]
W20240503 02:01:51.218089 17422 consensus_peers.cc:485] T ec69ad86be3d4793b10aca98dc4c1736 P 16d437fe629643948c95d628926d5ffa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:01:51.275964 17026 consensus_peers.cc:485] T ae4259dd3dc745869256269632886f8a P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:01:51.414733 17554 consensus_peers.cc:485] T d81dcfefb4fd401b9cfd2ca36034a010 P c843193b801b4f1a88f630ed3ed0f375 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:01:51.424860 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:51.434528 17686 consensus_peers.cc:485] T 5d2104698d5f452c8350cce10afd3898 P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:01:51.435581 17686 consensus_peers.cc:485] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:01:51.621066 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:51.708257 17026 consensus_peers.cc:485] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:01:51.732966 17026 consensus_peers.cc:485] T e245c9cf642043cfbb3b87114beeb002 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:01:51.748515 17290 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240503 02:01:51.749637 17290 consensus_peers.cc:485] T 63962b169caf4f14888871e246c00075 P d43ed6b8663947d99a72cd922cc323aa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:01:51.841894 17026 consensus_peers.cc:485] T 30abfada8ab44449b9a4237c57093b73 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:01:51.869473 17026 consensus_peers.cc:485] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:01:51.907066 17422 consensus_peers.cc:485] T 42a900125241432a8bb31dad0b940968 P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:01:51.926481 17290 consensus_peers.cc:485] T ab536115ac7a4ba193f718d421f4000d P d43ed6b8663947d99a72cd922cc323aa -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:01:51.931331 17422 consensus_peers.cc:485] T acfd773c1f384c219d776878407d1144 P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:01:51.997596 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:52.047444 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:52.113669 17554 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240503 02:01:52.334808 19173 consensus_peers.cc:485] T 885fe0c3a5e04f408e5b02c3c133248a P 9564311af29a4493831a8db658a085c4 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Couldn't send request to peer 375d0f2ec11d4e25bd9d093bb6a5572d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:01:52.363319 17686 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240503 02:01:52.364022 17686 consensus_peers.cc:485] T fdf0a4b9e17642849a4094d2f0b9a255 P 002cc3411acb4fdfa0b67c6f57f687a6 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:01:52.474895 17026 consensus_peers.cc:462] T 81b54f3bcf34485eb11b1bc954a2bd8f P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:52.585258 17422 consensus_peers.cc:462] T 3f67591d6a3744ab90a3e4d2375a59db P 16d437fe629643948c95d628926d5ffa -> Peer 375d0f2ec11d4e25bd9d093bb6a5572d (127.0.99.136:40167): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:40167: connect: Connection refused (error 111)
W20240503 02:01:52.622767 17158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 02:01:52.623487 17158 consensus_peers.cc:485] T 4782de9826d6460e9e993e01327034e4 P 5b8291b26a7f448b8636187d1caf86d0 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn't send request to peer 838ef968bb45484bb27f5af6f3989bae. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:41887: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:01:52.784241 17026 consensus_peers.cc:485] T 5833b4f9a5434734817f154d660aa9f7 P 89dd09f8aaee4b249ae3ce924a422a67 -> Peer 838ef968bb45484bb27f5af6f3989bae (127.0.99.137:41887): Couldn'